The experimental method in Public Administration: lessons from replication in Psychology
DOI:
https://doi.org/10.1590/0034-761220200746Abstract
In recent years, an increasing number of studies adopting the experimental method have appeared in Public Administration journals. It has been argued that the advantage of experiments in behavioral sciences is that researchers can control contextual factors while investigating the effect of manipulations on a variable of interest. Another point is that experiments can be replicated and, thus, increase confidence in research results. However, replications are rarely undertaken, especially in the behavioral sciences. This article examines the results of the “Open Science Reproducibility Project: Psychology,” which replicated 100 experiments previously published in leading Psychology journals. Based on the findings of this project, we present seven recommendations to Public Administration scholars that can improve the quality of their experiments.
Downloads
References
American Psychological Association. (s.d.). APA Dictionary of Psychology. Recuperado de https://dictionary.apa.org/pretest-posttest-design
Armhein, V., & Greenland, S. (2018). Remove, rather than redefine, statistical significance (correspondence). Nature Human Behaviour, 4(1), 4.
Baker, M. (2016, março 03). Psychology’s reproducibility problem is exaggerated — say psychologists. Nature News. Recuperado de http://www.nature.com/news/psychology-s-reproducibility-problem-is-exaggerated-say-psychologists-1.19498#/b4
Baldassarri, D., & Abascal, M. (2017). Field experiments across the social sciences. Annual Review of Sociology, 43, 41-73.
Banerjeea, A., & Solomon, B. D. (2003). Eco-labeling for energy efficiency and sustainability: a meta-evaluation of US programs. Energy Policy, 31, 109-123.
Baranski, E. (2015). Replication of “On the relative independence of thinking biases and cognitive ability” by KE Stanovich, RF West (2008, Journal of Personality and Social Psychology). Recuperado de https://osf.io/p3gz2/
Barber, T. X., Calverley, D. S., Forgione, A., McPeake, J. D., Chaves, J. F., … Bowen, B. (1969). Five attempts to replicate the experimenter bias effect. Journal of Consulting and Clinical Psychology, 33(1), 1-6.
Bellé, N. (2013). Experimental evidence on the relationship between public service motivation and job performance. Public Administration Review, 73(1), 143-153.
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E. J., Berk, R., ... Cesarini, D. (2018). Redefine statistical significance. Nature – Human Behaviour, 2(1), 6-10.
Bonate, P. L. (2000). Analysis of pretest-posttest designs. Boca Raton, FL: Chapman & Hall/CRC Press.
Bouwman, R., & Grimmelikhuijsen, S. (2016). Experimental public administration from 1992 to 2014: a systematic literature review and ways forward. International Journal of Public Sector Management, 29(2), 110-131.
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., ... Van’t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217-224.
Brown, B., Brown, K., Attridge, P., DeGaetano, M., Hicks, G., Humphries, D. ... Mainard, H. (2013). Replication of Study 5 by Centerbar, Schnall, Clore, & Gavin, (2008, JPSP). Recuperado de https://osf.io/wcgx5/
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., ... Altmejd, A. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637-644.
Carpenter, S. (2012). Psychology’s bold initiative. Science, 335(6076), 1558-1561.
Cooper, H. M. (2017). Research synthesis and meta-analysis: A step-by-step approach (5a ed.). London, UK: Sage.
Embley, J., Johnson, L. G., & Giner-Sorolla, R. (2015). Replication of Study 1 by Vohs & Schooler (2008, Psychological Science). Recuperado de https://osf.io/2nf3u/
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science” (Technical Comments). Science, 351(6277), 1037b.
Goodman, S. (2008, julho). A dirty dozen: twelve p-value misconceptions. Seminars in Hematology, 45(3), 135-140.
Grimmelikhuijsen, S., Jilke, S., Olsen, A. L., & Tummers, L. (2017). Behavioral public administration: Combining insights from public administration and psychology. Public Administration Review, 77(1), 45-56.
Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (2016). What if there were no significance tests? (Original work published 1997). London, UK: Routledge.
Holubar, T. (2015). Replication of “The rejection of moral rebels,” Study 4, by Monin, Sawyer, & Marquez (2008, JPSP). Recuperado de https://osf.io/a4fmg/
Howell, D. C. (2012). Statistical methods for psychology. Belmond, UK: Wadsworth/Cengage Learning.
Ioannidis, J. P., Munafo, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences, 18(5), 235-241.
Jacob, T. (1968). The experimenter bias effect: a failure to replicate. Psychonomic Science, 13(4), 239-240.
James, O., Jilke, S. R., & Van Ryzin, G. G. (2017). Behavioural and experimental public administration: Emerging contributions and new directions. Public Administration, 95(4), 865-873.
Jilke, S., Van de Walle, S., & Kim, S. (2016). Generating Usable Knowledge through an Experimental Approach to Public Administration. Public Administration Review, 76(1), 69-72.
Johnson, K. M., Hayes T., & Graham, J. (2015). Replication of Study 2 by Amodio, Devine, & Harmon- Jones (2008, Journal of Personality and Social Psychology). Recuperado de https://osf.io/ysxmf/
Jost, J. T., & Kruglanski, A. W. (2002). The estrangement of social constructionism and experimental social psychology: History of the rift and prospects for reconciliation. Personality and Social Psychology Review, 6(3), 168-187.
Kelson, K. (2015). Replication of “The space between us: Stereotype threat and distance in interracial contexts” by P.A. Goff, C.M. Steele, and P.G. Davies (Journal of Personality and Social Psychology, 2008). Recuperado de https://osf.io/7q5us/
Lane, K., & Gazarian, D. (2015). Replication of “The effects of an Implemental mind-set on attitude strength” by Henderson, de Liver, & Gollwitzer (2008, Journal of Personality and Social Psychology). Recuperado de https://osf.io/xqjf4/
Lecoutre, M. P., Poitevineau, J., & Lecoutre, B. (2003). Even statisticians are not immune to misinterpretations of Null Hypothesis Significance Tests. International Journal of Psychology, 38(1), 37-45.
Lehmann, E. L. (2011). Fisher, Neyman, and the creation of slassical Statistics. New York, NY: Springer.
Lemn, K. (2013). Replication of Blankenship and Wegener (2008, JPSP, Study 5A). Recuperado de https://osf.io/v3e2z/
Lewis, M., & Pitts, M. (2015). Replication of “Errors are Aversive” by Greg Hajcak & Dan Foti (2008, Psychological Science). Recuperado de https://osf.io/tkq9n/
Lin, S. (2013). Replication of Study 7 by Exline, Baumeister, Zell, Kraft & Witvliet (2008, Journal of Personality and Social Psychology). Recuperado de https://osf.io/svz7w/
Lindsay, D. S. (2020). Seven steps toward transparency and replicability in psychological science. Canadian Psychology/Psychologie canadienne, 61(4), 310-317. Recuperado de https://doi.org/10.1037/cap0000222
Marigold, D. C., Forest, A. L., & Anderson, J. E. (2015). Replication of “How the head liberates the heart: Projection of communal responsiveness guides relationship promotion” by EP Lemay Jr and MS Clark (2008, JPSP). Recuperado de https://osf.io/mv3i7/
Mechin, N., & Gable, P. (2015). Replication of “Left frontal cortical activation and spreading of alternatives: Test of the action- based model of dissonance” by E Harmon-Jones, C Harmon-Jones, M Fearn, JD Sigelman, P Johnson (2008, Journal of Personality and Social Psychology). Recuperado de https://osf.io/zpwne/
Moynihan, D. P. (2013). Does public service motivation lead to budget maximization? Evidence from an experiment. International Public Management Journal, 16(2), 179-196.
Niskanen, W. A. (1968). The peculiar economics of bureaucracy. The American Economic Review, 58(2), 293-305.
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615-631.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716-1-aac4716-8.
Passmore, D. L., & Chae, C. (2019). Potential for meta-scientific inquiry to improve the usefulness of HRD research outcomes for practice. Advances in Developing Human Resources, 21(4), 409-420.
Perry, J. L., & Wise, L. R. 1990. The Motivational Bases of Public Service. Public Administration Review, 50(3), 367-73.
Reinhard, D. (2014). Replication of Förster, J., Liberman, N., & Kuschel, S. (2008). The effect of global versus local processing styles on assimilation versus contrast in social judgment, Journal of Personality and Social Psychology, 94, 579-599. Recuperado de https://osf.io/mxryb/
Robertson, P. J., & S. J. Seneviratne (1995). Outcomes of planned organizational change in the public sector: a meta-analytic comparison to the private sector. Public Administration Review, 55(6), 547-558.
Rosenthal, R., & Fode, K. L. (1963). The effect of experimenter bias on the performance of the albino rat. Systems Research and Behavioral Science, 8(3), 183-189.
Rozeboom, W. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57(5), 416-428.
Shadish, W., Cook, T. D., & Campwell, D. T. (2002). Experimental and quasi-exprimental design for generalized causal inference. Boston, MA: Houghton Mifflin Company.
Sobis, I., & De Vries, M. S. (2014). The social psychology perspective on values and virtue. In M. S. De Vries, & P. S. Kim (Eds.), Value and virtue in Public Administration: a comparative perspective (IIAS Series: Governance and Public Management). London, UK: McMillan-Palgrave.
Streiner, D. L. (2003). Unicorns do exist: A tutorial on “proving” the null hypothesis. The Canadian Journal of Psychiatry, 48(11), 756-761.
Talhelm, T., Lee. M., & Eggleston, C. (2015). Replication of Poignancy: Mixed Emotional Experience in the Face of Meaningful Endings by Ersner-Hershfield, Mikels, Sullivan, & Carstensen (2008, Journal of Personality and Social Psychology). Recuperado de https://osf.io/fw6hv/
Tepe, M., & Prokop, C. (2017). Laboratory experiments: their potential for public management Research. In O. James, S. R. Jilke, & G. G. Van Ryzin (Eds.), Experiments in public management research: challenges and contributions (1st ed., pp. 139-164) Cambridge, UK: Cambridge University Press.
Trafimow, D. (2014). Editorial. Basic and Applied Social Psychology, 36(1), 1-2. Recuperado de https://doi.org/10.1080/01973533.2014.865505
Trafimow, D., & Marks, M. (2015). Editorial. Basic and Applied Social Psychology, 37(1), 1-2. Recuperado de http://dx.doi.org/10.1080/0197353 3.2015.1012991
Van de Walle, S. (2016). The experimental turn in public management: How methodological preferences drive substantive choices. In O. James, S. Jilke, & O. Van Ryzin (Eds.), Experiments in public management research. Cambridge, UK: Cambridge University Press.
Van Kolfschooten, F. (2014). Fresh misconduct charges hit Dutch social psychology. Science Magazine, 344(6184), 566-567.
Vohs, K. D., & Schoolre, J. W. (2008). The value of believing in free will: encouraging a belief in determinism increases cheating. Psychological Science, 19(1), 49-54.
Wasserstein, R., & Lazar, N. (2016). The ASA’s statement on P values: context, process, and purpose. The American Statistician, 70(2), 129-133.
Downloads
Published
How to Cite
Issue
Section
License
The Brazilian Journal of Public Administration (RAP) undertakes to contribute to the protection of authors’ intellectual rights. On this matter:
- It uses the Creative Commons BY (CC-BY) license for all texts it publishes, except when there is indication of specific holders of copyrights and property rights;
- It uses the similarity verification software of content - Plagiarism (Crossref Similarity Check);
- It takes actions to fight against plagiarism and ethical misconduct aligned with the guidelines of the Committee on Publication Ethics (COPE).
Further information on the Code of Ethics adopted by RAP can be found in Ethical Standards and Code of Conduct.
