Evaluators sometimes wish for a Fairy Godmother who would make decision makers pay attention to evaluation findings when choosing programs to implement. The U.S. Department of Education came close to creating such a Fairy Godmother when it required school districts to choose drug abuse prevention programs only if their effectiveness was supported by “scientific” evidence. The experience showed advantages of such a procedure (e.g., reduction in support for D.A.R.E., which evaluation had found wanting) but also shortcomings (limited and in some cases questionable evaluation evidence in support of other programs). Federal procedures for identifying successful programs appeared biased. In addition, the Fairy Godmother discounted the professional judgment of local educators and did little to improve the fit of programs to local conditions. Nevertheless, giving evaluation more clout is a worthwhile way to increase the rationality of decision making. The authors recommend research on procedures used by other agencies to achieve similar aims.
Alkin, M.C., Daillak, R., & White, P. (1979). Using evaluations: Does evaluation make a difference?Beverly Hills, CA: Sage.
2.
Anderson, L. (2003). Pursuing truth, exercising power: Social science and public policy in the 21st century. New York: Columbia University Press.
3.
Baklien, B. (1983). The use of social science in a Norwegian ministry, as a tool of policy or a mode of thinking? Acta Sociologica , 26, 33-47.
4.
Becker, H. (1984). The case of Germany, experiences from the education council. In T. Husen & M. Kogan (Eds.), Educational research and policy: How do they relate? (pp. 103-119). Oxford, UK: Pergamon.
5.
Beyer, J.M., & Trice, H.M. (1982). The utilization process: A conceptual framework and synthesis of empirical findings. Administrative Science Quarterly, 27, 591-622.
6.
Biesta, G. (2007). Why “what works” won't work: Evidence-based practice and the democratic deficit in educational research. Education Theory, 57(1), 1-22.
7.
Birkeland, S., Murphy-Graham, E., & Weiss, C.H. (2005). Good reasons for ignoring good evaluation: Evidence from a study of the Drug Abuse Resistance Education (D.A.R.E.) program. Evaluation and Program Planning, 28(3), 247-256.
8.
Bobrow, D.B. (2006). Social and cultural factors: Constraining and enabling. In M. Moran, M. Rein, & R. E. Goodin (Eds.), The Oxford handbook of public policy (pp. 572-586). Oxford, UK: Oxford University Press.
9.
Boudett, K. P., City, E. A., & Murnane, R. J. (Eds.). (2005). Data wise: A step-by-step guide to using assessment results to improve teaching and learning. Cambridge, MA: Harvard Education Press.
10.
Bowen, S., & Zwi, A.B. (2005). Pathways to “evidence-informed” policy and practice: A framework for action. Public Library of Science , 2(7), e166, 600-605.
11.
Breslau, D. (1998). In search of the unequivocal: The political economy of measurement in U.S. labor market policy. Westport, CT : Praeger.
12.
Bulmer, M. (1986). Social science and social policy. London: Allen and Unwin.
13.
Burton, M., & Chapman, M. (2004). Problems of evidence based practice in community health and social services. Journal of Learning Disabilities , 8, 56-70.
14.
Campbell, S., Benita, S., Coates, E., Davies, P., & Penn, G. (2007). Analysis for policy: Evidence-based policy in practice. London: Government Social Research Unit, HM Treasury.
15.
Canadian Health Services Research Foundation. ( 2006). An emerging role for deliberative processes. Links, 9(4), 1-6.
16.
Caplan, N. (1977). A minimal set of conditions necessary for the utilization of social science knowledge in policy formulation at the national level. In C. H. Weiss (Ed.), Using research in public policy making (pp. 183-197). Lexington, MA: Lexington-Heath.
17.
Center for Substance Abuse Prevention, Substance Abuse and Mental Health Administration, Department of Health and Human Services. (2007, January). Identifying and selecting evidence-based interventions: Guidance document for the strategic prevention framework, State Incentive Grant Program. Washington, DC: Author.
18.
Center on Education Policy. (2006). Keeping watch on reading first. Washington, DC: Author.
19.
Clark, G., & Kelly, L. (2005). New directions for knowledge transfer and knowledge brokerage in Scotland. Scotland: Office of Chief Researcher, Knowledge Transfer Team, Scottish Executive Social Research [On-line]. Retrieved from http://www.scotland.gov.uk/Resource/Doc/69582/0018002.pdf
20.
Clayton, R., Cattarello, A.M., & Johnstone, B.M. (1996). The effectiveness of Drug Abuse Resistance Education (Project D.A.R.E.): 5-year follow up results. Preventative Medicine, 25, 307-318.
21.
Cousins, B., & Leithwood, C.K. (1986). Current empirical research on evaluation utilization. Review of Educational Research , 36(3), 331-364.
22.
Davies, H., Nutley, S., & Smith, P. (Eds.). (2000 ). What works? Evidence-based policy and practice in public services. Bristol, UK: Policy Press.
23.
DeJong, W. (1987). A short term evaluation of project D.A.R.E. (Drug Abuse Resistance Education): Preliminary indicators of effectiveness. Journal of Drug Education, 17, 279-294.
24.
Dopson, S., & Fitzgerald, L. (Eds.). (2005). Knowledge to action? Evidence-based health care in context. Oxford, UK: Oxford University Press.
25.
Dukes, R.L., Stein, J.A., & Ullman, J.B. (1997). Long-term impact of Drug Abuse Resistance Education (D.A.R.E.): Results of a 6-year follow-up. Evaluation Review, 21(4), 483-500.
26.
Ellickson, P.L., Bell, R.M., & McGuigan, K. (1993). Preventing adolescent drug use: Long-term results of a junior high program. American Journal of Public Health , 83, 856-861.
27.
Ennett, S.T., Tobler, N.S., Ringwalt, N.S., & Flewelling, F.L. (1994). Resistance education: A meta-analysis of project D.A.R.E. outcome evaluations. American Journal of Public Health, 84(9), 1394-1401.
28.
Furubo, J.E. (1994). Learning from evaluation: The Swedish experience . In F. L. Leeuw, R. C. Rist, & R. E. Sonnichsen (Eds.), Can governments learn? Comparative perspectives on evaluation and organisational learning (pp. 45-65). New Brunswick, NJ: Transaction.
29.
Gandhi, A.G., Murphy-Graham, E., Petrosino, A., Chrismer, S.S., & Weiss, C.H. (2007). The devil is in the details: Examining the evidence for “proven” school-based drug abuse prevention programs. Evaluation Review, 31(1), 43-74.
30.
Gorman, D.M. (2002). The “science” of drug and alcohol prevention: The case of the randomized trial of the Life Skills Training program . International Journal of Drug Policy, 13, 21-26.
31.
Greenberg, D., Mandell, M., & Onstott, M. (2000). The dissemination and utilization of welfare-to-work experiments in state policymaking. Journal of Policy Analysis and Management, 19(3), 367-382.
32.
Greenberg, D., & Shroder, M. (2004). The digest of social experiments (3rd ed.). Washington, DC: Urban Institute Press.
33.
Gueron, J.M., & Pauly, E. (1991). From welfare to work. New York : Russell Sage.
34.
Henig, J.R. (2007, May). The evolving relationship between researchers and public policy. Paper presented at American Enterprise Institute Conference, Washington, DC.
35.
Kemm, J. (2006). The limitations of “evidence-based” public health. Journal of Evaluation in Clinical Practice , 12(3), 319-324.
36.
Knott, J., & Wildavsky, A. (1980). If dissemination is the answer, what is the problem? Knowledge: Creation, Diffusion, Utilization, 1, 537-578.
37.
Lakoff, G. (1996). Moral politics. Chicago: University of Chicago Press.
38.
Lampinen, O. (1992). The utilisation of social science research in public policy. Helsinki, Finland: VAPK-Publishing.
39.
Lemieux-Charles, L., & Champagne, F. (Eds.). (2004). Using knowledge and evidence in health care: Multidisciplinary perspectives. Toronto, Ontario, Canada: University of Toronto Press.
40.
Lindblom, C.E., & Cohen, D.K. (1979). Usable knowledge: Social science and social problem solving. New Haven, CT: Yale University Press.
41.
Lyons, G.M. (1969). The uneasy partnership: Social science and the federal government in the twentieth century. New York: Russell Sage.
42.
Mulgar, M. (2005). Government, knowledge and the business of policy making: The potential and limits of evidence-based policy. Evidence and Policy, 1(2), 215-226.
43.
National Academy of Sciences. (1968). The behavioral sciences and the federal government. Washington, DC: Author.
44.
National Science Foundation. (1968). Knowledge into action: Improving the nation's use of the social sciences. Washington, DC: Author.
45.
Office of Chief Researcher, Knowledge Transfer Team (G. Clark & L. Kelly), Scottish Executive Social Research. (2005 ). New directions for knowledge transfer and knowledge brokerage in Scotland. Edinburgh, UK: Office of Chief Researcher.
46.
Office of Inspector General, U.S. Department of Education. (2006, September). The Reading First program's grant application process; final inspection report. Available at http://ED-OIG/I13-F0017
47.
Pawson, R. (2006). Evidence-based policy: A realist perspective. London: Sage.
48.
Petersilia, J. (1987). The influence of criminal justice research. Santa Monica, CA: Rand.
49.
Petrosino, A. (2003). Standards for evidence and evidence for standards: The case of school-based drug prevention. Annals of the American Academy of Political and Social Science, 587, 180-207.
50.
Petrosino, A., Birkeland, S., Hacsi, T.A., Murphy-Graham, E., & Weiss, C.H. (2006). US state government and D.A.R.E. : The story in four states. Evidence and Policy, 2(3), 291-319.
51.
Petrosino, A., & Soydan, H. (2005). The impact of program developers as evaluators on criminal recidivism: Results from meta-analyses of experimental and quasi-experimental research. Journal of Experimental Criminology, 1(4), 435-450.
52.
Pogrow, S. (1996). Reforming the wannabe reformers: Why education reforms almost always end up making things worse. Phi Delta Kappan, 77(10), 656-663.
53.
Preskill, H., & Torres, R. (1999). Evaluative inquiry for learning in organizations . Thousand Oaks, CA: Sage.
54.
Pyra, K. (2003). Knowledge translation: A review of the literature . Nova Scotia, Canada: Nova Scotia Health Research Foundation.
55.
Rich, A. (2004). Think tanks, public policy, and the politics of expertise. Cambridge, UK: Cambridge University Press.
56.
Rigby, E. (2005). Linking research and policy on Capitol Hill: Insights from research brokers. Evidence and Policy, 1(2), 195-213.
57.
Rosenbaum, D.P., Gordon, S., & Hanson, S. (1998). Assessing the effects of school-based drug education: A six-year multilevel analysis of project D.A.R.E. Journal of Research in Crime and Delinquency, 35, 381-412.
58.
Rosenstock, L., & Lee, L.J. (2002). Attacks on science: The risks to evidence-based policy. American Journal of Public Health, 92(1), 14-18.
59.
Rosenthal, R., & DiMatteo, M.R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59-82.
60.
Sanderson, I. (2002). Evaluation, policy learning and evidence-based policy making. Public Administration, 80(1), 1-22.
61.
Saunders, L. (2005). Research and policy: Reflections on their relationship . Evidence and Policy, 1(3), 383-390.
62.
Schoenfeld, A.H. (2006). What doesn't work: The challenge and failure of the What Works Clearinghouse to conduct meaningful review of studies of mathematics curricula. Educational Researcher, 35(2), 13-21.
63.
Schweinhart, L.J., Barnes, H.V., & Weikart, D.P. (1993). Significant benefits: The High/Scope Perry Preschool study through age 27. Ypsilanti, MI: High/Scope Press.
64.
Sherman, L.W. (2003). Misleading evidence and evidence-led policy: Making social science more experimental. Annals of the American Academy of Political and Social Science, 589, 6-19.
65.
Shonkoff, J.P. (2000). Science, policy, and practice: Three cultures in search of a shared mission. Child Development, 71(1), 181-187.
66.
Shulha, L., & Cousins, B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18(3), 195-208.
67.
Simmons, W. (2005). Evidence-based practice: Building capacity for informed professional judgment. Voices in Urban Education (Annenberg Institute for School Reform), 6, 5-13.
68.
Sowell, T. (1980). Knowledge and decisions. New York: Basic Books.
69.
St. Pierre,R.G., & Puma, M.J. (2000). Toward the dream of the experimenting society . In L. Bickman (Ed.), Validity and social experimentation: Donald Campbell's legacy (pp. 169-192). Thousand Oaks, CA: Sage.
70.
Taylor, D. (2005). Governing through evidence: Participation and power in policy evaluation. Journal of Social Policy, 34(4), 601-618.
71.
Tobler, N. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results compared to a control or comparison group. Journal of Drug Issues, 16, 537-556.
72.
Walter, I., Nutley, S., & Davies, H. (2005). What works to promote evidence-based practice? A cross-sector review. Evidence and Policy, 1(3), 335-363.
73.
Walter, I., Nutley, S., Percy-Smith, J., McNeish, D., & Frost, S. (2004). Improving the use of research in social care practice (Knowledge Review 7). London: Social Care Institute for Excellence.
74.
Weiss, C., Murphy-Graham, E., & Birkeland, S. (2005). An alternate route to policy influence: Evidence from a study of the Drug Abuse Resistance Education (D.A.R.E.) program. American Journal of Evaluation, 26, 12-31.
75.
Weiss, C. H. (Ed.). (1977). Using research in public policy making. Lexington, MA: Lexington Books/D.C. Heath and Co.
76.
Weiss, C.H. (1989). Congressional committees as users of analysis . Journal of Policy Analysis and Management, 8(3), 411-431.
77.
Weiss, C.H. (1995). The haphazard connection: Social science and public policy. International Journal of Educational Research, 23(2), 137-150.
