Recent reforms in K-12 education have increasingly emphasized the educational system as the unit of change. The theory underlying systemic or system wide educational reform is that powerful improvements in teaching and learning can come from developing coherence and alignment across the complex and different elements and components of an educational system. The growth of systemic reform efforts has led to a rise in attempts to systemically evaluate these initiatives. Systemic evaluations involve several distinct challenges for evaluators including capturing the system as the unit of intervention, assessing the alignment and coherence between components within the system, and identifying reasonable external comparison groups. Using the example of a school district’s systemic reform initiative, the authors examine the challenges and implications of systemic evaluation.
Allison, P. (2005). Fixed effects regression methods for longitudiual data using SAS. Cary, NC: SAS Institute.
2.
Berends, M., Kirby, S. N., Naftel, S., & McKelvey, C. J. (2001). Implementation and performance in New American Schools: Three years into scale-up. Santa Monica, CA: RAND.
3.
Berman, P. (1978). The study of macro- and micro-implementation. Public Policy, 26(2), 157-184.
4.
Brookings Institution. (1998). Learning what works: Evaluating complex social interventions. Washington, DCAuthor.
5.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
6.
Chatterji, M. (2002). Models and methods for examining standards-based reforms and accountability initiatives: Have the tools of inquiry answered pressing questions on improving schools?Review of Educational Research, 72(3), 345-386.
7.
Cohen, D. K. (1995). What is the system in systemic reform?Educational Researcher, 24(9), 11-17.
8.
Conrad, K. J., & Roberts-Gray, C. (1988). K. J. Conrad & C. Roberts-Gray (Eds.), In Evaluating program environments (pp. 1-5). San Francisco: Jossey-Bass.
9.
Cook, T. D. (2000). The false choice between theory-based evaluation and experimentation. In P. Rogers, T. Hacsi, A. Petrosino, & T. Huebner (Eds.), New directions for evaluation: Vol. 87. Program theory in evaluation: Challenges and opportunities (pp. 27-43). San Francisco: Jossey-Bass.
10.
Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.
11.
Datnow, A., Borman, G. D., Stringfield, S., Overman, L. T., & Castellano, M. (2003). Comprehensive school reform in culturally and linguistically diverse contexts: Implementation and outcomes from a four-year study. Educational Evaluation and Policy Analysis, 25, 143-170.
12.
Doran, H. C., & Izumi, L. T. (2004). Putting education to the test: A value-added model for California. San Francisco: Pacific Research Institute.
13.
Elmore, R. F., & Burney, D. (1997). Investing in teacher learning: Staff development and instructional improvement in Community School District #2, New York City. New York: National Commission on Teaching & America’s Future; Philadelphia: Consortium for Policy Research in Education.
14.
Firestone, W. A., & Pennell, J. R. (1997). Designing state-sponsored teacher networks: A comparison of two cases. American Educational Research Journal, 34(2), 237-266.
15.
Fuhrman, S. H. (1993). The politics of coherence. In S. H. Fuhrman (Ed.), Designing coherent education policy: Improving the system (pp. 1-34). San Francisco: Jossey-Bass.
16.
House, E. R. (2001). Unfinished business: Causes and values. American Journal of Evaluation, 22(3), 309-315.
17.
Kane, T. J., & Staiger, D. O. (2001). Improving school accountability measures (Working Paper No. 8156). Cambridge, MA: National Bureau of Economic Research.
18.
Koretz, D., & Hamilton, L. (2001). The performance of students with disabilities on New York’s Revised Regents Comprehensive Examination in English (CSE Tech. Rep. No. 540). Los Angeles: Center for Research on Evaluation, Standards, and Student Testing.
19.
Kubisch, A. C., Fulbright-Anderson, K., & Connell, J. P. (1998). Evaluating community initiatives: A progress report. In A. Kubisch, K. Fulbright-Anderson, & J. Connell (Eds.), New approaches to evaluating community initiatives: Vol. 2. Theory, measurement, and analysis (introduction). Retrieved October 8, 2004, from http://www.aspeninstitute.org/Programt1.asp?bid=1277
20.
Laguarda, K. G., Goldstein, D. S., Adelman, N. E., & Zucker, A. A. (1998). Assessing the SSI’s impacts on student achievement: An imperfect science. Menlo Park, CA: SRI International.
21.
Moos, R. (1987). The social climate scales: A user’s guide. Palo Alto, CA: Consulting Psychologists Press.
22.
National Science Foundation. (2004). A more synergistic whole. Education: Lessons about learning. Retrieved July 17, 2004, from http://www.nsf.gov/od/lpa/news/publicat/nsf0050/education/moresynergistic.htm
23.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
24.
Roberts-Gray, C., & Scheirer, M. A. (1988). Checking the congruence between a program and its organizational environment. In K .J. Conrad & C. Roberts-Gray (Eds.), New directions for program evaluation: Vol. 40. Evaluating program environments (pp. 63-82). San Francisco: Jossey-Bass.
25.
Rogers, P., Petrosino, A., Huebner, T., & Hacsi, T. (2000). Program theory evaluation: Practice, promise, and problems. In P. Rogers, T. Hacsi, A. Petrosino, & T. Huebner (Eds.), New directions for evaluation: Vol. 87. Program theory in evaluation: Challenges and opportunities (pp. 5-13). San Francisco: Jossey-Bass.
26.
Smith, M. S., & O’Day, J. (1991). Systemic school reform. In S. Fuhrman & B. Malen (Eds.), The politics of curriculum and testing (pp. 233-267). New York: Falmer.
27.
Snipes, J., Doolittle, F., & Herlihy, C. (2002). Foundations for success: Case studies of how urban school systems improve student achievement. Washington, DC: Council of Great City Schools.
28.
Supovitz, J. A., & Weathers, J. (2004). Dashboard lights: Monitoring implementation of district instructional reform strategies. Philadelphia: Consortium for Policy Research in Education.
29.
Togneri, W., & Anderson, S. E. (2003). Beyond islands of excellence: What districts can do to improve instruction and achievement in all schools—A leadership brief. Washington, DC: Learning First Alliance.
30.
Weiss, C. H. (1997). How can theory-based evaluation make greater headway?Evaluation Review, 21, 501-524.
31.
Weiss, C. H. (2000). Which links to which theories shall we evaluate?New Directions for Evaluation, 87, 35-45.
32.
Weitzman, B. C., Silver, D., & Dillman, K. (2002). Integrating a comparison group design into a theory of change evaluation: The case of the urban health initiative. American Journal of Evaluation, 23(4), 371-385.
33.
Zucker, A. A., Shields, P. M., Adelman, N. E., Corcoran, T. B., & Goertz, M. E. (1998). Statewide systemic initiatives programs: A report on the evaluation of the National Science Foundation’s Statewide Systemic Initiatives (SSI) program. Menlo Park, CA: SRI International.
