This article presents discussion and recommendations on approaches to retrospectively evaluating development interventions in the long term through a systems lens. It is based on experiences from the implementation of an 18-month study to investigate the impact of development interventions on economic and social change over a 40-year period in the Koshi Hills region of Nepal. A multi-disciplinary team used a mixed-methods approach to data collection and analysis. A theory-based analytical approach was utilized to produce narratives of plausible cause and effect and identify key drivers of change within the context of cumulative and interconnected impacts of multiple programs and factors. This article responds to increasing interest in development evaluation to look beyond intervention-specific impact to broader determinants of change to assist with intervention planning. It is the authors’ hope that this will stimulate debate and progress in the use of high quality research to inform future development work.
AxinnW. G.GhimireD. J. (2011). Social organization, population and land use. American Journal of Sociology, 117, 209–258.
2.
BambergerM. (2009). Why do so many evaluations have a positive bias? Paper presented at the Australasian Evaluation Society Annual Conference, Canberra, Australia.
3.
BambergerM.VijayendraR.WoolcockM. (2010). Using mixed methods in monitoring & evaluation: Experiences from international development. (World Bank Policy Research Working Paper no 5245). Washington, DC: World Bank.
4.
BanerjeeA. V. (2007). Making aid work. Cambridge, MA: MIT Press
5.
BanerjeeA. V.DufloE. (2009). The experimental approach to development economics. Annual Review of Economics, 1, 151–178.
6.
BlossfieldH. P.RohwerG. (1995). Techniques of event history modeling: New approaches to causal analysis. Hillsdale, NJ: Lawrence Erlbaum Associates.
7.
Central Department of Geography. (2001). Study on physical planning of human settlements and emerging towns along the Banepa-Bardibas highway corridor. Kathmandu, Nepal: Department of Urban Planning and Building Construction, Government of Nepal.
8.
ChambersR. (1994). Paradigm shifts and the practice of participatory research and development(IDS Working Paper no. 2). Brighton, England: IDS.
9.
Coffey & Metcon Consultants. (2010). Research into the long term impact of development interventions in the Koshi hills: Data collection phase. London: UK Department for International Development. Unpublished manuscript.
10.
ColemanJ. S. (1981). Longitudinal data analysis. New York, NY: Basic Books.
11.
CorynC. L. S.NoakesL. A.WestineC. D.SchroterD. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32, 199–226.
12.
CousinsJ. B.EarlL. M. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397–418.
13.
DaviesR. B.DaleA. (1994). Analysing social and political change: A casebook of methods. London, England: Sage.
14.
DonaldsonS. I. (2007). Program theory-driven evaluation science. New York, NY: Lawrence Erlbaum.
15.
DufloE.KremerM. (2005). Use of randomization in the evaluation of development effectiveness. In FeinsteinO.IngramG.PitmanG. (Eds.), Evaluating development effectiveness (pp. 205–231). New Brunwick, NJ: Transaction Publishers.
16.
FrechtlingJ. A. (2007). Logic modelling methods in program evaluation. San Francisco, CA: Jossey-Bass.
17.
GautamA. P.WebbE. L.EiumnohA. (2002). GIS assessment of land use/land cover changes associated with community forestry implementation in the middle hills of Nepal. Mountain Research and Development, 22, 63–69.
18.
GershunyJ. (1998). Thinking dynamically: Sociology and narrative data. In LeiseringL.WalkersR. (Eds.), The dynamics of modern society (pp. 34–48). Bristol, England: The Policy Press.
19.
GertlerP. J.MartinezS.PremandP.RawlingsL.VermeerschC. M. J. (2011). Impact evaluation in practice. Washington, DC: World Bank.
20.
HageboeckM.FrumkinM. (2009). Trends in international development evaluation theory, policy and practices(USAID Publication No. IMR2 Assignment Plan 72). Washington, DC: U.S. Agency for International Development.
21.
MagnussenD.BergmannL. R. (Eds). (1990). Data quality in longitudinal research. Cambridge, MA: Cambridge University Press.
22.
MayneJ. (2011). Contribution analysis: Addressing cause and effect. In SchwartzR.ForssK.MarraM. (Eds.), Evaluating the complex (pp. 53–96). New Brunswick, NJ: Transaction Publishers.
23.
MenardS. (1991). Longitudinal research. Newbury Park, CA: Sage.
24.
MiyoshiK. (2013). Toward a more holistic evaluation approach for rural development. American Journal of Evaluation, 34, 587.
25.
MorraL.RistR. (2009). The road to results: Designing and conducting effective evaluations. Washington, DC: World Bank.
26.
National Planning Commission & Department for International Development. (2013). Long term impact of development interventions – Koshi Hills, Nepal. Retrieved November 7, 2013, fromhttp://r4d.dfid.gov.uk/Output/194107/
27.
Organization of Economic Cooperation in Development. (2008). The Paris declaration on aid effectiveness and the Accra agenda for action. Geneva, Switzerland: OECD. Retrieved fromhttp://www.oecd.org/dac/effectiveness/34428351.pdf
28.
RaoV.WoolcockM. (2003). Integrating qualitative and quantitative approaches in program evaluation. In BourguignonF.Pereira da SilvaL. (Eds.), The impact of economic policies on poverty and income distribution: Evaluation techniques and tools (pp. 165–190). New York, NY: Oxford University Press.
29.
RossiP. H.FreemanH. E.LipseyM. W. (1999). Evaluation: A systematic approach (6th ed.). Thousand Oaks, CA: Sage.
30.
RuspiniE. (Ed.). (1999). Longitudinal analysis: A bridge between quantitative and qualitative social research [Special Issue]. Quality and Quantity, 33, 219–227.
31.
SternE.StameN.MayneJ.ForssK.DaviesR.BefaniB. (2012). Broadening the range of designs and methods for impact evaluations (Working Paper No. 38). London: UK Department for International Development.
32.
ThorntonP.AlbertynC.Bertelsmann-ScottT.VaillantC.VickeryC. (2010). Southern Africa Programme Evaluation 2004-2009. London, UK: Department for International Development.
