Ngaa-bi-nya Aboriginal and Torres Strait Islander program evaluation framework

Abstract

The Ngaa-bi-nya framework presented here is a practical guide for the evaluation of Aboriginal and Torres Strait Islander health and social programs. It has a range of prompts to stimulate thinking about critical success factors in programs relevant to Aboriginal and Torres Strait Islander people’s lives. Ngaa-bi-nya was designed from an Aboriginal practitioner-scholar standpoint and was informed by the holistic concept of Aboriginal health, case studies with Aboriginal-led social and emotional well-being programs, human rights instruments, and the work of Stufflebeam. Aboriginal and Torres Strait Islander health and social programs have been described as suffering from a lack of evaluation. Ngaa-bi-nya is one of the few tools developed specifically to reflect Aboriginal and Torres Strait Islander peoples’ contexts. It prompts the user to take into account the historical, policy, and social landscape of Aboriginal and Torres Strait Islander people’s lives, existing and emerging cultural leadership, and informal caregiving that supports programs. Ngaa-bi-nya’s prompts across four domains—landscape factors, resources, ways of working, and learnings—provide a structure through which to generate insights necessary for the future development of culturally relevant, effective, translatable, and sustainable programs required for Australia’s growing and diverse Aboriginal and Torres Strait Islander populations.

Introduction

Health and social programs developed by and aimed at improving the lives of Aboriginal and Torres Strait Islander people have been credited with bringing about significant change for participants (Aboriginal and Torres Strait Islander Healing Foundation, 2014; Bulman & Hayes, 2011; Tsey et al., 2010).

However, many Aboriginal and Torres Strait Islander health and social programs are not evaluated. A recent analysis found that only 88 of 1082 such programs (8%) had been evaluated or were under evaluation (Hudson, 2017). Of the 490 programs delivered by Indigenous organisations, 20 had been evaluated or were under evaluation (4%) (Hudson, 2017). Furthermore, a recent review of evaluation theories and approaches relevant for Aboriginal peoples reported that “research and evaluation are generally considered in the literature as a threat to Aboriginal communities, because of the history of the role of research in the colonisation of Aboriginal peoples” (Katz, Newton, Bates, & Raven, 2016, p. 39).

Leadership by Aboriginal and Torres Strait Islander people in research and evaluation has the potential to shift this dynamic. Focused attention on research capacity building by the Lowitja Institute in particular, Australia’s National Aboriginal and Torres Strait Islander Health Research Institute, has enabled such research leadership to strengthen (Sweet, 2017). A range of materials have been developed to guide engagement of Aboriginal and Torres Strait Islander service providers and community members in research, and many examples of co-created research have been reported (Laycock, Walker, Harrison, & Brands, 2011). Attention is increasingly on evaluation, and the need to better understand principles for evaluation, methodological approaches, tools, and mechanisms for translating findings in ways that improve health equity. Just as no “one size fits all” strategy for improving individual health or health equity exists, multiple adaptable frameworks and methods for evaluation are required (Blignault & Williams, 2017).

This article makes a contribution by describing a framework designed from Aboriginal health service delivery and research experience, which is flexible yet specific to the lives of Aboriginal and Torres Strait Islander peoples. It builds on the well-known Context, Input, Processes, Products (CIPP) model of Stufflebeam (2003), which has been used in a wide variety of settings. The CIPP model asks four overarching questions, with sub-prompts for each: What needs to be done? How should it be done? Is it being done? Is it succeeding? (Stufflebeam, 2003). Ngaa-bi-nya extends CIPP by providing prompts to stimulate data collection and analysis of factors relevant to Aboriginal and Torres Strait Islander peoples’ knowledge, values, ethics, and ways of caregiving that influence health and social support program success. This article describes the shape of the Ngaa-bi-nya framework and provides a list of prompt questions which, while not exhaustive, offers users a starting point for evaluation and to inspire identification of further local factors.

Development of the Ngaa-bi-nya evaluation framework

Ngaa-bi-nya (pronounced naa-bi-nya) means to examine, try, and evaluate in the language of the Wiradjuri peoples of central New South Wales (Grant & Rudder, 2010). The foundations of Ngaa-bi-nya were initially gradually developed across 15 years of service delivery and evaluation experience by the author, by adding prompts to the CIPP model to ensure that in evaluations, data about cultural and contextual factors relevant to Aboriginal and Torres Strait Islander people and programs were consciously collected, analysed, and reported. With the increasing amount and range of added prompts came the need to refine these, thereby triggering the focused development of a stand-alone framework. Several engagements with health promotion professionals provided instrumental guidance, including the International Union of Health Promotion and Education South West Pacific Group, who worked with the Australian Health Promotion Association, Cooperative Research Centre for Aboriginal Health, and Queensland Aboriginal and Islander Health Council to host an interactive Population Health Congress 2008 workshop on “What constitutes success in Aboriginal and Torres Strait Islander health?” The subsequent report produced six case studies, as well as critical insights into evaluation needs and opportunities (Erben, Judd, Ritchie, & Rowling, 2009). This collaboration led to a series of workshops for the 20th International Union for Health Promotion and Education (IUHPE) World Conference on Health Promotion in 2010, about the nature of evidence for effectiveness in Indigenous health programs. These diverse engagements with Indigenous peoples internationally, as well as from urban, regional, and remote Australia, timed with a literature review for Doctoral studies about definitions of success and effectiveness in health and criminal justice programs (Williams, 2015). Furthermore, collaboration with Aboriginal organisations through Project 10%, an Aboriginal-led campaign to reduce incarceration rates, resulted in clarification of indicators of quality and effectiveness to advocate to governments (Project 10%, 2010) and to guide the selection of 20 success stories showcased in an Aboriginal community publication (Williams, 2011).

The opportunity to evaluate an Aboriginal male well-being program, Be the Best You Can Be, which accompanies the Australian feature film Mad Bastards, saw the naming and refinement of Ngaa-bi-nya in 2012. A diverse, eight member Mad Bastards Working Group of experienced Aboriginal service providers from across Australia had several discussions about evaluation, and the need for tools that capture social, cultural, and economic influences on individuals and programs, which could be discerned from program influences (Williams, 2016b). This reflected work by Haswell, Blignault, Fitzpatrick, and Jackson Pulver (2013) on a set of Critical Success Factors (CSFs) from case studies with six Indigenous youth social and emotional well-being programs. In working with Haswell, Blignault, and Jackson Pulver on evaluation of three diverse post-prison release support programs around Australia for Aboriginal and Torres Strait Islander women, the Returning Home program (Haswell, Williams, Blignault, Grande, & Jackson Pulver, 2014), another iteration of Ngaa-bi-nya was developed. This clarified how Ngaa-bi-nya related to CSFs, with the purpose of Ngaa-bi-nya being an overall framework to guide evaluation, and prompting the user to select methods and data, rather than being a data collection tool in itself.

Returning Home findings pointed to a number of system-level factors that influenced program success; additional literature reviews to inform Ngaa-bi-nya further about these factors was subsequently undertaken, including about implementation science (such as Fixsen, & Blase, Naoom, & Wallace, 2009) and cost-effectiveness (Aboriginal and Torres Strait Islander Healing Foundation, 2014; Alford, 2014). These reflections, along with feedback from an international conference presentation (Williams, 2016a) and a statewide Aboriginal health workforce development session (Williams, 2017), led to the current version of Ngaa-bi-nya, with refined language, instructions, and prompts. Ngaa-bi-nya has most recently been used in a multi-disciplinary context for the Aboriginal-led #JustJustice social journalism project. This showed Ngaa-bi-nya’s appropriateness outside a direct service delivery context, helping ascertain the extent to which the project met its principles and objectives, and how the 90 articles it published reflected Aboriginal and Torres Strait Islander community principles and aspirations (Williams, Sweet, Finlay, McInerney, & Ward, 2017).

Principles underpinning Ngaa-bi-nya

Ngaa-bi-nya is underpinned by the principles espoused in guidelines for ethical conduct in Aboriginal and Torres Strait Islander research, including reciprocity, respect, equality, responsibility, survival and protection, and spirit and integrity (Australian Institute for Aboriginal and Torres Strait Islander Studies (AIATSIS), 2012; National Health and Medical Research Council, 2003). Ngaa-bi-nya aligns with:

• Aboriginal and Torres Strait Islander people’s view of health, which refers not just to an individual, but to “the social, emotional and cultural well-being of the whole community” (National Aboriginal Health Strategy Working Party, 1989, p. ix);

• priorities of the National Aboriginal and Torres Strait Islander Health Plan (Australian Department of Health and Ageing, 2013);

• the United Nations Declaration on the Rights of Indigenous Peoples, which asserts the right to self-determination, to strengthen cultural, social, and political life, free from threats of assimilation, and in accordance with diverse local needs and traditions (United Nations, 2007).

Ngaa-bi-nya prompts the user to take into account social determinants of health, which are “the conditions in which people are born, grow, work, live, and age, and the wider set of forces and systems shaping the conditions of daily life” (World Health Organisation, 2015, p. 1). These forces and systems include economic policies and systems, colonialism, development agendas, social norms, social policies, and political systems.

Ngaa-bi-nya acknowledges that the past affects the present, and that the present affects the future, and that the future cannot be shaped without a consideration of and reckoning with the past. It thereby acknowledges the need for an intergenerational perspective to program delivery, caregiving, and healing. Ngaa-bi-nya privileges Aboriginal and Torres Strait Islander people’s priorities, perspectives, and voices, given that programs are most successful when Aboriginal and Torres Strait Islander community members have power over governance, design, and delivery, including building capacity of community members to do so, aligned with cultural practices and values (e.g., Whiteside et al., 2016). This extends to conducting evaluation, and translating findings from evaluation.

Elements of Ngaa-bi-nya

The Ngaa-bi-nya evaluation framework has four domains—landscape, resources, ways of working, and learnings—which are shown in Figure 1. Prompts within each of these domains reflect statements of good practice and critical success factors among Aboriginal and Torres Strait Islander programs, and provide instrumental points of reflection for the conduct of evaluation. These can be used with other evaluation questions that ascertain program logic between aims, objectives, impacts and outcomes, theories of change, and cost-effectiveness (Patton, 2015). Ngaa-bi-nya’s prompts are to stimulate discussion, data collection, and analysis. Mixed quantitative and qualitative methods are recommended to collect relevant data and perspectives, and using a variety of each will allow the gathering of rich, culturally relevant material. These are best selected from the outset of programs and can inform program design; they can also be collected concurrent with or after program delivery.

Figure 1.

Ngaa-bi-nya evaluation framework.

Use of an Evaluation Reference Group is recommended, comprising a diverse range of Aboriginal and Torres Strait Islander service providers, community members, and other stakeholders. This group should help prepare for evaluation, guide the evaluation process, make choices of data, interpret results, and guide the reporting of findings and transfer of knowledge to a range of audiences. This group should also guide decisions about ownership and storage of data, its use in the future, and protocols for acknowledgment of data sources and authorship.

Prompt statements are provided in the following for each of the Ngaa-bi-nya domains, as are suggested data sources. Prompts are not intended as an exhaustive list, but to offer users a starting point to consider and inspire identification of further local factors. Prompts can be used, further developed, or omitted as relevant. They can be reworded into questions, or matched to a rating scale to help ascertain the extent to which programs meet desirable characteristics of Aboriginal and Torres Strait Islander programs. Many of the prompts can be used to assess and plan for the future growth and sustainability of programs.

Respecting the landscape

The landscape domain of Ngaa-bi-nya represents the broadest context a program is located in and influenced by. It is akin to a “system” level and the external influences identified in an ecological model of health (Bronfenbrenner, 1979), and signifies the whole-of-life perspective of the Aboriginal definition of health. Landscape factors influence the foundations and capacity of programs to deliver care, and therefore underpin other domains of Ngaa-bi-nya. Prompts to understand the landscape are outlined below; to understand landscape factors, data should be gathered and discussed about as many of the following items as possible:

History

Consider . . .
The history of the local area, and experiences of the traditional owners.
The history of program establishment.
The extent to which the program acknowledges the act and nature of colonisation, dispossession and disempowerment experienced by many Aboriginal and Torres Strait Islander people, such as child removal, racism, exclusion, poverty, trauma, and poor health occurring across generations.

Environment

Consider . . .
Local Aboriginal and Torres Strait Islander population characteristics, including proportion, recent changes and comparisons to local, state, and national proportions.
The location’s socioeconomic position, housing affordability, education, and employment rates.
Differences and similarities between the lives of Aboriginal and Torres Strait Islander peoples compared to others in the local area.
Proximity to and accessibility of major health and social support services, as well as barriers to and enablers of this accessibility.

Programs and services

Consider . . .
Other programs that influence the program, its resources, services, and accessibility.
Service-level collaborations and shared-care arrangements with other programs and services.
The extent to which roles, responsibilities and expectations across related organisations and sectors are articulated and realised.
The extent to which coordinated mechanisms for sharing information, resources and responsibilities occur with other related programs and services.
Investment in relevant infrastructure to meet need and serve future generations, such as building and using new technologies, service delivery guidelines, and workforce development and supervision strategies.

Self-determination

Consider . . .
The extent to which local Aboriginal and Torres Strait Islander people have been engaged in identifying needs and setting priorities.
The role of Aboriginal and Torres Strait Islander people in program design, delivery, and governance.
The role and influence of Aboriginal and Torres Strait Islander people in research, monitoring, and evaluation related to the program.
The extent to which Aboriginal and Torres Strait Islander people’s ways of relating, caregiving, and doing business are embedded throughout.

Policy

Consider . . .
The alignment between legislation and policies in relation to the issues addressed by the program, and any shifts in that alignment.
The role Aboriginal and Torres Strait Islander people have in policy development and reviews that impact on the program.
The extent of intergovernmental and intersectoral collaboration to support the program.
The impact of quality assurance and accreditation processes on the program.

Possible sources of data about landscape factors include population- and service-level data, books and media, research publications, service delivery reports, program websites, and interviews with program participants, staff, informal supports, key stakeholders, and community members.

Resources

Usually, physical and financial resources only are counted in program evaluations, which means that a wide range of other resources are often overlooked. The resources domain of the Ngaa-bi-nya framework seeks to identify the human, material, non-material, and in-kind resources, and informal economies and relationships that often support Aboriginal and Torres Strait Islander programs.

Financial resources

Consider the adequacy of financial resources to . . .
Meet the demand for services and support, and the needs of the local Aboriginal and Torres Strait Islander people.
Support Aboriginal and Torres Strait Islander workforce development.
Support program monitoring and evaluation and participation in research.
Support transfer of knowledge and policy advocacy.

Human resources

Consider . . .
How program development processes draw on local Aboriginal and Torres Strait Islander people’s knowledge and resources.
The mix of skills, experience and roles among program staff, support staff, and volunteers.
Networks that support the program, and how they are used.
The in-kind contributions, volunteer community participation, and informal supports drawn into the program, and their roles and outputs.
The types of culturally relevant training and supervision that program staff, board members, and other significant support people undertake and provide to others.

Material resources

Consider . . .
The use of data and other evidence to inform program design and development.
The extent to which equipment and information technology are sufficient to meet program needs, with growth needs identified.
The extent to which the physical location is suitable to meet program needs and future growth.
Plans, theories, stakeholders, mapping, investments and relevant community and political contributions that have been made for the program and its sustainability.

Possible sources of data about resources include service agreements, strategic plans, service delivery record-keeping, budgets, income and expenditure statements, service and staff reports, relationship mapping, media, as well as interviews with program participants, staff, informal supports, key stakeholders, and community members.

Ways of working

In this domain, the focus is on the delivery of programs and understanding the types of activities, relationships, frameworks, principles, and accountability mechanisms that support program delivery. Prompts reflect culturally safe, holistic processes through which Aboriginal and Torres Strait Islander people work to support others. Particularly drawn on here is the work of Bulman and Hayes (2011), Haswell et al. (2013), and Williams (2015).

Holistic caregiving principles

Consider the extent to which . . .
Caregiving is holistic, addressing multiple social, emotional, mental, environmental, spiritual, and physical elements of well-being.
Caregiving is provided beyond the individual person to families and across generations.
The program is sensitive to the impacts of trauma, developing opportunities for individual and collective healing.
The program addresses social determinants of health and well-being.
Intergovernmental and interdisciplinary partnerships support the program.
Actions are taken at workforce, community, services, and systems levels.

Quality caregiving in practice

Consider how program staff and other care providers . . .
Negotiate individual and group safety, shared agreements and boundaries with program participants, including reflecting on and enhancing these.
Are positive role models making progress in their own health care, relationships, and well-being.
Develop trusting relationships with program participants through which to influence change.
Are non-judgmental, ethical, responsive, and solution-focused.
Promote and achieve participant engagement in programs.
Celebrate achievements, remembering days of significance, anniversaries, and turning points.
Facilitate connection to Aboriginal and Torres Strait Islander culture and identity, through strengthening individual, group, and community connections.
Provide coordinated case management and wrap-around care, supported by partnerships, shared-care arrangements, and referral pathways with other services and supports.
Build capacity among participants through leadership, role-modeling, mentoring, and planning, including for knowledge and skills development, autonomy, and empowerment.
Create opportunities for program participants’ family and loved ones to feel comfortable, access support, and strengthen their capacity to provide support.
Provide timely continuity of care and follow-up.

Staff support and development

Consider . . .
Mechanisms for recognising and celebrating achievements by staff, volunteers, and governance committees.
The extent to which staff influence the development of the program, aligned with their experience and issues they witness in the community.
Whether the aim of the program is meaningful to and motivates staff, including through periods of change and difficulty.
The extent to which staff feel safe and supported to discuss challenges and make improvements.
Peer support opportunities among staff to share strategies and solutions, address challenges, and grow together.
The access that program staff, volunteers, and other support people have to education, training and skills development, and career pathways.

Sustainability

Consider . . .
The extent to which Aboriginal and Torres Strait Islander people experienced in relevant program delivery are involved.
The extent to which the program is self-determined by the Aboriginal and Torres Strait Islander community, and community controlled.
How timeframes and expectations of the program align.
The extent to which local community needs, resources available, actions taken, and broader policies align.
How change is managed at program and workforce levels, especially so that they are respectful of cultural protocols, history, and inputs over time, community needs, and likely characteristics of future generations.
The extent to which collaboration with other services and supports occurs, with agreements and reviews to support these.
The engagement of program staff, volunteers, and other supports in translating learnings about program successes to other contexts, including through presentations, networking, advisory roles, policy submissions, and media.

Evaluation

Consider . . .
The extent to which accountability, monitoring and evaluation processes are embedded in the program and resourced.
The extent to which culturally relevant data collection tools are used, and the involvement of Aboriginal and Torres Strait Islander people in leading evaluation design, data collection, analysis and reporting, and other knowledge exchange actions.
The impact of conducting evaluation and other quality assurance processes on the program.

Sources of data to best understand ways of working include service agreements, strategic plans, service delivery and staff reports, media, program webpages, presentations, networking, and individual and group interviews with program participants, staff, informal supports, key stakeholders, and community members.

Learnings

The learnings domain of Ngaa-bi-nya prompts users to reflect on insights gained and what the range of stakeholders and participants have learned, in addition to assessing the extent to which program objectives were met. This domain is to understand progress made, including in empowerment, attitude shifts, relationship strengthening, and self-determination of Aboriginal and Torres Strait Islander peoples. Learnings relate to movement—of ideas, of actions, of purpose, of ways of being, and of ways of relating. These are important Wiradjuri values, and are important to other Aboriginal peoples (Sheehan, 2004). Given that few tools are available to assess effectiveness of programs generally, of holistic care or of making progress, particularly from Aboriginal and Torres Strait Islander people’s perspectives, a dedicated process of critical reflection in dialogue with the Evaluation Reference Group is recommended.

Self-determination

Consider . . .
When, how and what aspects of self-determination and rights were or were not experienced, including informed leadership of local community members in priority-setting and planning, and community control in service delivery.
The commitments made to making progress as well as to outcomes, with progress expressed also through perseverance, despite challenges, constraints, and set-backs.
Whether non-Indigenous people undertake critical reflection on their standpoints, engage in cultural awareness and anti-racism training, or demonstrate commitment to strengthening personal and professional relationships with Aboriginal and Torres Strait Islander people.

Aboriginal and Torres Strait Islander cultural care

Consider . . .
The extent to which appropriate sharing of knowledge and processes strengthen Aboriginal and Torres Strait Islander people’s cultural identity and personal security.
The extent to which the program preserves Aboriginal and Torres Strait Islander cultural heritage.
The extent to which the program strengthens connection to country, waterways, knowledge, material items, and people.
The ways that taking responsibility for future generations are expressed by Aboriginal and Torres Strait Islander Elders, leaders and others involved in program delivery, including through role-modeling, mentoring, and knowledge exchange.
The opportunities for relationships to begin, or to strengthen, alter, cease, or otherwise transform.
The strength of ties, and how relationships are experienced such as through trust, integrity, equality, reciprocity, flexibility, and sharing.
The use of strengths-based program delivery and connecting people to supports they prioritise, rather than assessing for and identifying deficits or making comparisons to non-Indigenous peoples.

Healing

Consider . . .
The extent to which there are safe processes for reflection, access to therapeutic care, identifying inner strengths and addressing trauma, suffering, loss, and grieving in individual as well as group contexts.
The extent to which the program addresses experiences of racism.
The opportunities for greater understanding and empathy for others and strengthening relationships, including with self, older and younger generations, and people of other cultures.
The opportunities to enhance re/connections between Aboriginal and Torres Strait Islander peoples and cultural identity, knowledge, and practices.
Opportunities to support others as part of one’s own healing process.

Developing the evidence base

Consider . . .
How the program contributes to an evidence base by developing culturally relevant tools, methods, measures, indicators and benchmarking, as well as record-keeping and monitoring.
The opportunities for building the capacity of Aboriginal and Torres Strait Islander staff and community members in research, monitoring, and evaluation.
Strategies and processes for decolonise research, evaluation, and program delivery, including raising awareness among enabling systems, and mainstream researchers, partners, and stakeholders.

Possible sources of data for learning include service records and reports, routinely collected administrative and linked data, pre- and post-data collection, cost–benefit analyses, observations, field notes and case studies, and relationship and resource mapping. Critical reflection is recommended such as on adaptations of programs implemented elsewhere and change management, with feedback from the Evaluation Reference Group and other stakeholders, and individual and group interviews with program participants, staff, informal supports, key stakeholders, and community members.

Discussion

Health equity for Aboriginal and Torres Strait Islander people is becoming more difficult to achieve in many areas of life. Attempts to redress this inequity take the form of policies and programs that are, or should be, amenable to evaluation. However, few Aboriginal and Torres Strait Islander programs are evaluated and even fewer occur from Aboriginal and Torres Strait Islander people’s perspectives. Furthermore, evaluation that is not relevant to its broader context cannot truly evaluate what has been achieved in a policy or program (Kushner, 2016).

The Ngaa-bi-nya framework introduced in this article is designed to stimulate data collection and critical thinking about desirable characteristics of Aboriginal and Torres Strait Islander programs, including contextual landscape factors, diverse resources used, culturally relevant ways of working, and the range of learnings realised. Ngaa-bi-nya has been designed to be flexible to use, with prompt statements being open and adaptable. This reflects the reality that there are a great many differences among Aboriginal and Torres Strait Islander communities and contexts.

A particular strength of Ngaa-bi-nya is that it extends and informs the program logic and cost-effectiveness tasks of evaluation, but given its focus on identifying progress, relationships, and critical success factors, avoids a one-off judgment of a program as a “success” or “failure,” whereas extant methods of evaluation tend to focus on an assessment of outcomes. Instead, “failures” are seen as opportunities for learnings; difficulties are lessons and turning points to help make important future improvements.

Ngaa-bi-nya is also useful to plan for contextualised, culturally relevant measurement of outcomes. Such outcomes and their relationship to processes are best assessed by Aboriginal and Torres Strait Islander program providers and community members, or else such evaluation may be meaningless. The Growth and Empowerment Measure (GEM) (Haswell et al., 2010) and the Indigenous Risk Impact Screen in the drug and alcohol and mental health context (Ober, Dingle, Clavarino, Najman, Alati, & Heffernan, 2013) are two of only a small number of data collection tools developed by and validated for use among Aboriginal and Torres Strait Islander peoples to help measure outcomes, albeit focused only on individuals. A necessary task is to develop better tools for assessing what constitutes “success” in programs and how these are expressed through evidence, particularly from the points of view of Aboriginal and Torres Strait Islander people.

Like all evaluation tools and methodologies, Ngaa-bi-nya has its shortcomings. Evaluation is best when carried out by those who understand the culture of the program being evaluated; Ngaa-bi-nya recommends leadership by a local Evaluation Reference Group and facilitation of community input, including to select which of Ngaa-bi-nya’s prompts are relevant and what other factors and data need to be considered. There are tensions between some of the principles, such as those which refer to community consultation and ownership, yet those which refer to instead minimising the burden on communities (Katz et al., 2016, p. 38). Again, an Evaluation Reference Group can provide leadership in decision-making about this. There is an urgent need too to redress the reality that “research is a dirty word” among Aboriginal and Torres Strait Islander communities, and to “change the narrative” about research and evaluation, including through strengthening Aboriginal and Torres Strait Islander research leadership and workforce development (AIATSIS and The Lowitja Institute, 2017).

The Ngaa-bi-nya framework outlined in this article makes a contribution to evaluation by providing an example of Aboriginal leadership, building on research training and service delivery experience among Aboriginal and Torres Strait Islander people and through partnerships. It seeks to avoid a researcher-researched divide and to transform relationships and processes in order to bring about more Aboriginal and Torres Strait Islander people involved in evaluation and interpreting program successes in relevant ways—culturally, historically, socially, and economically, and through collective effort.