Abstract
Significant progress has been made in specifying sector or institutional learning outcomes and measuring their achievement in many countries, such as Canada, China, Germany, Russia, and the United States, to name a few (see Coates, 2014). However, Australian higher education appears to be falling behind internationally. Although there has been some success at defining learning outcomes at an institutional level, their assessment remains problematic. This is partly due to the large degree of autonomy Australian university teaching staff have in the design and assessment of courses; the lack of use of generalized testing programs, such as those used in other countries; and the common use of a bottom-up, subject-based model for specifying and assessing learning outcomes.
This paper describes research undertaken in Australia to specify a set of generalized learning outcomes for undergraduate courses at the national level. The research is twofold. First, it analyzes and synthesizes institutional-level statements made by Australian public universities on graduate capabilities. Second, based on the synthesis, a set of associated statements of standards for each of the learning outcomes is developed to form a basis of a national framework for their assessment. The objectives of the national framework are to enable criterion-based assessment to be used consistently to measure individual student attainment of these outcomes at different levels of competence, build comparability of local achievement assessment tasks and grades across a range of disciplines, and permit performance indicators to be established at discipline, course, and institution levels. The ideal of the framework is that the quality of teaching and learning of institutions is reliably and validly measured through assessment of students’ achievement of the stated learning outcomes.
The discussion in this paper is positioned in terms of the assessment transparency model (ATM) (Coates & Lennon, 2014) applied to the Australian environment. Subsequently, the paper proposes a framework for assessment of learning outcomes that could be implemented in Australia in order to move developments in the Australian higher education system from the “as-is” situation to one that is more ideal. The paper concludes with the opportunities and challenges in implementing such a framework.
The ATM
Internationally, there is a large number of actors and initiatives engaged in the assessment of learning outcomes, at varying stages of development. Harvested from the field of quality management (Burnstein, Suwanassart, & Carlson, 1996; Crosby, 1979), the ATM (Coates & Lennon, 2014) blends developmental and activity dimensions to evaluate activities pertaining to the assessment of higher education learning outcomes. A brief overview of the two-dimensional model follows, which is based on Coates and Lennon’s (2014) work. The model is shown in Figure 1.
Assessment transparency model.
Following Cerych and Sabatier (1986), the first dimension is
The first activity stage is
The Australian Context
The Australian higher education system comprises 53 self-accrediting providers, including 40 universities, 38 of which are public institutions; one university of specialization; two overseas universities; and 128 other non-university providers (Tertiary Education Quality and Standards Agency [TEQSA], 2016). Universities have self-accrediting status, that is, they can accredit all of their own courses without any further regulatory intervention, and 10 of the private higher education providers also have self-accrediting authority. Each university has its own establishment legislation (generally state and territory legislation except for the Australian National University), and the public universities receive the majority of their funding from the Australian government through the Higher Education Support Act 2003. As self-accrediting institutions, Australia’s universities have a high level of autonomy to operate within the legislative requirements associated with their Australian government funding. This means that they have been able to operate individually in relation to specifying learning outcomes for their graduates and also act independently about assessing those learning outcomes. This has led to the anarchic activity state identified in the ATM earlier.
Some of the significant forces that have had a substantial effect on Australian higher education and the learning outcomes of students include moving from elite to mass education over the past two decades, an emerging concern with the quality of education in the wake of massification, increasing demands for university accountability, diversification of higher education providers to include private providers and other “hybridized” forms (see also Coates & Mahat, 2014), and increasing shift of the financial burden of tuition to students, making higher education an investment from which graduates expect an employment and monetary return.
Learning Outcomes Policy Developments
These expectations resulted in a government requirement for all universities in Australia to develop and apply various sets of
More recent learning outcomes developments in Australian higher education can be traced to the impetus for reform provided by the national review of higher education conducted in 2008 (Bradley, Noonan, Nugent, & Scales, 2008). The review argued that for Australia to remain internationally competitive, it “must enhance its capacity to demonstrate outcomes and appropriate standards” through “more systematic processes . . . at both institutional and individual discipline level to provide stronger assurance of academic and organisational standards” (Bradley et al., 2008, p. 128). In response to the review, the Australian government committed itself to “new quality assurance arrangements involving the development of standards and implementation of a transparent process for assuring the quality of learning outcomes across all providers of higher education” (Commonwealth of Australia, 2009, p. 60). Explicit in the establishment of the TEQSA was the need for the development of “a set of indicators and instruments to directly assess and compare learning outcomes; and a set of formal standards along with processes for applying those standards” (Commonwealth of Australia, 2009, p. 61).
Additionally, the establishment of the Advancing Quality in Higher Education initiative prompted specification of a revised suite of key performance measures in learning and teaching based on some new student and employer surveys. These include the University Experience Survey (Radloff, Coates, James, & Krause, 2012), the Employer Satisfaction Survey (Oliver, Freeman, Young, Yu, & Verma, 2014), and a Graduate Outcomes Survey, developed by the Australian National University’s Social Research Centre. These are still surrogate measures of teaching and learning. What this initiative fell short of doing was to use forms of routine assessment to measure the actual levels of student attainment rather than just experience of the teaching and the educational environment.
As in many other higher education systems around the world, qualification frameworks have been one of the major expressions of expected learning outcomes for different levels of study. The Australian Qualifications Framework (2013) was first established in 1995 to map the education levels of each educational sector’s required knowledge and skills and incorporates explicit reference to “generic learning outcomes.” Following a review by the Australian Qualifications Framework Council (2010), the framework was revised with an enhanced architecture and updated with more coherent descriptors of learning outcomes. It is now mandatory for all Australian higher education providers to meet the requirements of the revised framework.
In terms of policy maturation—the first dimension of the ATM—the policy developments on assessing learning outcomes in Australia have achieved formulation stage but fall short of actual implementation. The Bradley review (Bradley et al., 2008) highlighted the issues surrounding quality and learning outcomes—identifying and defining the problem—and provided the impetus for policymakers and other stakeholders to advance the agenda within the higher education system. A number of these initiatives have not gained traction within the Australian higher education system in a comprehensive way. The responses of universities to the specification of learning outcomes are idiosyncratic and are not guided by a sectorwide view on what might represent a coherent set of learning outcomes, unlike a number of other countries. Also the government’s commitment to the development of a set of indicators and instruments to directly assess and compare learning outcomes has extended only to the measurement of student and graduate experiences and opinions. In addition, their commitment to performance funding for improvements in quality and student outcomes (Commonwealth of Australia, 2009) has not been fulfilled, largely because there has been no articulation of learning outcomes at a national level. This gap needs to be filled before success at assessment of achievement and institutional performance can be ascertained.
Current State of Learning Outcomes Specification and Assessment in Australia
In order to assess the current state of developments in Australia, the study first involved the development of a conceptual framework for specification of learning outcomes and standards for the sector and analysis of all universities’ statements of graduate attributes to identify commonalities and see whether it was possible to identify a comprehensive set of knowledge and skills appropriate to baccalaureate degree graduates. The analysis showed that institutional attributes were generally very broad, not distinguished by level of course or discipline, and much simpler than the detailed statements of learning outcomes in place in several other countries. The statements identified were grouped by a taxonomy of broad traits a graduate might be expected to have and a set of related standards formulated based on findings from the Australian Qualification Framework and work undertaken in a series of projects on graduate attributes and standards funded by the Office of Learning and Teaching. This analysis resulted in a large set of 38 generalized forms of these attributes, which formed the basis of further work on assessment and performance measurement that followed.
The next challenges were to explore the acceptability of this set of broad outcomes to individual universities and to identify the best way such outcomes could be assessed. The importance of aligning assessment with the outcomes and standards was stressed, and an attempt was made through two approaches to determine whether universities were adhering to these principles of assessment. First, a desktop review of Australian university websites was conducted to ascertain the policy framework in place in relation to assessment of learning outcomes in the universities. This review examined for each university, as a whole, policies and procedures about framing learning outcomes, their assessment, and the means of linking them together. The review of these websites revealed that the majority (66%) of the 38 public Australian universities for which website information was available specify general learning outcomes at the institutional level for all their graduates. In addition, according to their websites, half of the universities identify and specify learning outcomes at subject level, but it is unclear whether the subject outcomes have any direct relationship to of the specified course or institutional graduate outcomes. The review also suggested that less than a quarter of the 38 universities engage in any form of curriculum mapping of learning outcomes from subject to course. The simple analysis of high-level website content revealed that universities are stronger on the rhetoric of the need to specify graduate learning outcomes at several levels than the practice of actually doing so or assessing against the learning outcomes and demonstrating that students achieve them.
In order to obtain more information for individual institutions, a detailed data collection instrument (see appendix) was sent to the senior executive with learning and teaching responsibility in each Australian university. This was aimed at building up a sectorwide view of learning outcomes and assessment practices by seeking information about the acceptability and frequency of use of the 38 selected learning outcomes, identified through the conceptual framework and the nature of assessment practices used by the institution to measure student achievement of such outcomes.
The data collection instrument was completed by 19 of the 38 universities, which included representatives from each of the various subgroups found in Australian higher education (i.e., an elite research-oriented group, newer research intensive universities, regional and rural institutions, technologically based universities, and unaligned), as shown in Table 1.
Distribution of Respondent Universities to Data Collection by Type
A frequency analysis of responses to each of the questions asked was undertaken, which revealed that there is considerable agreement on the appropriateness of the 38 learning outcomes identified. At least 80% of respondents cited 15 learning outcomes that were common across institutions: discipline knowledge, discipline-related skills, learning collaboratively, participating as a team member, oral communication, written communication, recognizing cultural difference and diversity in work, professional ethics, learning independently in a self-directed manner, direction of own professional development, problem solving, critical thinking, using information and communication technologies to solve problems, ability to solve real-world problems, and international perspectives on the discipline. This suggests that the above subset could potentially be regarded as a minimal common set of undergraduate learning outcomes regardless of disciplines. Ten of these learning outcomes were cognitive, whereas the remaining five were affective in their nature.
Two thirds of respondents cited having university-wide learning outcomes but, with a few exceptions, do not appear to test student achievement against these in any holistic way. Those that do assess achievement against learning outcomes seem to conduct these assessments in two ways: either individual-subject learning outcomes in isolation or in limited alignment to course-, curriculum- and university-level learning outcomes. The main focus seems to be more on ensuring that curriculum and learning outcomes are aligned rather than on assessment being appropriate. Although criterion-based assessment was reported as commonly used among institutions, it appears that normalized grading schemes are also sometimes used. Universities often require conformity or standardization in their subject grading systems and assessment information stored for certification purposes according to their policy documentation. Although such an approach gives an impression of comparability of results, the validity of measurement is within the subject and not necessarily within the course or discipline.
Third, in Australia it seems that there is a narrow baseline of experience in universities for meaningful assessment of course-based learning outcomes even though there may be widespread effort in linking assessment and subject learning outcomes (Barrie, Hughes, Crisp, & Bennison, 2014). There is little evidence external to the institutions on whether the forms of assessment used to measure student attainment are well aligned with the institutional learning outcomes they specify. Using local achievement classroom-based assessment at a course level and linking this to the course learning outcomes and then to institutional performance measurement, as planned in this study, is a long way from what most higher education institutions in Australia are doing at present.
In the data collection, universities were asked how they ensured that their institution-wide learning outcomes were aligned with their curriculum and assessment in individual courses of study. Their responses are shown in Figure 2 and indicate that the most common approach is to map the curriculum and assessment tasks (probably at the subject level). In some fields, universities relied upon their own internal quality assurance processes or the requirements of accreditation in their professional degrees to ensure alignment between content or assessment and the learning outcomes.
Means of ensuring alignment between learning outcomes and assessment.
The percentage of respondents reporting use of whole-of-course assessment (as opposed to subject-based assessment) is higher than expected from the review of websites, but it is not clear from the responses what form this course-based assessment takes. In some cases the responses were also internally inconsistent with the responses given against the separate learning outcome statements in the data collection instrument. For example, the number of course learning outcomes stated as being used in the early part of the data collection ranged from three to 13 course-based learning outcomes, whereas most respondents ticked all of the 38 learning outcomes as being used in the institution either across the board or in at least some disciplines. Several respondents reported that because assessment occurred at a subject level, there were too many diverse approaches in the control of individual subject coordinators to be able to state in a general way what type of assessment was or could be best used across a course of study. Two universities provided some detailed statements of the type of assessment employed for each of their learning outcomes but made the point that this was in effect an “averaged” result across subjects and the reality was likely to be more diverse.
The two collections of information from the university websites and the data survey suggest that there has been a universal attempt from institutions to specify learning outcomes at various levels (subject, course, and institution). However, there is often a limited holistic institutional view of aligning the assessment and curriculum with learning outcomes. From a policy maturation perspective, this synchronizes with the developments (or lack of) at the national level. While most institutions have identified a ‘problem’ with measuring learning outcomes, little is done to actually implement the assessment of learning outcomes in considered ways. Using the Assessment Transparency Model, institutional developments on learning outcomes can be considered to be generally at the formulation stage.
Collaborative Assessment Ventures
In concert with the policy priorities at national and institutional levels, many individuals or groups have worked on a range of initiatives that attempt to respond to the challenges surrounding learning outcomes. From an activity perspective, developments in Australia have been quite anarchic. Since the adoption in 2009 by the government of the Bradley review recommendations, however, there has been a greater appreciation of the need to work on student learning outcomes. Funding was provided by the government to the then Australian Learning and Teaching Council and its successor, the Office for Learning and Teaching, to support research work on learning outcomes and academic standards. This resulted in a number of projects related to graduate attributes and learning outcomes and the development of standards across a range of disciplines, including engineering, chemistry, pharmacy, and creative and performing arts, which were referenced in this research.
One notable example of a more successful approach was the Australian Medical Assessment Collaboration (AMAC). Through grants provided by the Office for Learning and Teaching, the AMAC involved a collaborative group of medical schools working together toward demonstrating good practice in the development of common items for the assessment of learning outcomes (Edwards, 2014). AMAC was conceived as a way of improving the quality of medical education in Australia by recognizing the need for quality comparison measures, a sharing of expertise, and the acknowledgement of the need for high-quality assessment material (Wilkinson, Canny, Pearce, Coates, & Edwards, 2013). The central tenet of the collaboration was the idea of cooperation between schools—a “pooling” of resources—to improve assessment processes (Edwards, Wilkinson, Canny, Pearce, & Coates, 2014). On the ATM matrix, this initiative could be positioned as articulation in terms of activity and formulation in terms of maturity, but this is a rare example and in only one discipline.
The AMAC presented an exemplary practice of articulation, application, and amalgamation—involving educators and clinicians in medical schools who worked on item submission and review, engagement of students, and implementation of assessment items as well as wider engagement across the sector. Despite such exemplar practices, these learning outcome initiatives, for example, those funded by the government, have become a “thousand flowers blooming,” that is, disparate activities occurring individually or across a single or some disciplines with little cross-collaboration. Additionally, although there have been contributions to knowledge in the field (see examples of Barrie, 2005, 2006; Chalmers & Partridge, 2012; Jones, 2012), and serendipitous adoption or adaptation of research outcomes to other disciplines and institutions, there has been little, if any, systemic change within or between institutions or nationally.
The recent Organisation for Economic Co-operation and Development feasibility study, Assessment of Higher Education Learning Outcomes (AHELO; Coates & Richardson, 2011), was an ambitious example of a globalized assessment of knowledge and generic skills ability for the disciplines of engineering and economics at the bachelor degree level. The project followed the approach of defining learning outcomes and devising appropriate assessment instruments aligned with those outcomes but then tested the instruments extensively in the field. Its goal was very broad—to be able to measure student learning outcomes in a comparable way not only across institutions but across national boundaries—and to provide metrics that complemented routine internal assessments of student achievement.
In order to overcome anticipated opposition to the concept of generalizable assessment tasks across disciplines, Coates and Richardson (2011) addressed this goal by framing an engagement strategy that involved the academic communities (staff and students) from all participating institutions and building a community of scholars that could be consulted so that “philosophical, political and historical scepticism to assessment innovation” could be understood and addressed (p. 59). This approach embodies the higher-level activity categories of the ATM. Issues addressed included how to account for institutional diversity in curriculum, course structure and duration in comparative analysis of test results, the desire of participating institutions to use approaches such as normalized scoring of assessment results and unstandardized rubrics, and adapting non-uniform assessment instruments through an assessment framework so that results from their application might be valid and reliable across institutions and other domains (Richardson & Coates, 2014, p. 829). Funded and supported by the Australian government, Australia participated in the civil engineering strand of the feasibility study. Although the AHELO trial was ultimately not extended to implementation, there are some learnings from its approach, such as the need for a conceptual assessment framework to assist in the design of appropriate assessment tasks for each of the generalized learning outcomes identified. Edwards and Pearce (2014) conclude that the trial offered “useful insight into the development, engagement, implementation and reporting of large scale common assessments intended to provide measures of learning outcomes” (p. 85). The desire to measure common learning outcomes accurately across state or international borders as yet remains unfulfilled. And so, an approach that explores a new model of assessment of generalizable learning outcomes based on use of local achievement assessment and taking account of the issues experienced in the AHELO study appears worthy of further examination.
Although there has been a number of national and institutional policy developments that involved research, consultation, and reviews across the sector and institutions in Australia pertaining to learning outcomes, little actual implementation and adoption have taken place. Additionally, although there has been an increasing awareness and engagement on work on learning outcomes and their assessment through developing new and relevant discourse, generating resources and funding, and forming communities and training, learning outcome initiatives have remained sporadic. Linking the two dimensions of the ATM, it would seem that developments in formulating and assessing learning outcomes in Australia sit generally in the formulation/appreciation space (see Figure 1). From the model’s perspective, this analysis provides a helpful account of how to advance a learning outcome assessment framework for the Australian sector as well as address the challenges and opportunities required to move Australia through the continuum of development.
Finding the Holy Grail
The maturity model described above inevitably relies on the development, collection, dissemination, and implementation of assessment of learning outcomes in an environment where articulation is developed and embedded, amalgamation-level work is expected, and informed evaluation is fostered. At the time of the Bradley review, the tools for undertaking this were, at best, underprepared and narrowly focused or, at worst, nonexistent in Australia (Edwards & Pearce, 2014).
Drawing on international developments in learning outcomes, such as the Tuning process and degree profiles in Europe (Tuning Association, 2011), the Liberal Education and America’s Promise (LEAP) and Valid Assessment of Learning in Undergraduate Education (VALUE) initiatives by the American Association of Colleges and Universities (AAC&U; Rhodes & Finley, 2013a, 2013b), the Degree Qualifications Profile (DQP) of the Lumina Foundation (Ewell, Kuh, & Ikenberry, 2013), and generalization of the Tuning process across disciplines by the Higher Education Quality Council of Ontario (2013), Martin (2014) concludes that Australia is not as far advanced as other countries or systems in engaging a range of relevant stakeholders in formulating sectorwide learning outcomes for higher education in a generalized way. This is not for lack of endeavor or lack of government funding but rather due to a different focus and a resistance to the general concept. Australia has to date not adopted a nation- or state-wide set of graduate attributes or learning outcomes as there has been for the Tuning process, LEAP, or the DQP in the United States and Europe. Additionally, the application of the ATM to the Australian context has also shown that although there has been some work on generalized learning outcomes and standards for a range of disciplines, the outcomes have not yet been synthesized or used across disciplines or for the sector as a whole.
The review of international developments shows that it is possible through various methodologies to specify a comprehensive set of generalized learning outcomes for different course study levels
From the analysis of the Australian context through the ATM, two findings can be derived as a point of departure for the development of an assessment framework for Australia. First, the analysis showed that Australian universities rarely specify their graduate attributes, competencies, or learning outcomes at a discipline level. Consequently, in practice, there appears to be no differentiation by discipline between the groups of graduate learning outcomes that are to be measured. Second, when discipline-based research studies are undertaken independently, comparison of the research outputs seems to suggest that there are very few differences between the learning outcomes of different disciplines, their associated standards, and the broad range of assessment tasks that could lead to their measurement.
Based on these observations, a proposed framework for assessment of learning outcomes is advanced here, one that is characterized by lower diversity of assessment tasks but potentially high comparability between disciplines, courses, and institutions. The proposed assessment framework involves defining a small set of common but customizable assessment tasks at the subject level that links to a set of core cognitive learning outcomes through a constructive alignment process. From the analysis of the data collection of Australian universities’ assessment practices, the set of core cognitive learning outcomes could potentially include 10 outcomes that were identified by the majority of universities as being common across institutions and disciplines. The assessment tasks for each learning outcome would involve students finding solutions to real-world problems (i.e., authentic assessment tasks). Each assessment task would assess a common type of knowledge or skill identified through the core learning outcomes in the context of the discipline although not necessarily framed specifically for that discipline. The assessments would be marked and graded in accordance with the levels of achievement specified using a criterion-referenced approach embedded in the rubrics or standards to provide the basis of comparability and reliability of measurement. The tasks developed would be common to subjects contributing to the relevant learning outcome at the appropriate level and would be supplemented by other local achievement assessment tasks unique to the particular institution, thereby avoiding the tendency to uniformity of curriculum and overall assessment in the sector. Specification of cross-disciplinary assessment tasks would necessarily involve extensive consultation and collaboration between academic staff from a range of disciplines across the higher education sector (a form of moderation of the design process for assessment). From the AAC&U experience, such a collaborative approach has provided considerable supporting evidence for the reliability of a small set of similar “signature assignments” in measuring achievement (Banta, Griffin, Flateby, & Kahn, 2009; Rhodes & Finley, 2013b).
The framework proposed here for Australia starts with the specification of desired sector learning outcomes and the selection from those for an institution or course from this larger set of 38. This is then followed by an examination of how the particular learning outcomes might be assessed for individual students and then how these results might be aggregated or combined to populate a set of institutional performance indicators related to student learning. A review of past and current theory (see examples of Astin, 2012; Otter, 1992; Shavelson, 2007) and practice in the assessment of learning outcomes (see examples of Ewell et al., 2013; HEQCO, 2013; Rhodes & Finley, 2013a, 2013b; Tuning Association, 2011) has shown that such a framework is feasible. This provides an initial indication that such an approach may be applicable to advance the assessment of learning outcomes in the Australian higher education sector. However, given the tendency reported for Australian universities to focus on learning outcomes for subjects, the framework builds up a picture of evaluation of the achievement of university- or course-level learning outcomes through a process of constructive alignment between the higher-level outcomes and the curriculum and assessment of subjects that contribute to their attainment. This is done through a bottom-up approach, as shown in Figure 3.
A framework for the assessment of learning outcomes.
Challenges and Opportunities
The capacity to frame descriptions of learning outcomes in a way that enables feasibility of assessment and performance evaluation of institutions has been identified as the next big challenge by the Australian government. Although the current study proposes a possible way of assessing the 15 core learning outcomes identified earlier, it has several limitations. The first is that the approach has not yet been able to be adequately tested on real achievement data because of the current absence of such data in institutional data repositories.
Adoption of some of the proposals on standards and grading rubrics have potential to cut across the way assessment has traditionally been carried out in many universities. The main areas of contention relating to the model are the use of generalized approaches to both learning outcomes’ specification and their assessment. Similarities in expression and content of the learning outcomes used across disciplines in a set of research projects funded in the sector were highlighted by using a generic descriptor of
Some of the benefits that might be expected to flow to the sector from accurate measurement of achievement of learning outcomes include the demonstration of the value of higher education to stakeholders, such as students, governments, employers, and the public, by producing evidence and transparency about what students learn as a result of their study. An assessment framework, such as that proposed here, can demonstrate the necessary alignment between achievement of cognitive learning outcomes and the way they are assessed through the signature assignments. It builds on the approach of the AMAC project but extends this across disciplines. Enacting an assessment of learning outcomes in this way in Australia can also improve the quality of teaching by involving teachers and other university staff in the specification and measurement process for the selected course learning outcomes. Most importantly, it assists students to become internationally competitive in their professional roles. If such a framework is to be implemented in Australia, each of the key stakeholder groups in higher education would reap a number of benefits.
Students would
have clear information about what the expectations of their university are for their learning outcomes,
have improved learning experiences,
experience greater engagement in their study because of the explicit identification of both cognitive and affective learning outcomes as the outcomes they are expected to achieve, and
receive statements of graduate attainment that would state in greater detail what they actually know and can do by the end of their studies rather than just grades and subjects listed.
Teachers would
improve their teaching and assessment by having greater clarity about what needs to be taught to achieve the learning objectives of the courses they teach and how that curriculum needs to align with the nature of assessment used,
gain a greater sense of teacher engagement because of the need to work with colleagues across disciplines as well as within their own, and
assist students to become internationally competitive in their achievements.
Employers would
be able to compare the knowledge and skills of graduates from a range of institutions more easily, and
identify the broad level of achievement attained by potential employees framed in terms of their knowledge and skill set.
Governments and funding agencies would
ensure institutional accountability for funding of teaching and learning outcomes,
be able to examine relative institutional performance in teaching and learning measured by achievement of learning outcomes through benchmarking based on achievement of learning outcomes,
fund on outcomes rather than inputs or outputs, and
incentivize universities to improve their quality of teaching and learning.
Any systemic implementation of reforms is not without challenges. The assessment framework proposed here is a regime that requires much greater academic collaboration and consultation up front in the design phase about the nature of the tasks than at present. This may require agreement within the academic community to change the nature of the assessment tasks set for the subjects they teach. Such a collaborative approach would necessarily require social acuity, defining problems, building teams, and leading by examples (Coates & Lennon, 2014)—an influential expert or group of knowledgeable experts who can champion, support, and lead such collaborative initiatives. However, changing century-old assessment regimes within some institutions may face considerable resistance because of autonomous academics with deep-rooted beliefs and disciplinary cultures.
Any national assessment framework is likely to lead to views that the approach may lead to conformity between institutions and reduce diversity in assessment forms—an educationally undesirable outcome. This issue is a recognized problem but has been addressed to some degree in the United States and Canada in their implementation of the VALUE rubrics and the Tuning process so does not appear to be insurmountable. The concerns are addressed by teachers being able to use a sufficiently diverse range of assessment tasks, which includes the signature assignments where appropriate but also a number of other tasks that would be specific to the particular subjects and institutions. A danger of having a small number of grading categories in the set of standards and broad statements of attainment that are ultimately linked to institutional performance is that the grading scheme would potentially be “gameable” with institutions marking student assignments and placing a relatively large number of students in the top achievement category. Quality assurance methods would need to be applied with some spot auditing of the marking of these assignments to guard against this happening. The design of the institutional performance measures can also mitigate against this.
There are other issues that would also need to be addressed if such an approach were to be implemented. These include how the framework could be sustained over a period of years, including how the signature assignments might need to change to support changes in curriculum and how the signature assignments could be adjusted for teaching at different levels in various courses of study. The issues of institutions teaching to the test if the assignments were not varied from year to year and the level of administration and management required for such a system to work would also be critical to its success.
The costs associated with the implementation of a national assessment would be significant. These would include the development of a new component in the national student data collection, changes to individual universities’ data collection systems, collaboration costs associated with the development of new assessment tasks, and quality assurance processes to ensure the validity and reliability of any comparisons made between the achievement levels of individual students and collectively. In addition, the application of the framework within institutions may require a reshaping of curriculum and modes of assessment that would be more labor-intensive, at least in the beginning.
The challenges are extensive, but not implementing a nationwide framework for the assessment of learning outcomes in Australia would be detrimental for the system. The barriers identified to the implementation of a national assessment framework and consequently moving Australia through the activity dimensions of the ATM could be overcome if there was a strong commitment within the sector. Any change of approach to assessment policy would need to be driven by the government and supported by institutions. Academic communities of scholars across a comprehensive range of disciplines would need to be engaged in benchmarking assessment tasks and couching them in appropriate disciplinary knowledge. This would best be conducted based on a collective will and in a spirit of collaboration. Through continuous improvements and the use of best-practice examples of assessments, Australia has the potential to lead such systemic developments and, accordingly, guide the advancement of the field, particularly in the Asia Pacific region.