Sage Journals: Discover world-class research

Abstract

Recently, the employability of higher education graduates has been a topic of interest within many policy documents and scientific studies. Given the lack of comprehensive research identifying the implications of this issue for practice, this article presents a maturity model to support graduates’ early careers within higher education institutions (HEIs). The model is developed using a five-step approach, following the design science paradigm and exploiting four case studies across Europe. The final maturity model contains 65 practices within four key process areas—13 within strategic planning, 26 within curriculum design and delivery, 16 within student support, and 10 within the provision of extracurricular activities. In addition, a description of capability assessment criteria at the five maturity levels for each practice is provided. The model presented in this article can be used by HEIs as a tool to assess the maturity of their practices that contribute to graduates’ employability.

Keywords

higher education graduate employability career development maturity model development case study research design science research

Introduction

The concept of employability has been researched for years from both scientific and professional points of view. Therefore, different understandings and definitions of employability exist in the recent literature, as can be seen in the overview provided by Mcquaid and Lindsay (2005). In one of its simplest forms, employability can be understood as the probability of getting a job or the “ability to be employed” (Finn, 2000). A more extensive definition from the Confederation of British Industry (CBI) recognizes employability as “a set of attributes, skills and knowledge that all labour market participants should possess to ensure they have the capability of being effective in the workplace—to the benefit of themselves, their employer and the wider economy” (CBI, 2009, p. 8). In addition, different theoretical models on employability indicate that it is usually a combination of both individual and institutional factors that affect graduates’ employability potential (Dacre Pool & Sewell, 2007; European Commission, 2011, p. 4; Jain & Jain, 2013; Knight & Yorke, 2002).

In the context of this research, it is crucial to recognize that all of the definitions and theoretical models indicate the inevitable role of higher education institutions (HEIs) to prepare employable individuals. More concretely, Holmes (2013) confirmed this thesis by recognizing a double role that HEIs should play in relation to graduate employment. The first concerns their responsibility to follow what is happening with graduates going into employment through various graduate studies. The second involves taking steps to promote the likelihood that their graduates will gain appropriate employment. A recent study by Sin and Amaral (2017) also exemplified the important role that HEIs play in preparing graduates for their future careers in the labor market from the perspectives of both academics and employers. Generally, it might be said that the employability of higher education graduates is important for HEIs as it is used as one of the main indicators of higher education quality. Therefore, HEIs have tended to take responsibility for enhancing students’ employability (Bach et al., 2014; Cai, 2013). This is especially evident through the development of career services, student support services, academic services, and other services at HEIs that are aimed at enhancing graduates’ employability potential and their future success in the labor market. As employable graduates contribute to the knowledge economy and social growth, the task of producing highly skilled and employable graduates may also be recognized as a part of universities’ third mission. This third mission is usually understood as all the activities that are not covered within the missions of teaching and research, which are the primary missions of higher education. This third mission would include universities’ contributions to knowledge for social, cultural, and economic development (European Indicators and Ranking Methodology for University Third Mission, 2008), but also their commercial engagement and entrepreneurial role (e.g., Etzkowitz et al., 2000; Nelles & Vorley, 2010). In short, it might be said that universities’ third mission is related to their contributions to wider society in different forms.

The importance of the issue of the employability of graduates and the role of HEIs in that process can also be seen from strategy and policy documents, in which strengthening the potential employability of young people is set as a key development priority. A few years ago, Europe 2020 and related initiatives, such as Youth on the Move, indicated that the modernization of higher education and the preparation of graduates for the world of work within higher education should be emphasized to a large extent in the coming years (European Commission, 2010a, 2010b, 2010c). This was further supported by different initiatives focused on the development of European higher education (European Commission, 2015; European Commission/Education, Audiovisual and Culture Executive Agency/Eurydice, 2014). However, the European Commission recently published A New Skills Agenda for Europe (2016a) and a communication on “a renewed EU agenda for higher education” (2017), which showed that the situation has not changed much in the last years. Large gaps still exist between the labor market’s need for highly skilled professionals and the outcomes of higher education in the context of producing employable graduates. This problem is not only evident in Europe, but also worldwide, as can be seen from the initiatives presented by other countries, such as the Youth Employment Package aimed at assisting vulnerable young people in gaining employment in Australia (Australian Government, 2015) or the Youth Employment Strategy in Canada (Government of Canada, 2016). Governments’ endeavors on different levels support the thesis of the importance of HEIs to equip graduates with relevant and current skills, which requires systemic reforms in education and the modernization of higher education in collaboration with relevant stakeholders. The societal importance of the topic of graduate employability is confirmed with the research from Johnston (2003), which showed that much research in this area is funded by governments or official bodies, even though empirical research on the role of higher education in enhancing graduate employability is missing.

As higher education systems are very complex and dynamic, it was necessary to focus the research on a specific field of education to make it feasible to develop a useful and applicable maturity model. For several reasons, the focus within this research is on the education of future information and communication technologies (ICT) professionals. For the past years, Europe’s ICT sector has been the only segment to record a structural increase (European Commission, 2016c), and in many other countries around the world, the value added of the ICT sector is significant (Organisation for Economic Co-operation and Development, 2018). The ICT sector exhibits high importance in the overall economy, with a tendency to promote growth, and this implies a need for an increased number of skilled workers who will contribute to further growth (European Commission, 2016b; Gareis et al., 2014). Implications for HEIs are most evident from the data indicating that, among ICT specialist education, those with a higher level (tertiary) of education are sought after most (Eurostat, 2016). The rationale for the development of a maturity model in the ICT domain is given within a meta-analysis of scientific literature in the field of ICT education and career development that included 7,179 papers obtained from the initial database query (Pažur Aničić et al., 2017a). The authors concluded that there exist a “gap between the many papers that deal with issues of employability of ICT graduates and employers’ requirements and the considerably smaller number of papers that focus on possible ways to achieve better employability of ICT professionals” and therefore suggested that “a comprehensive theoretical framework should be developed to enable further structured research as well as to guide HEIs in designing curricula, services, and strategies for the employability of ICT graduates” (Pažur Aničić et al., 2017a, p. 196). Although the research will be focused on ICT, it is important to emphasize that the literature stresses that, contrary to the traditional meaning, careers do not depend to a great extent on the specific field of study (Teichler, 2009, p. 17). This favors the fact that the maturity model developed in the frame of this research will be, with minor modifications, applicable to other study fields as well.

As employability is a complex concept, with different meanings that interest groups (policy-makers, HEIs, academics, students, employers) attach to it, Sin and Neave (2016) described higher education as a “service provider” and “the only one vehicle” that improves individual employment prospects. They claimed that, “the power to define what constitutes employability determines the framework, which, in turn, shapes the operational activities of higher education institutions” (Sin & Neave, 2016, p. 1448). In this respect, the overall objective of the research presented within this article is to explore and identify key higher education system elements aimed at supporting graduates’ early careers, including the perspectives of different stakeholders. Meanwhile, its specific objective is to develop a strategic framework with an accompanying maturity model to support graduates’ early careers, as a tool aimed at helping HEIs to improve their practices of contributing to the employability of graduates. This article is based on the research methodology and strategic framework/maturity model elements presented by Pažur Aničić and Divjak (2015).

Scope of the Research

This section explains in more detail the strategic framework and the maturity model as the main outcomes of this research. Generally, the strategic framework represents an “outline of main objectives and initiatives” (Rademakers, 2014, p. 22), while maturity models provide guidelines for organizations to increase their capabilities from an initial stage of maturity through several stages until the desired end stage of maturity is reached (Mettler, 2010; Mettler & Rohner, 2009; Pöppelbuß & Röglinger, 2011). The basic purpose of maturity models is to describe levels and maturation paths. They are seen as both models that provide an answer to the question, “What are the elements of an ideal solution?” in the form of state descriptions, that is, the maturity levels; and methods, which include the activities to answer the question “How can the ideal solution be achieved?” (Mettler, 2010; Mettler & Rohner, 2009). As such, a maturity model serves as a useful benchmarking and managerial tool.

The literature on maturity models focuses mostly on the area of software process improvements (Paulk, Curtis, et al., 1993; Paulk, Weber, et al., 1993), but it is also evident in business areas such as project management, IT management (Andersen & Jessen, 2003), knowledge management (Kulkarni & Freeze, 2004), and business process management. The application of maturity models in the context of educational organizations may also be found, such as the complex elearning maturity model (Marshall & Mitchell, 2002). However, the review of educational maturity models provided by Duarte and Martins (2013) recognized nine educational maturity models, none of which were focused on graduate employability. In this article, the logic of maturity models is applied to HEIs, with the focus on their role in preparing graduates for their future early careers. As reasons for using maturity models, some authors (Mettler & Rohner, 2009) stress the pressure on organizations to retain a competitive advantage, reduce costs, improve the quality of products or services, and so on. In terms of higher education, key strategies and policies at the EU and national levels indicate that HEIs are under pressure to educate graduates who will be employable after graduation. This was recognized by Pažur Aničić and Divjak (2015), who proposed the connection between a strategic framework and a maturity model, and described the strategic framework as a basis for the development of a maturity model. The final strategic framework to support higher education graduates’ early careers presented in the scope of this article contains three main elements: key process areas, practices, and capability dimensions; while the maturity model additionally contains five maturity levels (capability assessment criteria) for each of the practices, further described within the “Research Method” section. The basic structure of the final maturity model is shown in Figure 1, which exemplifies four practices from the final model.

Figure 1.

The basic structure of the final maturity model.

Finally, along with the presented research objectives, three research questions related to the development of the final maturity model guide this research:

Research Question 1: Which are the key higher education systems’ determinants having a major impact on the preparation of higher education graduates for their early careers?

Research Question 2: Which are the key HEIs’ practices having impact to the preparation of higher education graduates for their early careers?

Research Question 3: Which are the capability assessment criteria of key HEIs’ practices having impact to the preparation of higher education graduates for their early careers?

Research Method

To provide answers to the set research questions, the research was carried out in several steps involving predominantly qualitative research methods, and to a lesser extent, quantitative methods, thus characterizing it as mixed method research (Creswell, 2009). The research follows the design science paradigm, which considers both the rigor and the relevance of the research in connection with real-world problems (Hevner et al., 2004; Vaishnavi & Kuechler, 2008), because it was found to be the common approach for the design of maturity models. According to the definition, design science is “a problem solving paradigm that involves building and evaluating innovative artifacts in a rigorous manner to solve complex, real-world problems, make research contributions that extend the boundaries of what is already known, and communicate the results to appropriate audiences” (Carcary, 2011, p. 109). Design science guidelines propose three main cycles (Hevner et al., 2004): (a) the relevance cycle, which presents connections to the real-world environment; (b) the rigor cycle, which is based on the use of knowledge sources; and (c) the design cycle, which represents a cycle of creating and evaluating artifacts until they work well for the studied problem. Consequently, although the research is primarily research-driven, it also includes policy aspects and practices through the application of a final model to several HEIs, thereby covering all three bases for higher education research: research, policy, and practice (Johnston, 2003). Following design science principles, the research was carried out through the adopted five-step methodology for maturity model development proposed by Mettler (2010) and initially presented by Pažur Aničić and Divjak (2015): (a) identify a need or new opportunity, (b) define the scope, (c) design the model, (d) evaluate the design, and (e) reflect the evolution. The research hodogram in Figure 2 shows the research methods used in each step of the maturity model design, indicating whether a certain method belongs to the rigor or relevance cycle, as well as the research output of each phase.

Figure 2.

Research hodogram.

Identify a Need or New Opportunity

In the first research step, a systematic literature review of research papers related to the education and early career development of future ICT graduates was conducted based on the results from five databases: ACM Digital Library, IEEE Xplore Digital Library, Scopus, Web of Science (WoS), and ScienceDirect. According to the set database query, 7,179 papers were obtained and read by title, 761 were analyzed at the abstract level, and 155 were examined in depth. Based on the findings from the content, cluster, and social network analysis (Pažur Aničić et al., 2017a, 2017b), a need for a more holistic and strategic approach to the education of future ICT professionals was identified, including career development support within formal processes of higher education. In turn, this confirmed the authors’ assumption about the need for the development of the proposed maturity model. Besides the systematic literature review of research papers, the authors conducted research into relevant strategic documents and projects, as well as the literature on maturity model development (Carcary, 2011; Marshall, 2007; Mettler & Rohner, 2009; Paulk, Curtis, et al., 1993; Paulk, Weber, et al., 1993), which helped to determine the basic maturity model elements, described as follows:

Key process areas: Building blocks indicating the main areas that institutions should focus on to improve their support for enhancing the employability of its graduates.

Practices: Each key process area is divided into several processes, containing different practices that describe activities or infrastructures within certain key process areas that together contribute to the achievement of key process area goals.

Dimensions of capability: Practices are organized according to dimensions of capabilities that address the level of implementation of a certain practice in the form of the Deming Plan-Do-Check-Act (PDCA) cycle, with four phases that are repeated iteratively—planning (plan), implementation (do), evaluation and control of the implementation of a specific practice and its effects (check) or the elaboration of new ideas for the improvement of practice in the next planning cycle (act).

Capability assessment criteria: Each practice at each dimension of capability has a defined maturity level or capability assessment criteria that indicate process capability. The capability assessment criteria are proposed on a kind of Likert-type scale with defined values, such as (adapted from Marshall, 2007) Not assessed (there is no evidence of the given practice at the HEI); Initial (there is some evidence of the given practice at the HEI, but this performance can be mostly characterized as ad hoc, informal and inconsistent); Partially adequate (the given practice is formally defined within the HEI to some degree, but it still lacks consistency in performance); Largely adequate (the given practice is formally defined within the HEI, and there is consistency in performance, but the institution is lacking suggestions for potential improvements in the given practice in the future); and Fully adequate (the given practice is formally defined within the HEI, the institution is consistent in its performance and there is documented reflection on the usage of the practice for further improvements). It means that each assessment criterion is evaluated according to a scale of defined performance standards.

In Pažur Aničić and Divjak (2015), the basic maturity model elements are described in more detail.

Define the Scope

After identifying a need for the development of a model that will contribute to better preparing graduates for their transition to the labor market, it was necessary to define the scope of the desired maturity model to focus further research on important elements (Eisenhardt, 1989). To that end, the conducted content analysis of different higher education strategies and related documents resulted in the identification of the initial key process areas and an initial list of practices. The key process areas were then confirmed through a focus group analysis (Cohen et al., 2011; Silverman, 2014), with two main aims: to determine the HEIs’ most important key areas in preparing students for their early careers and to detect the key persons at HEIs who could help with determining all the practices contributing to students’ employability. The focus groups involved experts with extensive experience at leading HEIs and universities, as well as different stakeholders within the higher education system.¹ This second research step resulted in the definitions of the following four key process areas: strategic planning of graduates’ employability, curriculum design and development, student support, and extracurricular activities, which present an answer to the first research question, “Which are the key higher education systems’ determinants having a major impact on the preparation of higher education graduates for their early careers?”

Design the Model

Case study research was the predominant method used in the third step of maturity model development. The goal of the case study research was to gather information from HEIs regarding their practices that foster the employability of graduates according to the four identified key process areas. The case study research was conducted at four HEIs in Europe: Vienna University of Economics and Business, The University of Belgrade’s Faculty of Organizational Sciences, University West in Sweden, and The University of Edinburgh. Studied cases cover four European regions: western, north, middle, and southeast Europe, which are all characterized by some specificities in educational systems. Cases also include an institution with a several-centuries-long tradition, two institutions with around 35-years long tradition, as well as one relatively young university. The institutions also differ in their size, expressed in the number of students and employees. Another characteristic that was considered is (de)centralization of processes and activities—the cases cover both the institutions with most of the centralized processes, as well as those with complete decentralization on the faculty level. Different characteristics of HEIs chosen as cases helped researcher to get insight in different organization of practices and help to further develop model that is applicable to different types of HEIs. In line with the methodology proposed by Yin (2014), it was carried out according to the following five phases: (a) design phase, (b) preparation phase, (c) collection phase, (d) data analysis, and (e) results sharing. From previous case study research, it was concluded that between four and 10 cases usually works well, while less than four cases make it difficult to generate theory (Eisenhardt, 1989). This research can be characterized as multiple-case design (Yin 2014) because it includes cases of four different HEIs in four different countries, where each country represents a different higher education system. The cases also differ in their size, mission, and organizational structure, which contribute to the identification of various examples of good practices within the four main key process areas. The case study research itself included semi-structured interviews with altogether 27 key players at the HEIs. They included vice-rectors (or equivalent) for students and study programs, teaching staff, students, directors of career support and student counseling centers and councilors in such centers, quality assurance experts, and other key persons who could provide relevant information on HEI processes that concern the career development of graduates. In the process of data analysis, several coding methods were used (Saldaña, 2013), both manually and supported by NVivo Computer-Assisted Qualitative Data Analysis Software (CAQDAS). The results from the case study research were amended with information from employers and other relevant stakeholders, collected through the focus groups.² This phase resulted in 110 practices within four key process areas, representing the initial strategic framework for supporting higher education graduates’ early careers within HEIs (Pažur Aničić & Divjak, 2016), which provides an answer to the second research question, “Which are the key HEIs’ practices having impact to the preparation of higher education graduates for their early careers?” Using the capability assessment criteria, this framework was further evaluated and amended to form the final maturity model.

Evaluate the Design

As maturity models are widely used in process improvement, their evaluation is an important activity to provide users with a confident guide that will help them identify potential improvements in a particular process. In this research step, the goal is to ensure the validity and reliability of the maturity model (Cohen et al., 2011; Merriam & Tisdell, 2015). To ensure content validity, 22 experts and 12 students evaluated the importance of 110 recognized practices. The term “experts” in this context refers to different higher education stakeholders, including higher education managers, teaching staff, non-teaching staff, alumni, and employers, as well as representatives of governing and supporting institutions. A group of students from different study years at the University of Zagreb, Faculty of Organization and Informatics, was chosen based on their academic achievement and student activity because it was important that students evaluating the model are familiar with different HEI practices related to four key process areas. Experts and students were asked to score 110 practices from the initial strategic framework using the scale “0—cannot answer, 1—not relevant, 2—important (but not essential), and 3—essential.” From the data obtained, the modified content validity ratio (CVR) was calculated for each item using a modified Lawshe’s (1975) formulation: CVR = (n − N/2) / (N/2), where n is the frequency count of the number of panelists who rated the item as either “2—important or 3—essential” and N is the total number of respondents. To develop a maturity model that would contain an acceptable number of practices applicable in practice, additional calculations were used to determine the most important practices as well as those practices that should be excluded from further development of the final maturity model. In the first round, all practices with a CVR < 0.7 were excluded from further analysis. In the second round, all the practices with an Average < 2.2 were excluded, calculated according to the following formula: $Average = (2 * \sum_{i = 1}^{22} {Score}_{E_{i}} + \sum_{k = 1}^{12} {Score}_{S_{k}}) / (2 * n_{E} + n_{S}),$ where ${Score}_{E}$ represents the score of each expert, ${Score}_{S}$ represents the score of each student, $n_{E}$ is the total number of experts, and $n_{S}$ is the total number of students. From the formula, it is evident that the experts’ evaluations are given double the weight of those of the students. Finally, all the practices with an Average < 2.34 and the results of the focus groups³ ≤3.5 were excluded. The final list of practices was also revised based on the qualitative comments of the experts, resulting in the retention of 68 practices. The overall obtained content validity index (CVI) for this shortened scale of practices was 0.903.

To assess construct validity, the card sorting technique, also known as Q-sorting, was applied. The main idea of this method was to have a panel of experts allocate each of the recognized practices into one of the four main proposed key process areas. All 22 experts from the group of those evaluating content validity participated in the Q-sorting. After removing the outliers, based on the calculations of Hamming distances, evaluations from 14 experts were taken into consideration. Experts rated the practices and sorted them into the four key process areas or the category Other if, in their opinion, the practice did not fit any of the predefined key process areas. The obtained Fleiss’ Kappa value of 0.7123 for 14 experts and 68 practices shows that the model can be considered good, approximating excellent (in Gwet, 2010, p. 113). This was confirmed by the calculation of a hit ratio of 98% for the practices within the key process area strategic planning, 89% for practices within curriculum design and delivery, 85% for practices within student support, 77% for practices within extracurricular activities, and 88% for the overall model.

To ensure the model’s reliability, which is a synonym for dependability, consistency, and replicability over time, instruments and groups of respondents (Cohen et al., 2011, p. 199), the model was evaluated by applying it to four real cases. Again, the chosen HEIs all educate students in the field of ICT, and due to research project budget constraints, all are in the Republic of Croatia: Josip Juraj Strossmayer University of Osijek, Faculty of Electrical Engineering, Computer Science and Information Technology Osijek; University of Zagreb, Faculty of Electrical Engineering and Computing; University of Zagreb, Faculty of Organization and Informatics; and University of Applied Sciences VERN. The evaluation was conducted through interviews with individuals who could provide deep insights into the maturity levels of certain practices at their institutions in the form of guided self-evaluation. The model showed itself to be applicable to institutions of different sizes and structures, and resulted in identifying the need to exclude three practices. Checking the model’s reliability based on other HEIs needs to be done in future research.

Finally, the Evaluate the design phase of the maturity model design resulted in the maturity model for supporting graduates’ early careers within HEIs. The final maturity model consists of 65 practices: 13 within strategic planning of graduates’ employability, 26 within curriculum design and delivery, 16 within student support, and 10 within extracurricular activities. In addition, each of them is described on five maturity levels. As maturity levels represent the capability assessment criteria, the final maturity model provides a comprehensive answer to the third research question, “Which are the capability assessment criteria of key HEIs’ practices having impact to the preparation of higher education graduates for their early careers?”

Reflect the Evolution

The evolutionary aspect of maturity model design is important because of the organizational development over time. In this particular case, because the student support system is changing and evolving, the maturity model should be redesigned accordingly. Changes may be triggered from the policy level (such as was with the start of Bologna process), driven by the labor market demands (i.e., need for the development of new study programs) or simply be the initiatives from HEI’s management (i.e., development of new activities within student support services, improved procedures for teachers’ education and evaluation of their work, etc.).

Results

The following chapter provides a short reflection on the research results within the framework of the presented research methodology. The main result of this comprehensive research is the development of a maturity model for supporting higher education graduates’ early careers. Each practice in model is assigned to one of the four phases of the Deming PDCA cycle (plan-do-check-act) and is described at five maturity levels. However, these general characteristics of the maturity levels cannot be used for all practices; instead, each practice requires unique descriptions of its maturity levels based on practice-specific characteristics, as shown in Figure 1. As the entire maturity model is too comprehensive to be presented within a research paper, only the final strategic framework (maturity model without descriptions of maturity levels) is presented in this section.⁴ This section further brings in a discussion about the main results according to four key process areas.

Strategic Planning of Graduate Employability

Among the 13 practices within the key process area of Strategic planning (Table 1), most of the practices fall within the implementation phase, which indicates the importance of the planning phase for the successful delivery of an employability strategy for graduates. One of the reasons why this phase is the most comprehensive is that it includes collecting information from all the relevant stakeholders, including alumni, students, the business sector, management, and supporting institutions. For the implementation phase, it is most important that the employability of graduates be addressed in institutional strategies once they are published and supported with action plans containing concrete actions that should be taken to implement the strategy successfully. The reports on the success of action plans are the most important asset in the evaluation phase. Finally, before starting a new process of strategic planning, it is crucial to analyze the success of the current strategy and check its compliance with changes in the internal or external environments, to provide recommendations for needed changes in the process of new strategy development. In this process, feedback from the students’ satisfaction survey can provide valuable information for a better understanding of strategy success.

Table 1.

Practices Within the Key Process Area of Strategic Planning.

Capability dimension	Practices related to Strategic planning
Plan	Procedures for (re)development of institutional strategies are defined.
	Information from graduate studies is used in the (re)development of institutional strategies.
	Inputs from students are included in the (re)development of institutional strategies.
	Inputs from relevant stakeholders are included in the (re)development of institutional strategies.
	Inputs from relevant HEI organizational units are included in the (re)development of institutional strategies.
Do	Employability of graduates is addressed in institutional strategies.
	Institutional strategies are communicated across the HEI.
	Institutional strategies are accompanied with action plans.
Check	Monitoring procedures regarding the implementation of institutional strategies are defined.
Check	Reports on success of action plans are collected from different organizational units.
Act	Feedback from student satisfaction survey is used for better understanding of strategy success.
	Compliance of strategy with changes in the external and internal environment is checked.
	Strategy success is analyzed before starting a new strategic planning process.

Note. HEI = higher education institution.

Curriculum Design and Delivery

Practices within the key process area of Curriculum design and delivery are presented in Table 2. This is the most comprehensive key area with 26 practices in the final maturity model. The planning and implementation phases contain detailed practices referring to main curriculum elements—learning outcomes, study content, teaching methods/pedagogic approaches, assessment methods, and teachers. In addition, practices within these two areas consider the business sector as the most relevant stakeholder in the process of curriculum design and delivery. This indicates a need for the curriculum to be based on relevant national and international frameworks and emphasizes the need for a student-centered approach and inclusion of employability skills incrementally across the whole curriculum. Practices within these two phases are also closely connected; for example, within the planning phase, it is important to establish formal procedures for student internships, while in the implementation phase, student internship should be included as an integral part of the curriculum. Practices within the evaluation phase are focused on the assessment of different elements, including the curriculum as a whole but also the assessment of the quality of students’ work, feedback on students’ internships, and evaluations of teachers’ performances. Similar to the key process area of Strategic planning, as a basis for starting a new process of curriculum design, it is necessary to check changes in the internal or external environments, ensure communication with the relevant stakeholders, and finally, discuss strengths and weaknesses of the current study program to be able to provide recommendations for potential improvements.

Table 2.

Practices Within the Key Process Area of Curriculum Design and Delivery.

Capability dimension	Practices related to Curriculum design and delivery
Plan	Procedures for curriculum design and development are established at the institutional level.
	Curriculum development is guided by a student-centered approach in which employability skills are considered to be interrelated and need to be developed incrementally across the curriculum.
	Inputs from employers are included in curriculum design and development process.
	Construction of learning outcomes is based on relevant national and international frameworks.
	Departments work together on the creation of coherent content for a certain study program.
	Formal procedures for student internship are established.
	Procedures and guidelines for the evaluation of teacher work are established.
	Continuous professional development of teachers is encouraged.
	Support for teachers to implement new technology (elearning) in courses is provided.
Do	Curriculum is delivered in partnership with industry.
	Interrelation between content of different courses is established.
	Teaching methods encourage independent, active learning and engagement with tasks (every student as a researcher/practitioner).
	Curriculum contains embedded work-based activities.
	Student internship is an integral part of the curriculum.
	Student theses are done in cooperation with the industry.
	Teaching and learning activities are taking place in an adequate working environment.
	Students are provided with proper feedback on their course performance.
Check	Curriculum relevance, consistency, practicality and effectiveness are monitored.
	Student feedback on curriculum implementation is collected.
	Quality of the assessment of student work is monitored.
	Feedback on internship is collected from students and employers.
	Teacher performance is evaluated by students.
	Control mechanisms based on the evaluation of teacher performance are established.
Act	Communication with the relevant stakeholders is done to ensure regular updates of learning outcomes in line with labor market needs.
	Strengths and weaknesses of the current study program are discussed prior to new curriculum design and development process.
	Results of teacher performances are discussed for potential improvements.

Student Support

Practices within the key process area of Student support (Table 3) are primarily focused on career services and related activities, so the planning phase considers the establishment of such services within an HEI and the preparation of its annual action plan. The set of different activities included in such a plan is evident from the implementation phase and covers all aspects of career-related activities: career information, career counseling, career education, and liaising with the business world (Lazić, 2012, pp. 6–7). The assessment and evaluation of career-service activities are key practices related to monitoring and improvement. In addition to career services, the practices within Student support also cover the areas of academic development, students’ personal development planning, and graduate tracking. From the previously described two areas, Strategic planning and Curriculum design and development, it can be seen that some practices are closely related to graduate tracking. Namely, information collected through graduate studies within this key process area is used as valuable input into the planning phase of both strategy development and curriculum design and development. This supports findings from previous studies on the role of HEIs in supporting graduate employability, which stress the importance of collecting data via graduate studies (Pavlin & Svetlik, 2014).

Table 3.

Practices Within the Key Process Area of Student Support.

Capability dimension	Practices related to Student support
Plan	Careers development service for students is established.
	Annual action plan for the careers development service is prepared.
	Graduate profiles for study programs are available to students entering the institution.
Do	Information on job opportunities and labor market is available to students.
	Career information is available to students through an organized career fair.
	Service of reviewing and correcting job application is offered to students.
	Simulation of job interviews is provided to students.
	Individual career counseling is offered to students.
	Information about graduate employment is collected.
	Students are encouraged and supported to create their own portfolio (personal development plan).
	Support for student academic development is provided.
	Mentorship for excellent students is provided.
	Support for students with disabilities is provided.
Check	Student feedback on the usefulness of career services provided is collected.
Check	Information on graduates’ satisfaction with their study and readiness for work is collected.
Act	Evaluation of career services activities is conducted.

Extracurricular Activities

Similar to the area of Student support, the key process area related to Extracurricular activities also contains mostly activities that are not obligatory neither for an institution to provide them nor for students to use them (Table 4). Within this area, practices are related to different extracurricular and co-curricular activities, such as participation in student competitions, students’ entrepreneurship initiatives, the work of different student organizations, and the organization of a student representative system. Within the planning phase, it is important for the HEI to recognize the contribution that such activities make to graduates’ employability and to establish a framework for their implementation, even though all of them are on a voluntary basis (Kinash et al., 2015, 2016). Similar to the practices within Student support, the implementation phase considers different activities that could be offered within an HEI, such as the work of different student organizations, support for students to participate in different competitions, and starting their own businesses. This support incorporates different aspects, including the provision of adequate working space for extracurricular activities, financial support, mentorship, additional education, connecting students with relevant experts outside the institution, and so on. In addition to monitoring the provision of different extracurricular activities, the evaluation phase also includes the recognition of students’ achievements in extracurricular activities, which was recognized as an important practice within some of the latest EU strategic documents (European Commission, 2016a).

Table 4.

Practices Within the Key Process Area of Extracurricular Activities.

Capability dimension	Practices related to Extracurricular activities
Plan	Framework for supporting student extracurricular activities is established (student organizations, student projects, participation in competitions, etc.).
	Framework for supporting student entrepreneurship initiatives is established.
	Institution has established system of student representatives.
Do	Students are provided with an adequate working environment for their extracurricular activities.
	Institution provides support for student start-ups.
	Students are involved in the work of student organizations.
	Students elect their representatives.
	Student representatives participate in different committees within the HEI.
Check	Student achievement in extracurricular activities is recognized.
Act	Potential improvements in support of student extracurricular activities are discussed at the institutional level.

Note. HEI = higher education institution.

Conclusion

This article presents the development of a comprehensive maturity model for supporting higher education graduates’ early careers within HEIs as applied to the field of ICT. In this last section, the authors would like to refer to the societal and scientific relevance of the presented research, as well as emphasize some limitations and implications for further work.

First, the scientific contribution is evident in the development of a comprehensive maturity model for supporting graduates’ early careers. This model contributes to the systematization and increase of knowledge in the field of higher education and the early career development of graduates, and summarizes all the relevant higher education practices to enhance graduates’ employability. As such, the final maturity model brings to defining a role of HEIs in supporting graduates employability, which was discussed from different authors (Cai, 2013; Holmes, 2013; Sin & Amaral, 2017; Sin & Neave, 2016). Moreover, it provides a framework that helps to define what constitutes employability from the HEIs perspective and shapes the operational activities of HEIs that should be taken to improve graduates’ employability potential, as was proposed by Sin and Neave (2016). In the context of bridging the employability gap in the ICT sector, this model responds to the needs for comprehensive theoretical framework “to enable further structured research as well as to guide HEIs in designing curricula, services, and strategies for the employability of ICT graduates,” as recognized by Pažur Aničić et al. (2017a, p. 196) based on a systematic literature review on education and career development of ICT graduates.

It also contributes to the general methodology for the design of maturity models by combining different research methods within the rigor and relevance cycles of the design science paradigm in a way that has not yet been proposed in previous researches. Moreover, the model contributes to the area of strategic planning and quality assurance within higher education for two main reasons: (a) those two areas are closely connected because quality assurance is among the most important tasks at the level of higher education governance and leadership, and (b) they are both characterized with the plan-do-check-act cycle, which is used as one of the main elements of the presented maturity model.

In addition to its scientific contribution, the results of the proposed research have significant practical implications for HEIs. These implications are evident with respect to the applicability of the developed maturity model to solve current problems and challenges in higher education, in the form of its guidelines for the design of higher education practices to support graduates in their early careers at the highest level of maturity. An example of potential model use is in the processes related to quality assurance in higher education, such as re-accreditation. An example of the re-accreditation of HEIs in Croatia shows the four-step procedure: (a) HEI drafts a self-evaluation report; (b) re-accreditation visit of an expert panel to the HEI; (c) the expert panel writes the report of the re-accreditation process, and the Agency for Science and Higher Education (ASHE) Accreditation Council provides their re-accreditation opinion; and (d) follow-up (ASHE, 2017). In the process of drafting self-evaluation reports, HEIs are provided with guidelines as well as the criteria for the assessment of the quality of HEIs. Maturity models, such as the one presented in this article, might be very useful as an addition to the self-evaluation guidelines to provide HEIs with clear guidance on what is expected within certain assessment criteria. Moreover, this maturity model relies on internal quality assurance standards, Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), which are used in the process of quality assurance in higher education (European Association for Quality Assurance in Higher Education et al., 2015).

One of the limitations of this research may be its focus on HEIs in the field of ICT. Another limitation lies in the number of institutions used for the case study research within the phases of both the model design and the model application. The reason for using four cases within the design phase was due primarily to the financial and time limitations of the thesis. As for all qualitative research, the constraint for this one is also that the data collection and analysis is to some point subjective and affected by the researcher’s skills and knowledge (Twining et al., 2017). Moreover, a large amount of qualitative and quantitative data that were analyzed in the scope of this research increases the possibility of errors in their manipulation and interpretation. To minimize these negative effects of a comprehensive qualitative research conducted by a single young researcher and ensure the high quality of research results, different methods were applied. Although this research included different stakeholders, their characteristics (expertise, experiences, etc.) certainly affected the research results to some point.

As the final step in maturity model design, Reflect the evolution, indicates, it is important to redesign the maturity model according to the changing nature of HEIs. One significant challenge for future work would be to apply the model to HEIs in different fields of study and environments. This would provide a good basis for a comparative study, resulting in implications for the creation of higher education and related policies at national and international levels for the early career development of higher education graduates.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research,authorship,and/or publication of this article: This work was supported by the European Union from the European Social Fund under Grant HR.3.2.01-0154 “Development of a model for supporting graduates’ early careers” and by the Croatian Science Foundation under Grant IP-2014-09-7854 “HigherDecision.” Those projects resulted with the PhD thesis that contains the research work presented in this paper.

ORCID iD

Katarina Pažur Aničić

Author Biographies

Katarina Pažur Aničić received her PhD from faculty of Organization and Informatics,University of Zagreb (FOI),Varaždin,Croatia in 2017. Now she is an assisant professor at the department for Information Systems Development and a Head of Student Support and Career Development Centre at FOI. Pažur Aničić was a recipient of the rector’s award for her student scientific work in 2010,the Summa cum Laude Honor for academic achievement in graduate study in 2011 as well as the dean’s award for a successful young scientist in 2017. Her research interest is related to graduate employability and career development within higher education institutions,with the emphasis on problem-based learning and work-based learning.

Blaženka Divjak received the PhD degree in mathematics from the University of Zagreb,Varaždin,Croatia,in 1998. Now she is on a position of a ministry of Science and Education of the Republic of Croatia. From 2014 to 2017 she was a full professor of mathematics and a scientific advisor in information sciences with the faculty of Organization and Informatics,University of Zagreb. Previously,from 2010 to 2014,she was a vice-rector for students,study programs,responsible for quality assurance system with the University of Zagreb. She has published over 60 research papers,about 20 professional papers,six university textbooks,and has presented at about 60 conferences,being an Invited Speaker at ten international conferences. She is an expert in higher education governance,e-learning,and learning analytics. She coordinated six international projects dealing with curriculum development,decision-making and quality assurance in higher education,and underrepresented groups in higher education.

References

Agency for Science and Higher Education. (2017). Re-accreditation of higher education institutions. https://www.azvo.hr/en/evaluations/evaluations-in-higher-education/re-accreditation-of-higher-education-institutions

Andersen

E. S.

Jessen

S. A.

(2003). Project maturity in organisations. International Journal of Project Management, 21(6), 457–461. https://doi.org/10.1016/S0263-7863(02)00088-1

Australian Government. (2015). Budget 2016—jobs & growth. Youth employment package. http://budget.gov.au/2016-17/content/glossies/jobs-growth/downloads/FS/Youth_Employment.pdf

Bach

Dragojević

Ð.

Findlay

Hering

Lauri

Lynch

Olcen

Udam

(2014). Transparency of European Higher Education through Public Quality Assurance Reports (EQArep) [Final report of the project, Kelo

Kerber

, Eds.]. https://enqa.eu/indirme/papers-and-reports/occasional-papers/Transparency%20of%20European%20higher%20education%20through%20public%20quality%20assurance%20reports%20(EQArep).pdf

Cai

(2013). Graduate employability: A conceptual framework for understanding employers’ perceptions. Higher Education, 65(4), 457–469. https://doi.org/10.1007/s10734-012-9556-x

Carcary

(2011). Design science research: The case of the IT capability maturity framework (IT CMF). The Electronic Journal of Business Research Methods, 9(2), 109–118.

Cohen

Manion

Morrison

(2011). Research methods in education (7th ed.). Routledge.

Confederation of British Industry. (2009). Future fit: Preparing graduates for the world of work. https://www.universitiesuk.ac.uk/policy-and-analysis/reports/Documents/2009/future-fit-preparing-graduates-for-the-world-of-work.PDF

Creswell

J. W.

(2009). Research design: Qualitative, quantitative, and mixed method approaches (3rd ed.). Sage.

10.

Dacre Pool

Sewell

(2007). The key to employability: Developing a practical model of graduate employability. Education + Training, 49(4), 277–289. https://doi.org/10.1108/00400910710754435

11.

Duarte

Martins

P. V.

(2013). A maturity model for higher education institutions. Journal of Spatial and Organizational Dynamics, 1(1), 25–45.

12.

Eisenhardt

K. M.

(1989). Building theories from case study research. The Academy of Management Review, 14(4), 532–550. https://doi.org/10.2307/258557

13.

Etzkowitz

Webster

Gebhardt

Terra

B. R. C.

(2000). The future of the university and the university of the future: Evolution of ivory tower to entrepreneurial paradigm. Research Policy, 29(2), 313–330. https://doi.org/10.1016/S0048-7333(99)00069-4

14.

European Association for Quality Assurance in Higher Education, European Students’ Union, European University Association, European Association of Institutions in Higher Education, Education International, BusinessEurope, & European Quality Assurance Register for Higher Education. (2015). Standards and guidelines for quality assurance in the European Higher Education Area (ESG). https://enqa.eu/wp-content/uploads/2015/11/ESG_2015.pdf

15.

European Commission. (2010a). An agenda for new skills and jobs: A European contribution towards full employment. http://eur-lex.europa.eu/resource.html?uri=cellar:776df18f-542f-48b8-9627-88aac6d3ede0.0003.03/DOC_1&format=PDF

16.

European Commission. (2010b). Europe 2020: A European strategy for smart, sustainable and inclusive growth. https://ec.europa.eu/eu2020/pdf/COMPLET%20EN%20BARROSO%20%20%20007%20-%20Europe%202020%20-%20EN%20version.pdf

17.

European Commission. (2010c). Youth on the move. Publications Office of the European Union. http://europa.eu/youthonthemove/docs/communication/youth-on-the-move_EN.pdf

18.

European Commission. (2011). Commission staff working paper on the development of benchmarks on education and training for employability and on learning mobility. http://csdle.lex.unict.it/Archive/LW/Data%20reports%20and%20studies/Reports%20and%20%20communication%20from%20EU%20Commission/20110525-040835_sec-670_2011_enpdf.pdf

19.

European Commission. (2015). New priorities for European cooperation in the field of education and training. http://eur-lex.europa.eu/legal-content/HR/TXT/PDF/?uri=CELEX:52015XG1215(02)&from=HR

20.

European Commission. (2016a). A new skills agenda for Europe: Working together to strengthen human capital, employability and competitiveness. https://ec.europa.eu/transparency/regdoc/rep/1/2016/EN/1-2016-381-EN-F1-1.PDF

21.

European Commission. (2016b). Employability and social inclusion. https://ec.europa.eu/jrc/en/research-topic/employability-and-social-inclusion

22.

European Commission. (2016c). Europe’s digital progress report 2016: The EU ICT sector and its R&D performance. https://ec.europa.eu/jrc/sites/jrcsh/files/ICT%20Sector%20and%20R&D%20performance%20in%20the%20EU%202016.pdf

23.

European Commission. (2017). Communication from the Commission to the European parliament, the Council, the European economic and social committee and the committee of the regions on a renewed EU agenda for higher education. http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52017DC0247&from=EN

24.

European Commission/Education, Audiovisual and Culture Executive Agency/Eurydice. (2014). Modernisation of higher education in Europe: Access, retention and employability 2014. http://commit.eucen.eu/sites/commit.eucen.eu/files/Eurydice_AccRetEmpl_May2014_165EN.pdf

25.

European Indicators and Ranking Methodology for University Third Mission. (2008). Needs and constraints analysis of the three dimensions of third mission activities. http://e3mproject.eu/Three-dim-third-mission-act.pdf

26.

Eurostat. (2016). Almost 8 million ICT specialists employed in the EU in 2014. http://ec.europa.eu/eurostat/documents/2995521/7141198/4-21012016-AP-EN.pdf

27.

Finn

(2000). From full employment to employability: A new deal for Britain’s unemployed? International Journal of Manpower, 21(5), 384–399. https://doi.org/10.1108/01437720010377693

28.

Gareis

Hüsing

Birov

Bludova

Schulz

Korte

W. B.

(2014). E-skills for jobs in Europe: Measuring progress and moving ahead [Report]. Empirica Gesellschaft fur Kommunikations- und Technologieforschung mbH.

29.

Government of Canada. (2016). Youth employment strategy. https://www.canada.ca/en/employment-social-development/services/funding/youth-employment-strategy.html

30.

Gwet

K. L.

(2010). Handbook of inter-rater reliability (2nd ed.). Advanced Analytics, LLC.

31.

Hevner

A. R.

March

S. T.

Park

Ram

(2004). Design science in information system research. MIS Quarterly, 28(1), 75–105. https://doi.org/10.2307/25148625

32.

Holmes

(2013). Competing perspectives on graduate employability: Possession, position or process? Studies in Higher Education, 38(4), 538–554. https://doi.org/10.1080/03075079.2011.587140

33.

Jain

(2013). Conceptualization, measure development and empirical assessment of career oriented attitudes and employability of technology graduates. Vision: The Journal of Business Perspective, 17(2), 143–157. https://doi.org/10.1177/0972262912483528

34.

Johnston

(2003). The shape of research in the field of higher education and graduate employment: Some issues. Studies in Higher Education, 28(4), 413–426. https://doi.org/10.1080/0307507032000122260

35.

Kinash

Crane

Judd

M.-M.

(2016). Good practice report: Nurturing graduate employability in higher education. Learning and Teaching Papers (Paper No. 136). https://eprints.usq.edu.au/33020/1/Kinash_Crane_Judd_2016_PV.pdf

36.

Kinash

Crane

Judd

M.-M.

Mitchell

McLean

Knight

Dowling

Schulz

(2015). Supporting graduate employability from generalist disciplines through employer and private institution collaboration [Final report]. Office for Learning and Teaching, Australian Government.

37.

Knight

P. T.

Yorke

(2002). Employability through the curriculum. Tertiary Education and Management, 8(4), 261–276. https://doi.org/10.1080/13583883.2002.9967084

38.

Kulkarni

Freeze

(2004, December 12–15). Development and validation of a knowledge management capability assessment model [Conference session]. 25th International Conference on Information Systems, Washington, DC, United States.

39.

Lawshe

C. W.

(1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x

40.

Lazić

Janković Barović

. (2012). Centres for career development and student counselling at Serbian Universities - Basic services. In Lazić

(Ed.), University centres for career development and student counselling: Career guidance at Universities in Serbia (pp. 5–7). University Centre for Career Development and Student Counselling, University of Belgrade.

41.

Marshall

(2007). E-learning maturity model (Version 2.3). [Process descriptions]. Victoria University of Wellington. http://www.utdc.vuw.ac.nz/research/emm/Publications.shtml

42.

Marshall

Mitchell

(2002, December 8–11). An e-learning maturity model? [Conference session]. 19th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education, Auckland, New Zealand.

43.

Mcquaid

R. W.

Lindsay

(2005). The concept of employability. Urban Studies, 42(2), 197–219. https://doi.org/10.1080/0042098042000316100

44.

Merriam

S. B.

Tisdell

E. J.

(2015). Qualitative research: A guide to design and implementation (4th ed.). Jossey-Bass.

45.

Mettler

(2010). Thinking in terms of design decisions when developing maturity models. International Journal of Strategic Decision Sciences, 1(4), 76–87. https://doi.org/10.4018/jsds.2010100105

46.

Mettler

Rohner

(2009, May 7–8). Situational maturity models as instrumental artifacts for organizational design [Conference session]. 4th International Conference on Design Science Research in Information Systems and Technology, Philadelphia, PA, United States. ACM Press. https://doi.org/10.1145/1555619.1555649

47.

Nelles

Vorley

(2010). Constructing an entrepreneurial architecture: An emergent framework for studying the contemporary university beyond the entrepreneurial turn. Innovative Higher Education, 35(3), 161–176. https://doi.org/10.1007/s10755-009-9130-3

48.

Organisation for Economic Co-operation and Development. (2018). Key ICT indicators: Value added of ICT sector and sub-sectors. http://www.oecd.org/internet/broadband/oecdkeyictindicators.htm

49.

Paulk

M. C.

Curtis

Chrissis

M. B.

Weber

C. V.

(1993). Capability maturity model for software (Version 1.1, Technical Report No., CMU/SEI-93-TR-024, ESC-TR-93-177). Carnegie Mellon University.

50.

Paulk

M. C.

Weber

C. V.

Garcia

S. M.

Chrissis

M. B. C.

Bush

(1993). Key Practices of the Capability Maturity Model Version 1.1., Technical Report, CMU/SEI-93-TR-025, ESC-TR-93-178, Carnegie Melon University.

51.

Pavlin

Svetlik

(2014). Employability of higher education graduates in Europe. International Journal of Manpower, 35(4), 418–424. https://doi.org/10.1108/IJM-05-2013-0104

52.

Pažur Aničić

Divjak

(2015, November 18–20). Development of strategic framework for supporting higher education graduates’ early careers [Conference session]. 8th International Conference of Education, Research and Innovation, Seville, Spain.

53.

Pažur Aničić

Divjak

(2016, May 29–31). Supporting student retention and employment capabilities within higher education institution: Four case studies research [Conference session]. 25th European Access Network Conference: Retrospective for Perspective: Access and Widening Participation 1991-2041, Dublin, Ireland.

54.

Pažur Aničić

Divjak

Arbanas

(2017a). Preparing ICT graduates for real-world challenges: Results of a meta-analysis. IEEE Transactions on Education, 60(3), 191–197.

55.

Pažur Aničić

Divjak

Arbanas

(2017b). Prestige and collaboration among researchers in the field of education and career development of ICT graduates: Is there a cross-fertilization of research and knowledge? Journal of Information and Organizational Sciences, 41(2), 231–262.

56.

Pöppelbuß

Röglinger

(2011, June 9–11). What makes a useful maturity model? A framework of general design principles for maturity models and its demonstration in business process management [Conference session]. European Conference on Information Systems Proceedings, Helsinki, Finland.

57.

Rademakers

M. F.

(2014). Corporate universities: Drivers of the learning organization. Routledge.

58.

Saldaña

(2013). The coding manual for qualitative researchers (2nd ed.). Sage.

59.

Silverman

(2014). Interpreting qualitative data (5th ed.). Sage.

60.

Sin

Amaral

(2017). Academics’ and employers’ perceptions about responsibilities for employability and their initiatives towards its development. Higher Education, 73(1), 97–111. https://doi.org/10.1007/s10734-016-0007-y

61.

Sin

Neave

(2016). Employability deconstructed: Perceptions of Bologna stakeholders. Studies in Higher Education, 41(8), 1447–1462. https://doi.org/10.1080/03075079.2014.977859

62.

Teichler

(2009). Higher education and the world of work: Conceptual frameworks, comparative perspectives, empirical findings. Sense Publishers.

63.

Twining

Heller

R. S.

Nussbaum

Tsai

C. C.

(2017). Some guidance on conducting and reporting qualitative studies. Computers & Education, 106, A1–A9.

64.

Vaishnavi

V. K.

Kuechler

Jr. (2007). Design science research methods and patterns: Innovating information and communication technology. Auerbach Publications.

65.

Yin

R. K.

(2014). Case study research: Design and method (5th ed.). Sage.