Sage Journals: Discover world-class research

Abstract

Deciding the proportion of online to face-to-face components of a blended learning course is a complex problem that entails many decisions. These decisions are affected by different criteria related to the nature of the course, the educational institution, the teacher, and the students. The impact of these influential criteria on the design process is not always obvious. The aim of this study is to assist academics to design successful blended courses by investigating the impact of criteria that need consideration when deciding the proportion of online components of a blended course. A modified Delphi survey was used in this study. Eighteen experts with experience in instructional design and online teaching participated in the study. Findings indicate that technological aspects, that is, Availability of technology and Students’ access to technology, have a strong negative impact on the proportion of online to face-to-face components. On the other hand, students’ flexibility and convenience as well as the availability of a high level of institutional support should motivate academics to incorporate a high proportion of online components in their courses. The results also indicate that blended learning courses should have medium to high proportions of online components.

Keywords

hybrid learning online learning Delphi method higher education blended learning

Introduction

Blending learning has emerged in an effort to enhance student learning experience whilst increasing their flexibility and convenience. The necessity of blended learning in higher education has become more obvious with the outbreak of the COVID-19 pandemic. An increasing number of researchers have stressed the need of adopting blended learning (Bennardo et al., 2020; Alammary et al., 2017; Finlay et al., 2022; Theoret & Ming, 2020). Many studies have reported that blended learning will become the normal approach for course delivery in the Post-COVID-19 era (Ehrlich et al., 2020; Jones & Sharma, 2020).

Developing a successful blended learning course is challenging. The literature shows that academics normally face several challenges when developing their blended courses. A major one is deciding the most appropriate proportion of online components that should be incorporated into their courses (Ashraf et al., 2021; Müller & Mildenberger, 2021; Oliver & Stallings, 2014). Reaching a harmonious balance between online and face-to-face components is vital for the success of blended learning courses. A harmonious balance can increase student engagement, provide them with more control over the time and place of their study and improve the overall student learning outcomes. In some courses, a higher proportion of online components can be more appropriate, while in others, a lower proportion can be more helpful (Müller & Mildenberger, 2021; Öncü, 2022). In our previous study (Alammary, Carbone et al., 2015), we identified 37 criteria that could influence the decision regarding the proportion of online to face-to-face components of blended courses. However, their impact on the design of blended learning courses has not been yet assessed. Therefore, this study aims to examine the impact of these criteria on the design process and on determining the proportion of online to face-to-face components of a blended course. More specifically, the study attempts to answer the following questions:

RQ1: How does each of the identified criteria affect the proportion of online to face-to-face components of a blended course?

RQ2: What is the most appropriate proportion of online to face-to-face components that should be incorporated into blended learning courses?

This study uses a two-round Delphi survey to assess the criteria and understand their impact on the proportion of online to face-to-face components of a blended learning course. The paper is organized as follows. Section 2 introduces the background of the research. Section 3 lists the criteria that have been assessed and describes the process that has been employed to select them. Section 4 provides a detailed description of how the Delphi method was used for the assessment. Section 5 presents the results that were obtained. Section 6 analyses and discusses the study’s findings. Section 7 explains the study’s limitations. Finally, Section 8 concludes the paper.

Research Background

There is continuing debate about the most appropriate definition of the concept of blended learning (Müller & Mildenberger, 2021). Picciano (2009) points out that “blended learning is not one thing but comes in many different flavors, styles, and applications; it means different things to different people.” According to Driscoll (2002), the confusion surrounding the term might appear to be a drawback. However, these multiple definitions also show the untapped potential of blended learning; Sharpe et al. (2006) argue that the ambiguity of the term blended learning allows teachers and instructional designers to develop their own understanding of the term within the context of their institutions or courses.

In this study, a blended learning course is defined broadly as “a course that: (i) thoughtfully integrates different delivery methods such as lectures, in-class discussion, online discussion, and self-paced activities and (ii) contains both face-to-face and online portions” (Alammary, 2022). A crucial point in this definition is that the final course should comprise a mixture of online and face-to-face components. This makes it encompasses a range of implementations from adding one single online activity to a face-to-face course to building a whole blended course from scratch with a variety of online activities.

Blended learning provides teachers with a wider choice of online and face-to-face components that can help them to develop solutions to course problems. These components support different types of interaction between learners, instructors, and content. Kearsley (1995) notes that interaction is a critical design factor for all types of courses, whether in the traditional, distance, or blended learning format. Bonk and Graham (2012) describe interaction as the “glue” that holds the components of a blended course together. Moreover, Wagner (2012) states that interaction “continues to be perceived as the defining attribute for quality and value” (p. 42) in the learning experience. Constructivist learning theory, the most influential theory impacting pedagogy and hence blended learning (Kirkley & Kirkley, 2005; Ogbona, 2021), suggests that learners “build personal interpretations of the world based on individual experiences and interactions” (Ertmer & Newby, 2008, p. 55).

Moore (1989) identifies three types of interaction: learner-instructor, learner-learner, and learner-content. Learner-content interaction is described by Moore (1989, p. 2) as “the process of intellectually interacting with content to bring about changes in the learner’s understanding, perspective, or cognitive structures.” Moore also points out that this type of interaction is a “defining characteristic of education.” According to constructivist theory, knowledge does not exist independent of the learner; it is constructed through interaction with the content (Piaget, 1977) or other individuals (Vygotsky, 1980). In constructivist settings, learners should have control over their own learning and should be provided with the resources, tools, and support necessary to manage their learning (Kwangmuang et al., 2021). Blended learning supports this type of interaction through online self-paced activities, for example, interactive videos, podcasts, and online reading. This type of activity allows students to study in their own time and at their own pace, from their own location (Kliziene et al., 2021).

Learner-instructor interaction attempts to “motivate and stimulate the learner and allows for clarification of any misunderstanding by the learner in regard to the content” (Moore, 1989). Cognitive development, according to Vygotsky (1980), begins with an interaction between the student and a more knowledgeable other. A number of studies (Iaconelli & Anderman, 2021; Liu, 2021) have found that a teacher’s nonverbal and verbal immediacy can increase students’ affective and cognitive learning. In blended learning, learner-instructor interaction is achieved through face-to-face instructor-led activities (e.g., face-to-face lectures and labs) and online instructor-led activities (e.g., virtual classrooms, webcasts, and scheduled internet instruction). These activities allow teachers to maintain control over their students’ learning experience and tailor teaching strategies accordingly.

Leaner-learner interaction, according to Moore (1989), occurs “between one learner and another learner, alone or in group settings, with or without the real-time presence of an instructor” (p. 4). Social constructivism, a parallel argument to constructivism, suggests that learning takes place because of students’ interactions in a group (Lave & Wenger, 1991; Vygotsky, 1980). Therefore, social constructivism stresses the need for student collaboration via approaches such as peer collaboration, problem-based instruction, or other methods that involve learning with others (Lombardo & Kantola, 2021; Woo & Reeves, 2007). In blended learning, collaboration occurs both online and offline through a variety of activities such as writing groups, peer teaching, online discussion, and online learning communities. This variety of activities available for instructors would allow them to better engage their students, help students to develop critical thinking, and help students to construct a deeper understanding of the material being learned (Lane et al., 2021).

To gain maximum benefit from blended learning, academics need to reach a harmonious balance between face-to-face and online components. A harmonious balance can help students better achieve the desired learning outcomes (Alammary, 2021). In some courses, a higher proportion of online components can be more appropriate, while in others a lower proportion can be more helpful (Hamann et al., 2021; Müller & Mildenberger, 2021). Setyaningsih (2020), for example, surveyed undergraduate students from three different higher education institutions in Indonesia. Most students expressed a strong preference for the face-to-face components of their blended learning courses. The two most frequently mentioned reasons for their preference were familiarity with the delivery mode and their ability to get direct clarification from their teachers.

Poon (2012) found different results when surveying a diverse group of students that included undergraduate, postgraduate, full-time, and part-time students. She noted that the majority of students were satisfied with the online components of their courses. Students reported that these components allowed them to better manage their own learning experience and learning pace. Poon also surveyed the academics who designed these blended courses. She found that institutional support (e.g., time release, technical support, funding, and professional development) was essential for successful online component integration. She also found that academics’ technological knowledge helped them in developing the online components.

In reviewing the literature, it was found that although the integration of the online components was driven by pedagogical needs, the proportion of online components seems arbitrary and not determined by certain criteria. Some courses were designed so they had more face-to-face than online components, while others tipped the balance in favor of online components. Other courses mixed the two modes of instruction somewhat equally.

One example is a first-year writing course taught at Brigham Young University. This course was redesigned so that students were required to attend one class a week instead of three (Waddoups et al., 2003). The two face-to-face classes were replaced by online lectures including explanations of difficult concepts. Students were also provided with online feedback and guidance on their writing. The integration of the online components was to improve student learning through using technology. However, no explanation was provided for how the proportion of online components was determined.

Another example is a third-year philosophy course which was originally delivered via two 90-minute seminar sessions per week (Garrison & Vaughan, 2011). This course was redesigned by reducing the number of face-to-face weekly sessions to one. An online portfolio system was introduced. Students were required to write short papers and upload them to the system. Then, they were asked to use a weblog feature in the system to provide reflective comments on their papers. They were also required to use a wiki feature of the system to comment on their peers’ papers. The integration of the online components was driven both by the students’ poor performance in the assignments and the fact that the majority of them were not reading the assigned articles before attending classes. However, again, no explanation was provided for how the proportion of online components was determined.

A third example is the blended courses taught to undergraduate and postgraduate students at the Saudi Electronic University. These courses were designed in the same format so they contained 33% face-to-face components and 67% online components. The decision regarding this proportion was made by the university. The main aim is to increase flexibility and convenience and to allow people with outside commitments such as family and work to continue their higher education (Saudi Electronic University, 2021).

In our previous study (Alammary, Carbone et al., 2015) we identified 37 criteria that need consideration when deciding the proportion of online to face-to-face components of a blended learning course. The study also rated the importance of these criteria and classified them into four categories: (i) criteria related to courses (11 criteria); (ii) criteria related to students (11 criteria); (iii) criteria related to teachers (8 criteria); and (iv) criteria related to educational institutions (7 criteria). The outcome of that study indicates that educational institutions play significant role in facilitating the integration of online components into the traditional face-to-face experience and helping academics to design successful blended learning courses.

To sum up, it is apparent from reviewing the literature that the appropriate proportion of online components can vary widely between courses. Our previous study shows that deciding this proportion is influenced by different criteria related to the nature of the course, the educational institution, the teacher, and the students. However, the impact of these criteria on the online proportion is not clear.

Criteria That Were Considered for Analysis

The 37 criteria that we found to influence the decision regarding the proportion of online components in a blended course were considered for analysis. Due to the large number of these criteria and due to the low importance rating of some of them, we decided to only analyze criteria that have achieved the highest importance ratings. The inclusion of 37 criteria in the assessment process would have resulted in a very long Delphi survey. According to Guthrie et al. (2022) and Keeney et al. (2001), the more statements in the questionnaire, the less likely experts will be to complete it.

To determine which criteria are the most important, a cut-off point was applied. Using a cut-off point is a common technique for determining Delphi consensus and carrying over statements from one Delphi round to the next (de Haan et al., 2022; von der Gracht, 2012). There are no recognized formulae or defined standards to guide the selection of cut-off points. They are normally determined based on the data obtained in the study (Cottam et al., 2004).

In this study, a mean score of 3.5 was set as the cut-off score. The selection of this cut-off score was arrived at in two steps. At first, and because of the 5-point ordinal scale (1 = very unimportant to 5 = very important) that has been used to rate the criteria, an initial cut-off score of 3.00 was chosen (Ahuja et al., 2009; Ugwude et al., 2021). However, this yielded a list of 34 criteria. The inclusion of 34 criteria in the survey would have resulted in a very large survey, especially when bearing in mind that each criterion had a number of options and participants were required to respond to each option, as can be seen in Figure 1.

Figure 1.

Extract from the survey.

Hence, in the second step, the cut-off was increased to 3.50. This yielded 17 criteria, then, two criteria, which were very close to the cut-off point, were also included. The final list had 19 criteria which was a reasonable size (see Table 1).

Table 1.

The Criteria that were Considered for Assessment.

	M	Met the cut-off point
Course-related
1. Course type (theoretical, practical, or a combination)	3.47	√
2. Number of students enrolled in the course	3.13	×
3. Intended learning outcomes	3.00	×
4. Course level (undergraduate, postgraduate)	2.60	×
5. How students are enrolled (on campus, off campus, both)	3.60	√
6. Offered across multiple campuses	3.40	×
7. Availability of technology to enable online delivery	4.07	√
8. Duration of current face-to-face classes (lecture and tutorial combined)	2.73	×
9. Accreditation: Is the course to comply with an external standard	3.27	×
10. Threat of MOOCs	2.60	×
11. The number of tutors/sessionals available for online delivery	3.13	×
Student-related
12. Students’ preferred learning style (online or face-to-face)	3.60	√
13. Students’ life situation (any outside commitments, such as work or family)	3.93	√
14. Students’ access to campus	4.07	√
15. Students’ technology literacy	3.67	√
16. Students’ preparedness for study	3.20	×
17. Students’ access to technology	4.00	√
18. Students’ attendance requirements (e.g., international students on visas)	3.47	√
19. Students’ language proficiency	3.33	×
20. Students’ expectations (misconceptions, e.g., online work is easier/harder than face-to-face)	3.20	×
21. Level of students (i.e., year of study)	3.00	×
22. Students’ experience with online delivery	3.07	×
Teacher-related
23. Teacher’s preferred teaching style (online or face-to-face)	3.07	×
24. Teacher’s technological knowledge	3.20	×
25. Teacher’s confidence in integrating technology	3.27	×
26. Teacher’s experience in designing for blended learning	3.87	√
27. Teacher’s experience in teaching a traditional course	3.07	×
28. Teacher’s workload	3.93	√
29. Teacher’s willingness to try new teaching methods	4.20	√
30. Peer support and mentoring	3.87	√
Institution-related
31. Blended learning alignment with institutional goals (culture)	3.87	√
32. Time release	3.60	√
33. Technical support	4.13	√
34. Funding	3.20	×
35. Professional development	3.87	√
36. Teacher performance evaluation	3.80	√
37. Supporting teaching innovation	4.13	√

Data Collection and Analysis

This study used a two-round online modified Delphi survey to assess the 19 criteria. We decided to limit the number of rounds to two, mainly because Round 1 is normally used to identify the criteria. This was not needed in this study as all the criteria were identified in a separate study.

Reasons for Selecting the Delphi Method

A Delphi method was used in this study for a number of reasons. First, and most importantly, the Delphi method can help in building consensus among a group of experts who are geographically spread across a wide area and unable to participate in a face-to-face consensus method. Second, the Delphi method is able to provide anonymity of responses. Anonymity is an important characteristic as it allows participants to respond freely without the effects of dominant individuals which is a normal concern when using a group-based approach (Nasa et al., 2021). Anonymity can also eliminate persuasion and coercion to adopt a certain view and reduce the disagreeable aspects of group discussion (Brooks, 1979; Rowe & Wright, 1999). This provides an opportunity for participants to express their opinions and refine their ideas based on their own values or experiences.

Another important characteristic of the Delphi method that can be useful when assessing criteria that might influence the design of a blended course is the iteration process. As has been discussed, the Delphi process consists of a series of rounds. In each round, a panel of experts is asked to respond individually to a questionnaire. They can freely express their opinions and return their answers to the researcher. The researcher collects the responses, revises them, and returns to each participant a summary of the position of the whole panel. Then, as part of the process, the participants rethink and revise their original responses based on the feedback from the previous round (Xie et al., 2021). This process offers the experts the opportunity to suggest an initial assessment of each criterion as they can think of and then rethink and revise their initial responses based on the range of opinions of the whole panel. As a result, experts can move toward more accurate findings that are the product of detailed examinations and discussions.

A final reason behind the selection of the Delphi method was its ability to minimize the effect of noise. Noise, according to Dalkey et al. (1969), is the type of communication that occurs in group discussions and focuses on participants’ interests rather than problem-solving. It causes collected data to be distorted as it is normally biased and not related to the purposes of the study. The controlled feedback process of the Delphi method can reduce the effect of noise (Barrios et al., 2021). Hsu and Sandford (2007) state that when a well-organized summary of the previous round is distributed to the participants, it can provide them with the opportunity to obtain additional insight and can more thoroughly clarify the data collected in previous rounds. By operating within multiple rounds, participants become more focused on problem-solving and they can offer their opinions more insightfully.

Participant Recruitment

A two-step process was used to recruit experts for the study. At first, experts who participated in the previous Delphi Study were asked, in the survey itself, if they were willing to participate in another Delphi study to analyze the impact of the identified criteria. Of the 15 participants who completed the Round 2 survey of the previous Delphi study, 13 agreed and their names and contact details were added to an initial list of potential participants.

In the second step, a purposive approach was adopted to find more experts. The three inclusion criteria were as follows:

A participant should have been involved in designing at least one course;

A participant should have experience with online components such as virtual classes and online discussion boards; and

A participant should have at least one publication in the field of educational technology in top-tier publication venues. Publication records, according to Mirza et al. (2021) and Yazit and Zainab (2007), are an indicator of the professional expertise of academics.

With these inclusion criteria in mind, several university websites were searched. While searching theses websites, the researcher was aiming to include experts from different academic disciplines to understand the impact of the experts’ discipline on criteria assessment. The search resulted in identifying another 42 participants. Eventually, a total of 55 academics were added to the potential participant list.

When recruiting participants, it was important to decide how many experts would be enough for the Delphi panel. There is a wide variation regarding the number of experts on a Delphi panel. While some studies have employed over 60-panel members (Alexander & Kroposki, 1999; Crawford-Williams et al., 2022), others have involved as few as 10 members or less (Steinmann et al., 2021; Zeedick, 2010). Ludwig (1997) notes that the majority of Delphi studies have between 15 to 20 participants. Skulmoski et al. (2007) state that a Delphi panel of 10 to 15 can yield sufficient results. Rowe and Wright (1999) argued that the number of participants should depend on the nature and quality of data being collected and that in-depth feedback might require a smaller panel. They also add that the number of experts available should be considered, as should cost.

In order to decide the minimum number of experts required to yield sufficient results for the study, three key factors were considered: the time available to complete the study, which was 3 to 4 months; the number of facilitators available to moderate the discussion, which was only one researcher; and, the scope of the study, which was the assessment of the impact of 19 criteria on the design of blended learning courses. Taking these factors into account, a decision was made to recruit a panel of 15 to 20 members. It was felt that recruiting more than 20 experts may generate a large amount of data, which can lead to issues of data handling and potential analysis difficulties. It was also thought that a panel of fewer than 15 participants might not yield sufficient results.

To ensure the right number of participants in the Delphi panel, only five experts were contacted at a time. This continued until 15 to 20 experts completed the Round 1 survey.

Developing Round 1 Survey

LimeSurvey was used to develop the surveys for the two rounds. The first round survey comprised two main sections. The first one explained the purpose of the study and asked questions to ascertain the participants’ expertise, and hence their eligibility for inclusion in the panel, that is, experience with online components and experience in course design.

The second section was designed to assess the 19 criteria. The experts were instructed to indicate the proportion of online components that they might incorporate into a blended learning course with respect to each criterion. They were presented with five options to choose from Very high online proportion, High online proportion, Medium online proportion, Low online proportion, and Very Low online proportion (see Table 2). They were also asked to provide any comments regarding their responses. Figure 2 shows an extract from the second section of the Round 1 survey.

Table 2.

Possible Online Proportions.

1 Very low proportion of online components (less than 20% of the course components are online with the rest being face-to-face)

2 Low proportion of online components (20%–45% of the course components are online with the rest being face-to-face)

3 Medium proportion of online components (46%–55% of the course components are online with the rest being face-to-face)

4 High proportion of online components (56%–80% of the course components are online with the rest being face-to-face)

5 Very high proportion of online components (more than 80% of the course components are online with the rest being face-to-face)

Figure 2.

Extract from the second section of Round 1 survey.

After developing the survey, it was pilot tested for reliability and validity. Two academics who have experience in online learning and instructional design were asked to assess the clarity and relevance of the survey items. Their feedback was collected, analyzed, and then several changes were made to the survey items.

Conducting and Analyzing Round 1 Survey

An email containing a link to the online survey was sent to the participants. Round 1 data analysis involved mainly quantitative methods. The standard deviation (level of dispersion) and mean (central tendency) were used to represent participants’ responses. The mean (M) was used to represent the proportion of online components that had been selected for each criterion.

Standard deviation (SD) was used to indicate the level of consensus among the experts. A consensus was considered to be reached when SD ≤ 1 (Cloquell-Ballester et al., 2006; von der Gracht, 2012). A content analysis approach similar to that which Burnard (1991) suggests, as described by Keeney et al. (2011), was used to analyze the written comments provided by the experts. The aim was to understand the reasons behind their ratings.

Developing Round 2 Survey

Round 2 had one section. Experts were presented with the criteria that they had responded to in Round 1 and the panel mean response for each item. They were encouraged to reconsider their responses while taking into account the panel’s mean response. The experts were asked to consider adjusting their responses toward the panel mean response. They were also instructed that if they want to keep any response that is more than one point away from the mean, they should provide justification. Figure 3 shows an extract from the first section of the Round 2 survey.

Figure 3.

Extract from the first section of Round 2 survey.

After developing the survey, it was pre-tested and refined. The two academics who had participated in piloting the Round 1 survey were requested to comment on the clarity and relevance of the survey items. Their feedback was collected, analyzed, and then several changes were made to the survey items.

Conducting and Analyzing Round 2 Survey

The same expert panel from Round 1 was sent e-mails containing a link to the Round 2 survey. Three of them pointed out the need for a No Answer or Not Relevant option suggesting that some criteria were irrelevant or difficult to respond to. The initial intention of not including such an option was to encourage experts to try and complete all the items. Despite that, experts were reminded, in the Round 2 invitation emails, that a previous Delphi study was used to identify criteria that influence the design of blended learning courses. Only criteria that achieved a high importance rating were included in the current study. They were also informed that if they felt that a certain criterion was irrelevant or they could not respond to it, they could leave it blank.

Similarly to Round 1, the Round 2 data analysis involved mainly quantitative methods. The mean (central tendency) and standard deviation (level of dispersion) were used to present information concerning the experts’ responses. Experts’ comments were also analyzed to obtain an understanding of the reasons behind their ratings.

Results

Round 1 Survey

Of the 55 experts who had been invited, 18 (33%) accepted our invitation and completed the Round 1 survey. Six of them were from the 13 experts who participated in the previous Delphi study and had indicated their willingness to participate in this second Delphi Study.

Demographic Data

Table 3 shows the demographic data of the expert panel that was recruited. As can be seen, the members came from different academic disciplines and had years of experience in course design and using online delivery methods (see Table 3). They all had several publications in the field of online learning and 10 of them occupy leading positions in their educational institutions in the field of teaching and learning.

Table 3.

Experts Involved in the Study.

	No. of participants
Discipline^a
1. Education	6
2. Information technology	5
3. Business	4
4. Science	4
5. Economics	3
6. Engineering	3
7. Medicine	2
8. Exercise science	1
9. Management	1
Experience in course design (years)
10. 1–5	2
11. 6–10	5
12. 11–20	2
13. 20+	9
Experience with online delivery (years)
14. 1–3	1
15. 4–6	0
16. 7–10	4
17. 10+	13
Course level
18. Undergraduate	6
19. Postgraduate	5
20. Both	7

Some experts belong to more than one discipline.

Course-Related Criteria

As can be seen in Table 4, all items in this category achieved consensus (SD ≤ 1). The experts gave a very high proportion of online components (M = 4.89) for courses with a majority of off-campus students. The experts also proposed a high proportion of online components (M = 3.94) for courses where most of the technology needed for online delivery was available. However, a largely lower online proportion was proposed for courses that do not have most of the technology needed for online delivery available (M = 1.56).

Table 4.

Ratings of Influence of Course-Related Criteria.

Course-related criteria	Survey items	M	SD
1. Course type	Majority of the course content is theoretical	3.83	0.96
	About half theoretical and half practical	3.39	0.68
	Majority of the course content is practical	2.72	0.73
2. How students are enrolled	Majority are studying on campus	3.28	0.87
	About half studying on campus while the rest are off campus students	4.00	0.75
	Majority are studying off campus	4.89	0.31
3. Availability of technology to enable online delivery	Most of the technology needed for online delivery is available	3.94	0.85
	About half of the technology needed for online delivery is available	2.72	0.80
	Most of the technology needed for online delivery is not available	1.56	0.68

Student-Related Criteria

As can be seen in Table 5, all items in this category but one, that is, Majority have difficulties using the technology required for the online components, achieved consensus. Students’ access to technology (criterion 5) had the greatest impact on the proportion of online components. Students’ attendance requirements (criterion 6) had very little impact on the proportion of online components. This criterion was considered to have the least impact compared with all the other criteria in the whole study.

Table 5.

Ratings of Influence of Student-Related Criteria.

Student-related criteria	Survey items	M	SD
1. Students’ preferred learning style	Majority prefer online instruction	4.17	0.76
	About half prefer online instruction while the rest prefer face-to-face instruction	3.33	0.75
	Majority prefer face-to-face instruction	2.56	0.76
2. Students’ life situation	Majority have outside commitments, such as work or family	4.17	0.69
	About half have outside commitments	3.72	0.8
	Majority have no outside commitments	3.39	0.95
3. Students’ access to campus	Majority live near the campus	3.61	0.83
	About half live near the campus	3.72	0.8
	Majority live away from the campus	4.11	0.94
4. Students’ technology literacy	Majority can easily use the technology required for the online components	4.11	0.74
	About half can easily use the technology required for the online components	3.33	0.82
	Majority have difficulties using the technology required for the online components	2.61	1.06
5. Students’ access to technology	Majority can easily access the technology required for the course	4	0.75
	About half can easily access the technology required for the course	2.67	0.82
	Majority have difficulties accessing the technology required for the course	1.56	0.68
6. Students’ attendance requirements	Majority are international students	3.67	0.94
	About half are international students	3.61	0.68
	Majority are domestic students	3.72	0.73

Teacher-Related Criteria

As can be seen in Table 6, all items in this category achieved consensus. The highest mean of online proportion was given to courses taught by teachers who are willing to try new teaching methods (4.33). Teachers’ willingness to try new teaching methods (criterion 3) recorded the highest impact on the proportion of online components compared with the other teacher-related criteria. The experts indicated a high proportion of online components (M = 4.33) for courses taught by teachers who are willing to try new teaching methods. However, a largely lower online proportion was indicated for courses taught by teachers who are reluctant to try new teaching methods (M = 2.00).

Table 6.

Ratings of Influence of Teacher-Related Criteria.

Teacher-related criteria	Survey items	M	SD
1. Teacher’s experience in designing for blended learning	Teacher has extensive experience	4.00	0.75
	Teacher has moderate experience	3.00	0.67
	Teacher has little experience	2.17	0.60
2. Teacher’s workload	Teacher has time allowance for major redevelopment	4.00	0.88
	Teacher has time allowance for moderate redevelopment	3.44	0.83
	Teacher has time allowance for minor redevelopment	2.44	0.83
3. Teacher’s willingness to try new teaching methods	Teacher is willing to try new teaching methods	4.33	0.67
	Teacher has some hesitation to try new teaching methods	2.94	0.70
	Teacher is reluctant to try new teaching methods	2.00	1.00
4. Peer support and mentoring	Teacher is provided with constant peer support and mentoring	4.11	0.74
	Teacher is provided with some peer support and mentoring	3.44	0.68
	Teacher lacks peer support and mentoring	2.33	1.00

Institution-Related Criteria

As can be seen in Table 7, of the 18 items listed in this category, 15 items achieved consensus. Technical support (criterion 3) had the greatest impact on the proportion of online components compared with the other institution-related criteria. Teacher performance evaluation (criterion 5) had very little impact on the proportion of online components. This criterion was considered to have the second least impact compared with all the other criteria in the study.

Table 7.

Ratings of Influence of Institution-Related Criteria.

Institution-related criteria	Survey items	M	SD
1. Blended learning alignment with institutional goals (culture)	Strong alignment	4.22	0.71
	Moderate alignment	3.44	0.96
	Weak alignment	2.94	1.27
2. Time release	Teacher is provided with adequate time release	4.11	0.74
	Teacher is provided with moderate amount of time release	3.00	0.88
	Inadequate time release is provided	2.17	1.12
3. Technical support	Constant support	4.22	0.85
	Moderate support	3.50	0.83
	Little support	2.17	0.90
4. Professional development	Professional development is provided	4.17	0.96
	Some professional development is provided	3.11	0.66
	Professional development is not provided	2.11	0.99
5. Teacher performance evaluation	Performance evaluation is conducted	3.72	0.99
	Some degree of performance evaluation is conducted	3.39	0.83
	Performance evaluation is not conducted	3.44	0.96
6. Supporting teaching innovation	Teaching innovation is valued	4.22	0.92
	Teaching innovation is somewhat valued	3.50	0.90
	Teaching innovation is not valued	3.00	1.29

Round 2 Survey

Fifteen of the expert panel who completed the Round 1 survey have also completed the Round 2 survey with a response rating of around 83%.

Course-Related Criteria

As can be seen in Table 8, no significant changes were noticed in the experts’ responses compared to the Round 1 results. Experts again gave a very high proportion of online components (M = 4.8) for courses that have a majority of off-campus students and this was the highest online proportion given to any item in the whole survey. The availability of technology to enable online delivery (criterion 3) continued to have the greatest impact on the proportion of online components compared with the other course-related criteria. All items in this subsection achieved consensus and except for one item, that is, Majority are studying off campus, all items moved closer to consensus compared to Round 1 results.

Table 8.

Ratings of Influence of Course-Related Criteria.

Course-related criteria	Survey items	M	SD
1. Course type	Majority of the course content is theoretical	3.80	0.40
	About half theoretical and half practical	3.33	0.60
	Majority of the course content is practical	2.53	0.62
2. How students are enrolled	Majority are studying on campus	3.36	0.61
	About half are studying on campus while the rest are off-campus students	4.07	0.68
	Majority are studying off campus	4.80	0.54
3. Availability of technology to enable online delivery	Most of the technology needed for online delivery is available	3.93	0.25
	About half of the technology needed for online delivery is available	2.67	0.79
	Most of the technology needed for online delivery is not available	1.53	0.62

Student-Related Criteria

As can be seen in Table 9 below, the experts gave the highest proportion of online components for courses that have a majority of students with outside commitments (M = 4.27). Similar to Round 1, Students’ access to technology (criterion 5) had the greatest impact on the proportion of online components. Also, Students’ attendance requirements (criterion 6) scored very little impact on the proportion of online components and continued to be the criterion with the least impact compared to all the other criteria in the subsection. Overall, all items in this subsection achieved a high level of consensus (one did not in Round 1) and their consensus scores moved closer to consensus when compared to the Round 1 results.

Table 9.

Ratings of Influence of Student-Related Criteria.

Student-related criteria	Survey items	M	SD
1. Students’ preferred learning style	Majority prefer online instruction	4.20	0.40
	About half prefer online instruction while the rest prefer face-to-face instruction	3.53	0.50
	Majority prefer face-to-face instruction	2.67	0.60
2. Students’ life situation	Majority have outside commitments, such as work or family	4.27	0.44
	About half have outside commitments	3.67	0.60
	Majority have no outside commitments	3.53	0.62
3. Students’ access to campus	Majority live near the campus	3.67	0.60
	About half live near the campus	3.80	0.75
	Majority live away from the campus	4.13	0.62
4. Students’ technology literacy	Majority can easily use the technology required for the online components	4.13	0.34
	About half can easily use the technology required for the online components	3.27	0.77
	Majority have difficulties using the technology required for the online components	2.33	0.70
5. Students’ access to technology	Majority can easily access the technology required for the course	4.13	0.34
	About half can easily access the technology required for the course	2.80	0.91
	Majority have difficulties accessing the technology required for the course	1.73	0.68
6. Students’ attendance requirements	Majority are international students	3.71	0.59
	About half are international students	3.64	0.48
	Majority are domestic students	3.64	0.48

Teacher-Related Criteria

As can be seen in Table 10, the highest online proportion was given to courses taught by teachers who are willing to try new teaching methods (M = 4.13). Similar to Round 1, Teacher’s willingness to try new teaching methods (criterion 3) recorded the highest impact on the proportion of online components compared with the other teacher-related criteria. All items in this subsection achieved consensus and they all moved closer to consensus when compared to the Round 1 results.

Table 10.

Ratings of Influence of Teacher-Related Criteria.

Teacher-related criteria	Survey items	M	SD
1. Teacher’s experience in designing for blended learning	Teacher has extensive experience	4.07	0.25
	Teacher has moderate experience	3.21	0.41
	Teacher has little experience	2.47	0.72
2. Teacher’s workload	Teacher has time allowance for major redevelopment	4.07	0.25
	Teacher has time allowance for moderate redevelopment	3.27	0.44
	Teacher has time allowance for minor redevelopment	2.53	0.62
3. Teacher’s willingness to try new teaching methods	Teacher is willing to try new teaching methods	4.13	0.34
	Teacher has some hesitation to try new teaching methods	3.07	0.57
	Teacher is reluctant to try new teaching methods	2.40	0.88
4. Peer support and mentoring	Teacher is provided with a constant peer support and mentoring	4.00	0.37
	Teacher is provided with some peer support and mentoring	3.47	0.50
	Teacher lacks peer support and mentoring	2.67	0.79

Institution-Related Criteria

Similar to Round 1, Technical support (criterion 3) continued to score the greatest impact on the proportion of online components compared with the other institution-related criteria. The score for Teacher performance evaluation (criterion 5) also continued to demonstrate very little impact on the proportion of online components. It was the criterion with the least impact compared with all the other criteria in the section. The three items that did not achieve consensus in Round 1 moved considerably closer toward consensus and achieved consensus in Round 2. Eventually, all items in this subsection (see Table 11), achieved consensus and they all moved closer to consensus when compared to Round 1 scores.

Table 11.

Ratings of Influence of Institutional-Related Criteria.

Institutional-related criteria	Survey items	M	SD
1. Blended learning alignment with institutional goals (culture)	Strong alignment	4.13	0.34
	Moderate alignment	3.40	0.49
	Weak alignment	3.07	0.68
2. Time release	Teacher is provided with adequate time release	4.00	0.00
	Teacher is provided with moderate amount of time release	3.27	0.44
	Inadequate time release is provided	2.47	0.81
3. Technical support	Constant support	4.13	0.34
	Moderate support	3.53	0.62
	Little support	2.47	0.81
4. Professional development	Professional development is provided	4.20	0.40
	Some professional development is provided	3.53	0.62
	Professional development is not provided	2.66	0.87
5. Teacher performance evaluation	Performance evaluation is conducted	3.92	0.47
	Some degree of performance evaluation is conducted	3.54	0.50
	Performance evaluation is not conducted	3.54	0.50
5. Supporting teaching innovation	Teaching innovation is valued	4.20	0.40
	Teaching innovation is somewhat valued	3.67	0.60
	Teaching innovation is not valued	3.47	0.81

Analysis and Discussion

This two-round Delphi study assessed the impact of criteria that need consideration when deciding the proportion of online components of a blended learning course. A total of 19 criteria were assessed in this study. Below is a detailed discussion of their impact on the design of blended courses.

Course-Related Criteria

Three course-related criteria were assessed and their impact on the proportion of online components can be seen in Figure 4.

Figure 4.

Impact of course-related criteria.

Course type (theoretical, practical, or a combination)

The experts recommended that the more practical content in the course, the fewer online components that should be incorporated into that course. This agrees with Díaz and Entonado’s (2009) finding that it is more satisfactory and efficacious to deliver theoretical content online and that students who are studying online may find it difficult to cope with practical content. This, however, should not be interpreted to mean that Online components are not appropriate for practical courses. A closer look at Figure 4 shows that experts indicated a notable proportion of online components for courses with a majority of practical content. An expert explained: “if you have a highly technical course to deliver, and the technology permits real-time online demonstrations, you should give some proportion to online delivery.”

How students are enrolled (on campus, off campus, and both)

The experts recommended that the more students studying off campus, the more online components that should be incorporated into the course (see Figure 4). Not surprising, a very high proportion of online components was indicated for courses with a majority of off-campus students. This was the highest mean of online proportion given to any item in the survey. It is probably due to the fact that off-campus students are usually seeking flexibility and convenience (Graham, 2012; Ota et al., 2018), and activities that require them to be time- and place-bound are not desirable. One expert commented: “online proportion should vary depending on student locations and study preferences.”

Availability of technology to enable online delivery (Do teachers have all the technology they need for the online delivery?)

The experts recommended that the more technology available, the more online components that could be incorporated into the course. Most importantly, Availability of technology to enable online delivery indicated a large impact on the proportion of online components (see Figure 4). While a very low online proportion was indicated for courses that lack most of the needed technology, a high proportion was suggested for courses that have most of the required technology available. One expert commented: “if you have decent internet access then you can put together a blended site. If you don’t have decent internet access then you need to focus on good f2f techniques.” This corresponds with the findings of our previous study (Alammary, Carbone et al., 2015) where Availability of technology to enable online delivery was identified as the most important course-related criterion that needed to be considered when deciding the proportion of online components in a blended course.

Student-Related Criteria

Six student-related criteria were assessed. The impact of three of them on the proportion of online components is presented in Figure 5; the impact of the other three is presented in Figure 6.

Figure 5.

Impact of student-related criteria (Part A).

Figure 6.

Impact of student-related criteria (Part B).

Students’ preferred learning style (online or face-to-face)

The experts recommended that the more students who prefer online instruction, the more online components that should be incorporated into the course (see Figure 5). This view is supported by the finding of Limniou and Smith (2010) and Wang et al. (2019), who stress the importance of providing students with a learning experience that matches their individual learning styles. However, it is worth noting that some experts highlighted the necessity for teachers to push their students to work outside their comfort zone. One expert remarked: “Student preferences should not govern teaching approaches but student capabilities must. That doesn’t mean they shouldn’t engage with online education at a high level but you may need to be wary.” Another commented: “It is good for instructors to push students work outside their preferred learning style and there has been curriculum work done on this in primary and secondary school.”

Students’ life situation (any outside commitments, such as work or family?)

The experts recommended that the more students who have outside commitments, the more online components that should be incorporated into the course. An explanation for this is that online components provide flexibility and convenience for students to pace their own studies. Flexibility and convenience are of great importance for students who seek additional education but who have commitments such as family and work (Graham, 2012; Peck et al., 2021). What was interesting is the medium proportion of online components that was indicated for courses with the majority of its students having no outside commitment (see Figure 5). It seems that the experts believe that blended courses should have more online than face-to-face proportions regardless of outside commitments.

Students’ access to campus (Do they live near or away from the campus?)

The experts recommended that the more students who live away from the campus, the more online components that should be incorporated into the course. This again seems to be related to the need for blended courses to provide a flexible and convenience learning experience. Aggun (2019) and Vaughan (2007) point out that an important aspect of blended learning is to allow students to choose the time and place for their learning rather than commuting and finding available parking spaces.

Students’ technology literacy (Can they use the technology required for the online components?)

The experts recommended that the more students who can easily use the technology required for the online delivery, the more online components that could be incorporated into the course. Dahlstrom and Bichsel (2014) and Zhu et al. (2021) recommend that it is important for teachers not to assume that all students are technically inclined. They stressed the need to assess students’ technology literacy and provide those who are less technically inclined with more personalized help. Interestingly, as can be seen in Figure 6, experts indicated a notable proportion of online components for courses with a majority of students who cannot use the required technology. It might be the case that experts believe that students’ technology literacy can be resolved by providing more support and resources to students commencing with a blended learning course.

Students’ access to technology (Can they access the technology required for the course?)

The experts recommended that the more students who have difficulties accessing the technology required for the course, the fewer online components that should be incorporated into that course. As can be seen in Figure 6, Students’ access to technology indicates a strong impact on the proportion of online components. One expert explained: “Technological aspects of the blended learning mode need to be addressed properly for the blended learning mode to work well.” Another one remarked: “For equity reasons, we can’t leave 50% or more unserviced.”

Students’ attendance requirements (e.g., international students on visas)

The experts recommended that, given the increased number of international students, slightly more online components than face-to-face components could be incorporated into the course. As can be seen in Figure 6, this criterion showed very little impact on the proportion. In fact, it scored the least impact on the proportion of online components compared to all the other criteria that have been analyzed. This result was not entirely unexpected given the fact that out of the 19 criteria that have been qualified for assessment, this criterion had the lowest importance rating of 3.47.

Teacher-Related Criteria

Four teacher-related criteria were assessed and their impact on the proportion of online components is shown in Figure 7.

Figure 7.

Impact of teacher-related criteria.

Teacher’s experience in designing for blended learning

The experts recommended that the more experience the teachers have in designing for blended learning, the more online components that could be incorporated into the course. This corresponds with the work of Duhaney (2004) who suggests that teachers with little experience in blended learning should start simply and implement incrementally. Ertmer and Ottenbreit-Leftwich (2010) and Alammary, Sheard et al. (2015) also point out that experience can help build teachers’ confidence in developing and running their blended learning courses.

Teachers’ workload (Do they have time allowance for redevelopment?)

The experts recommended that the more time allowance the teachers have for redevelopment, the more online components that could be incorporated into the course. This agrees with many studies such as those of Dekeyser et al. (2014) and Ibrahim and Ismail (2021) that identified teachers’ workload as a constraint that may limit the proportion of online components that could be incorporated into blended courses.

Teacher’s willingness to try new teaching methods

The experts recommended that the more willingness the teachers have to try new teaching methods, the more online components that could be incorporated into the course. Teachers’ willingness, as can be seen in Figure 7, seems to be a major contributor that can motivate the integration of a high proportion of online components. However, it also seems that teachers’ negative attitudes toward new technology should not stop them completely from integrating some online components for the benefit of their students.

Peer support and mentoring (Are they provided with peer support and mentoring?)

The experts recommended that the more peer support and mentoring provided to the teachers, the more online components that could be incorporated into the course. However, as can be seen in Figure 7, even when teachers lack peer support and mentoring, the majority of experts recommended a low to a medium proportion of online components be incorporated into the course. One of them commented: “Peer mentoring is valuable but its absence should not be a show stopper.”

Institution-Related Criteria

Six institution-related criteria were assessed. Three of them are presented in Figure 8 while the other three are presented in Figure 9.

Figure 8.

Impact of institution-related criteria (Part A).

Figure 9.

Impact of institution-related criteria (Part B).

Blended learning alignment with institutional goals (culture)

The experts recommended that the more blended learning aligns with institutional goals, the more online components that could be incorporated into the course. One expert explained: “Institutional policy and support for blended delivery is a major factor in considering what level of blending should be adopted (for example my institution has a mandatory basic level of online defined for all courses).” However, as can be seen in Figure 8, a medium proportion of online components was indicated for courses offered in an institution where blended learning has weak alignment with institutional goals. It seems that experts want teachers to be self-starters who adopt new teaching approaches to improve their students’ learning experiences.

Time release (Does the institution provide teachers with time release?)

The experts recommended that the more time release the institution provides to teachers, the more online components that could be incorporated into the course. According to many studies, it is vital to provide time release for teachers engaged in blended learning developments(Benson & Avery, 2009; Wedding et al., 2018). Vaughan (2007) proposes that the development of a blended course normally takes two to three times longer than the development of a similar course in the traditional format. However, the findings of this study, as can be seen in Figure 8, suggest that the lack of time release should not stop teachers from incorporating some proportion of online components in their courses.

Technical support

The experts recommended that the more technical support the institution provides to teachers, the more online components that could be incorporated into the course. Technical support seems to be important to facilitate the design of the blended course and to minimize the risk of failure. One expert explained: “Adequate support needs to be in place to do things properly. If support isn’t in place, then no one is going to have any fun at all.” Another commented: “I would tend to largely minimise or avoid online to mitigate risks of poor student experience and instructor frustration.”

Professional development (Does the institution provide professional development?)

The experts recommended that the more professional development the institution provides to teachers, the more online components that could be incorporated into the course. Similar to Technical support, professional development seems to be useful to facilitate the design of the blended course and minimize the risk of failure. One expert noted: “Training and support are key issues for many staff and are the most likely influences of decisions.” However, as can be seen in Figure 9, it seems that the absence of professional development should not stop teachers from incorporating some online components in their courses. One expert commented: “Professional development is valuable but its absence should not be a show stopper.”

Teacher performance evaluation (Is performance evaluation conducted?)

It seems that performance evaluation can motivate academics to incorporate more online components into their courses. However, it is important to note that there was some divergence of opinions. While the majority of experts believe that online components might result in better evaluation, some think that it might result in poorer evaluation. One of them explained: “Performance evaluation may vary depending on student locations and study preferences. More online in a primarily campus based educational setting (unless done well) can result in poorer student evaluations.”

Supporting teaching innovation

The experts recommended that the more support for teaching innovation, the more online components that could be incorporated into the course. Supporting teaching innovation, as can be seen in Figure 9, seems to be a major contributor that can facilitate the integration of a high proportion of online components. However, when it is not valued, experts also recommended that a medium proportion of online components be incorporated.

Implications for Teaching and Best Practices

This study investigated the impact of 19 different criteria on the design of blended learning courses and on determining the most appropriate proportion of online to face-to-face components of a blended course. The study shows that designing a successful blended learning course is a complex problem that requires academics to compromise between what is good for their students and what is possible, considering different influential criteria. Tables 12 and 13 show the top cases where the lowest and highest proportions of online components were proposed by the experts who participated in this study. These cases show that academics should start their design process by paying careful attention to the technologies needed for online delivery. When most of these technologies are not available, or when students cannot access or use these technologies, only a small proportion of online components should be added. Academics should then look at the life situations and preferences of their students. If the majority of students have a strong preference for online instruction or have outside commitments, this should encourage the integration of the highest possible proportion of online components. Institutional support is a key factor for the success of blended learning. Considering the large number of delivery options available for blended courses, academics should be provided with professional development focused on the proper use of online components that academics have never experienced before. Valuing teaching innovation and dealing with academics’ resistance to new teaching approaches can encourage them to step out of their comfort zone and commit to blended learning.

Table 12.

Top 5 Cases Where the Lowest Proportions of Online Components Were Proposed.

Cases	M ^a
1. Most of the technology needed for online delivery is not available	1.53
2. Majority of students have difficulties accessing the technology required for the course	1.73
3. Majority of students have difficulties using the technology required for the online components	2.33
4. Teacher is reluctant to try new teaching methods	2.40
5. Teacher has little experience in designing for blended learning	2.47

Mean proportion of online components.

Table 13.

Top 5 Cases Where the Highest Proportions of Online Components Were Proposed.

Cases	M ^a
1. Majority of students are studying off campus	4.80
2. Majority of students have outside commitments, such as work or family	4.27
3. Majority of students prefer online instruction	4.20
4. Professional development is provided by the educational institution	4.20
5. Teaching innovation is valued by the educational institution	4.20

Mean proportion of online components.

In around 76% of the cases that were presented to the experts, they recommended medium to high proportions of online components to be added to blended courses, that is, 46% to 80% of course components are online with the rest being face-to-face. This indicates that blended learning courses should normally have more online than face-to-face components. The inclusion of a higher proportion of online components could overcome the shortcomings of traditional teaching methods and encourage the use of innovative teaching approaches. It would increase students’ flexibility and provide them with a larger range of course-scheduling options. It also gives students more autonomy in coordinating their learning, helping them to become independent learners. This learning independence is attributed to the limited attendance time. While students are spending most of their time online, there are still face-to-face collaborative activities with peers and teachers to increase learning motivation and reduce procrastination. Moreover, the majority of online activities are student-centered approaches that are useful to meet diverse students’ needs. Student-centered teaching approaches focus on improving students’ learning skills. Academics role is only to facilitate learning by understanding students’ different needs and developing a learning environment that can meet these needs. As a result, students become more engaged in the learning process and take more responsibility for their own learning.

The Impact of Disciplines and Academic Levels

The experts’ area of discipline does not seem to have an effect on their preferences regarding the proportion of online components. For example, the four experts from the Business discipline indicated different proportions of online components for courses that mainly comprise theoretical content. One indicated a very high proportion, another indicated a high proportion, the third indicated a medium proportion and the last one indicated a low proportion. Moreover, the three experts who indicated a very high proportion of online components for courses that have a majority of domestic students, were from three different disciplines: Information Technology, Exercise Science, and Education.

The course level at which the experts teach, that is, undergraduate, postgraduate, or both, also does not seem to have an effect on the experts’ preferences regarding online proportion. For example, the five experts who teach at the postgraduate level indicated different proportions of online components for courses that have a majority of students with no outside commitments. Two indicated a very high proportion, one indicated a high proportion, the third indicated a medium proportion and the last one indicated a low proportion.

Limitations

Despite the contribution that this research has made to the understanding of blended learning course design, it has some limitations that warrant mention. First, the experts who participated in the study comprised academics from New Zealand and Australian universities only. Therefore, some of the identified criteria might only be relevant in the New Zealand and Australian contexts. A limitation associated with the Delphi method is the problem of requiring all participants to respond to the open-ended questions that ask them to justify their responses. While a good number of experts provided justification for their ratings, some did not. According to Adler and Ziglio (1996) and Lim et al. (2014), the nature of the open-ended questions in a survey format may not yield the rich explanation that is characteristic of one-on-one interviews.

Conclusion

This study provided a detailed description of how a two-round modified Delphi method was used to assess 19 criteria that have been found to influence the decision regarding the proportion of online components in a blended course. To our knowledge, this is the first study to investigate this aspect of blended learning course design. The study has made two primary contributions. First, it identified the impact of the 19 criteria on the design process, and accordingly, the cases in which the lowest and the highest proportions of online components should be added to blended courses. Second, it helped in answering the frequent question regarding the most appropriate proportion of online to face-to-face components that should be incorporated into blended learning courses. The results show that technological aspects, that is, Availability of technology and Students’ access to technology, have a strong negative impact on the proportion of online components of a blended course. On the other hand, students’ flexibility and convenience as well as the availability of a high level of institutional support can highly motivate the integration of a high proportion of online components. The results also indicate that blended courses should have medium to high proportions of online components, that is, 46% to 80% of course components are online with the rest being face-to-face. This study opens up new areas for future work. An important one is to investigate the influential criteria from the student’s perspective. This investigation would look at criteria that influence students’ preference for a certain proportion of online components and assess the impact of these criteria.

Footnotes

I would like to thank Professor Judithe Sheard and Professor Angela Carbone for their invaluable help in conducting this study.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author received no financial support for the research,authorship,and/or publication of this article.

ORCID iD

Ali Saleh Alammary

Ethical Approval

Ethical approval for this study was obtained from the Monash University Human Research Ethics Committee (Reference number: CF15/273 - 2015000132).

References

Adler

Ziglio

(1996). Gazing into the oracle: The Delphi method and its application to social policy and public health. Jessica Kingsley Publishers.

Aggun

(2019). A comparison of online courses with traditional face-to-face courses in the English language teaching. International Journal of Languages’ Education and Teaching, 7(3), 126–137.

Ahuja

Yang

Shankar

(2009). Study of ICT adoption for building project management in the Indian construction industry. Automation in Construction, 18(4), 415–423. https://doi.org/10.1016/j.autcon.2008.10.009

Alammary

A. S.

(2021). LOsMonitor: A machine learning tool for analyzing and monitoring cognitive levels of assessment questions. IEEE Transactions on Learning Technologies, 14(5), 640–652.

Alammary

A. S.

(2022). A toolkit to support the design of blended learning courses. IEEE Access, 10, 85530–85548.

Alammary

Carbone

Sheard

(2017, June 27–30). Curriculum transformation using a blended learning design toolkit. In 40th HERDSA Annual International Conference. Higher Education Research and Development Society of Australasia, Inc.

Alammary

Sheard

J. I.

Carbone

(2015, August 25–27). The application of multiple-criteria decision analysis (MCDA) to address blended learning design challenges. In the 24th International Conference on Information Systems Development 2015 (pp. 52–63), Harbin, China.

Alammary

Carbone

Sheard

(2015, January). Identifying criteria that should be considered when deciding the proportion of online to face-to-face components of a blended course. In Bui

T. X.

Sprague

R. H.

Jr (Eds.), Proceedings of the 48th Annual Hawaii International Conference on System Sciences (HICSS 2015) (pp. 72–80). IEEE.

Alexander

Kroposki

(1999). Outcomes for community health nursing practice. Journal of Nursing Administration, 29(5), 49–56. http://journals.lww.com/jonajournal/Fulltext/1999/05000/Outcomes_for_Community_Health_Nursing_Practice.7.aspxanonymized

10.

Ashraf

M. A.

Yang

Zhang

Denden

Tlili

Liu

Huang

Burgos

(2021). A systematic review of systematic reviews on blended learning: Trends, gaps and future directions. Psychology Research Behavior Management, 14, 1525.

11.

Barrios

Guilera

Nuño

Gómez-Benito

J. J. T. F.

Change

(2021). Consensus in the delphi method: What makes a decision change? Technological Forecasting and Social Change, 163, 120484.

12.

Bennardo

Buffone

Fortunato

Giudice

(2020). COVID-19 is a challenge for dental education-A commentary. European Journal of Dental Education, 24, 822–824. https://doi.org/10.1111/eje.12555

13.

Benson

Avery

(2009). Embedding web 2.0 strategies in learning and teaching. In Lytras

M. D.

Damiani

Ordóñez

de Pablos (Eds.), Web 2.0 (pp. 1–12). Springer. https://doi.org/10.1007/978-0-387-85895-1_13

14.

Bonk

C. J.

Graham

C. R.

(2012). The handbook of blended learning: Global perspectives, local designs. John Wiley.

15.

Brooks

K. W.

(1979). Delphi technique: Expanding applications. North Central Association Quarterly, 53(3), 377–385.

16.

Burnard

(1991). A method of analysing interview transcripts in qualitative research. Nurse Education Today, 11(6), 461–466.

17.

Cloquell-Ballester

V.-A.

Cloquell-Ballester

V.-A.

Monterde-Díaz

Santamarina-Siurana

M.-C.

(2006). Indicators validation for the improvement of environmental and social impact quantitative assessment. Environmental Impact Assessment Review, 26(1), 79–105. https://doi.org/http://dx.doi.org/10.1016/j.eiar.2005.06.002

18.

Cottam

Roe

Challacombe

(2004). Outsourcing of trucking activities by relief organisations. Journal of Humanitarian Assistance, 1(1), 1–26.

19.

Crawford-Williams

Koczwara

Chan

R. J.

Vardy

Lisy

Morris

Iddawela

Mackay

Jefford

(2022). Defining research and infrastructure priorities for cancer survivorship in Australia: A modified Delphi study. Supportive Care in Cancer, 30(5), 3805–3815.

20.

Dahlstrom

Bichsel

(2014). ECAR study of undergraduate students and information technology. Retrieved 10 January, 2016, from http://www.csus.edu/irt/cio/strategicplanning/Documents/Undergrad-Students-and-IT-2014-Full-Report.pdf

21.

Dalkey

N. C.

Brown

B. B.

Cochran

(1969). The Delphi method: An experimental study of group opinion (Vol. 3). Rand Corporation.

22.

de Haan

Boon

W. P. C.

Amaratunga

Dondorp

A. M

. (2022). Expert perspectives on the introduction of Triple Artemisinin-based Combination Therapies (TACTs) in Southeast Asia: A Delphi study. BMC Public Health, 22(1), 864. https://doi.org/10.1186/s12889-022-13212-x

23.

Dekeyser

Watson

Baré

(2014, August 31–September 3). Designing an academic workload model in the age of blended learning [Conference session]. Tertiary Education and Management Conference, Cairns, Australia.

24.

Díaz

L. A.

Entonado

F. B.

(2009). Are the functions of teachers in e-learning and face-to-face learning environments really different? Educational Technology & Society, 12(4), 331–343.

25.

Driscoll

(2002). Blended learning: Let’s get beyond the hype. Retrieved 4 April, 2015, from http://www-07.ibm.com/services/pdf/blended_learning.pdf

26.

Duhaney

D. C.

(2004). Blended learning in education,training, and development. Performance Improvement, 43(8), 35–38. https://doi.org/10.1002/pfi.4140430810

27.

Ehrlich

McKenney

Elkbuli

(2020). We asked the experts: Virtual learning in surgical education during the COVID-19 pandemic-shaping the future of surgical education and training. World Journal of Surgery, 44(7), 2053–2055. https://doi.org/10.1007/s00268-020-05574-3

28.

Ertmer

P. A.

Newby

T. J.

(2008). Behaviorism, cognitivism, constructivism: Comparing critical features from an instructional design perspective. Performance Improvement Quarterly, 6(4), 50–72.

29.

Ertmer

P. A.

Ottenbreit-Leftwich

A. T.

(2010). Teacher technology change: How knowledge, confidence, beliefs, and culture intersect. Journal of Research on Technology in Education, 42(3), 255–284.

30.

Finlay

M. J.

Tinnion

D. J.

Simpson

(2022). A virtual versus blended learning approach to higher education during the COVID-19 pandemic: The experiences of a sport and exercise science student cohort. Journal of Hospitality, Leisure, Sport & Tourism Education, 30, 100363. https://doi.org/10.1016/j.jhlste.2021.100363

31.

Garrison

D. R.

Vaughan

N. D.

(2011). Blended learning in higher education: Framework, principles, and guidelines. John Wiley & Sons. http://books.google.com.au/books?id=UhYnZbYhDl0C

32.

Graham

C. R.

(2012). Blended learning systems: Definition, current trends, and future directions. In Bonk

C. J.

Graham

C. R.

(Eds.), Handbook of blended learning: Global perspectives, local designs (pp. 3–21). Wiley, John & Sons, Incorporated.

33.

Guthrie

Walker

James

M. T.

Selby

N. M.

Tonelli

Bell

(2022). Developing an AKI consensus definition for database research: Findings from a scoping review and expert opinion using a Delphi process. American Journal of Kidney Diseases, 79(4), 488–496.e481. https://doi.org/https://doi.org/10.1053/j.ajkd.2021.05.019

34.

Hamann

Glazier

R. A.

Wilson

B. M.

Pollock

P. H.

(2021). Online teaching, student success, and retention in political science courses. European Political Science, 20(3), 427–439. https://doi.org/10.1057/s41304-020-00282-x

35.

Hsu

C.-C.

Sandford

B. A.

(2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1–8.

36.

Iaconelli

Anderman

E. M.

(2021). Classroom goal structures and communication style: The role of teacher immediacy and relevance-making in students’ perceptions of the classroom. Social Psychology of Education, 24(1), 37–58.

37.

Ibrahim

Ismail

(2021). University ESL instructors’ reflections on the use of blended learning in their classrooms. TESOL Technology Studies, 2(1), 25–35.

38.

Jones

Sharma

(2020). On reimagining a future for online learning in the post-COVID era. First posted on medium. com.

39.

Kearsley

(1995, May 18–21). The nature and value of interaction in distance learning [Conference session]. Invitational Research Conference in Distance Education: Towards Excellence in Distance Education: A Research Agenda, Pennsylvania, PA, United States. Pennsylvania State University.

40.

Keeney

Hasson

McKenna

H. P.

(2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38(2), 195–200. https://doi.org/10.1016/S0020-7489(00)00044-4

41.

Keeney

Hasson

McKenna

(2011). Analysing data from a Delphi and reporting results. In The Delphi technique in nursing and health research (pp. 84–95). Wiley-Blackwell.

42.

Kirkley

(2005). Creating next generation blended learning environments using mixed reality, video games and simulations. TechTrends, 49(3), 42–53. https://doi.org/10.1007/BF02763646

43.

Kliziene

Taujanskiene

Augustiniene

Simonaitiene

Cibulskas

G. J. S.

(2021). The impact of the virtual learning platform EDUKA on the academic performance of primary school children. Sustainability, 13(4), 2268.

44.

Kwangmuang

Jarutkamolpong

Sangboonraung

Daungtod

S. J. H.

(2021). The development of learning innovation to enhance higher order thinking skills for students in Thailand junior high schools. Heliyon, 7(6), e07309.

45.

Lane

Hoang

J. G.

Leighton

J. P.

Rissanen

(2021). Engagement and satisfaction: Mixed-method analysis of blended learning in the sciences. Canadian Journal of Science, Mathematics Technology Education, 21(1), 100–122.

46.

Lave

Wenger

(1991). Situated learning: Legitimate peripheral participation. Cambridge University Press.

47.

Lim

D. H.

Morris

M. L.

Kupritz

V. W.

(2014). Online vs. blended learning: Differences in instructional outcomes and learner satisfaction. Journal of Asynchronous Learning Networks, 11(2), 27–42.

48.

Limniou

Smith

(2010). Teachers’ and students’ perspectives on teaching and learning through virtual learning environments. European Journal of Engineering Education, 35(6), 645–653.

49.

Liu

(2021). Does teacher immediacy affect students? A systematic review of the association between teacher verbal and non-verbal immediacy and student motivation. Frontiers in Psychology, 12, 2475.

50.

Lombardo

Kantola

(2021). Social constructivism. In Abels

Krizsán

MacRae

van der Vleuten

(Eds.), The Routledge handbook of gender and EU politics (pp. 43–55). Routledge.

51.

Ludwig

(1997). Predicting the future: Have you considered using the Delphi methodology. Journal of Extension, 35(5), 1–4.

52.

Mirza

Harrison

E. A.

Miller

K. A.

Jacobs

(2021). Indicators of quality rehabilitation services for individuals with limited English proficiency: A 3-round Delphi study. Archives of Physical Medicine Rehabilitation, 102(11), 2125–2133.

53.

Moore

M. G.

(1989). Editorial: Three types of interaction. American Journal of Distance Education, 3(2), 1–7.

54.

Müller

Mildenberger

(2021). Facilitating flexible learning by replacing classroom time with an online learning environment: A systematic review of blended learning in higher education. Educational Research Review, 34, 100394. https://doi.org/10.1016/j.edurev.2021.100394

55.

Nasa

Jain

Juneja

D. J. W. j. o. m

. (2021). Delphi methodology in healthcare research: How to decide its appropriateness. 11(4), 116.

56.

Ogbona

(2021). Rethinking learning in schools: Understanding the design of in-class flip blended learning (p. 26). La Institución Universitaria Colombo Americana.

57.

Oliver

Stallings

(2014). Preparing teachers for emerging blended learning environments. Journal of Technology and Teacher Education, 22(1), 57–81.

58.

Öncü

(2022). Blended learning: Blending in the era of advanced technology. In Durak

Çankaya

(Eds.), Handbook of research on managing and designing online courses in synchronous and asynchronous environments (pp. 178–195). IGI Global.

59.

Ota

Peck

Porter

(2018). Evaluating a blended online learning model among undergraduate nursing students: A quantitative study. Computers, Informatics, Nursing, 36(10), 507–512.

60.

Peck

Smith

Terry

Porter

J. E.

(2021). Self-regulation for and of learning: Student insights for online success in a bachelor of nursing program in regional Australia. Nursing Reports, 11(2), 364–372. https://www.mdpi.com/2039-4403/11/2/35

61.

Piaget

(1977). The development of thought: Equilibration of cognitive structures ( Rosin

, Trans). Viking.

62.

Picciano

A. G.

(2009). Blending with purpose: The multimodal model. Journal of the Research Center for Educational Technology, 5(1), 4–14.

63.

Poon

(2012). Use of blended learning to enhance the student learning experience and engagement in property education. Property Management, 30(2), 129–156. https://doi.org/doi:10.1108/02637471211213398

64.

Rowe

Wright

(1999). The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15(4), 353–375.

65.

Saudi Electronic University. (2021). Saudi Electronic University - Faculty members directory. Author. https://seu.edu.sa/caic/en/staff/

66.

Setyaningsih

(2020). Face to face or online learning: Students’ perspectives on blended learning in Indonesia. Journal of English Language Studies, 5(1), 1–14.

67.

Sharpe

Benfield

Roberts

Francis

(2006). The undergraduate experience of blended e-learning: A review of UK literature and practice. Retrieved November 24, 2022, from https://www.heacademy.ac.uk/system/files/Sharpe_Benfield_Roberts_Francis_SUMMARY_0.pdf

68.

Skulmoski

Hartman

Krahn

(2007). The Delphi method for graduate research. Journal of Information Technology Education: Research, 6(1), 1–21.

69.

Steinmann

Delnoij

van de

Bovenkamp H.

Groote

Ahaus

(2021). Expert consensus on moving towards a value-based healthcare system in the Netherlands: A Delphi study. BMJ Open, 11(4), e043367.

70.

Theoret

Ming

(2020). Our education, our concerns: Medical student education impact due to COVID-19. Medical Education, 54(7), 591–592.

71.

Ugwude

A. C.

Agu

A. N.

Ekweogu

L. B.

(2021). University administrators’ use of management information systems (mis) feedback mechanisms for decision-making in south east Nigerian universities. UNIZIK Journal of Educational Research Policy Studies, 4, 178–191.

72.

Vaughan

(2007). Perspectives on blended learning in higher education. International Journal on E-Learning, 6(1), 81.

73.

von der Gracht

H. A

. (2012). Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technological Forecasting and Social Change, 79(8), 1525–1536. https://doi.org/10.1016/j.techfore.2012.04.013

74.

Vygotsky

L. S.

(1980). Mind in society: The development of higher psychological processes. Harvard University Press.

75.

Waddoups

G. L.

Hatch

G. L.

Butterworth

(2003). Case 5: Blended teaching and learning in a first-year composition course. Quarterly Review of Distance Education, 4(3), 271–278.

76.

Wagner

E. D.

(2012). On desiging interaction experience for the next generation of blended learning. In Bonk

C. J.

Graham

C. R.

(Eds.), The handbook of blended learning: Global perspectives, local designs (pp. 41–55). Pfeiffer Publishing.

77.

Wang

Kremer

G. E.

Jackson

K. L.

(2019). An investigation of effectiveness differences between in-class and online learning: An engineering drawing case study. International Journal on Interactive Design Manufacturing, 13(1), 89–98.

78.

Wedding

Cousins

Quinn

(2018). Transitioning staff, students and course materials to blended and online learning environments. In Rahman

Ilic

(Eds.), Blended learning in engineering education: Recent developments in curriculum, assessment and practice (pp. 35–63). CRC Press.

79.

Woo

Reeves

T. C.

(2007). Meaningful interaction in web-based learning: A social constructivist interpretation. The Internet and Higher Education, 10(1), 15–25. https://doi.org/http://dx.doi.org/10.1016/j.iheduc.2006.10.005

80.

Xie

Dong

Chen

Peng

X. J. R. E.

Safety

(2021). A novel risk evaluation method for fire and explosion accidents in oil depots using bow-tie analysis and risk matrix analysis method based on cloud model theory. Reliability Engineering & System Safety, 215, 107791.

81.

Yazit

Zainab

(2007). Publication productivity of Malaysian authors and institutions in LIS. Malaysian Journal of Library and Information Science, 12, 35–55.

82.

Zeedick

D. M.

(2010). The modified delphi method to analyze the application of instructional design theory to online graduate education. The George Washington University.

83.

Zhu

Berri

Zhang

(2021). Effective instructional strategies and technology use in blended learning: A case study. Education and Information Technologies, 26(5), 6143–6161. https://doi.org/10.1007/s10639-021-10544-w

How to Decide the Proportion of Online to Face-to-Face Components of a Blended Course? A Delphi Study

Abstract

Keywords

Introduction

Research Background

Criteria That Were Considered for Analysis

Data Collection and Analysis

Reasons for Selecting the Delphi Method

Participant Recruitment

Developing Round 1 Survey

Conducting and Analyzing Round 1 Survey

Developing Round 2 Survey

Conducting and Analyzing Round 2 Survey

Results

Round 1 Survey

Demographic Data

Course-Related Criteria

Student-Related Criteria

Teacher-Related Criteria

Institution-Related Criteria

Round 2 Survey

Course-Related Criteria

Student-Related Criteria

Teacher-Related Criteria

Institution-Related Criteria

Analysis and Discussion

Course-Related Criteria

Course type (theoretical, practical, or a combination)

How students are enrolled (on campus, off campus, and both)

Availability of technology to enable online delivery (Do teachers have all the technology they need for the online delivery?)

Student-Related Criteria

Students’ preferred learning style (online or face-to-face)

Students’ life situation (any outside commitments, such as work or family?)

Students’ access to campus (Do they live near or away from the campus?)

Students’ technology literacy (Can they use the technology required for the online components?)

Students’ access to technology (Can they access the technology required for the course?)

Students’ attendance requirements (e.g., international students on visas)

Teacher-Related Criteria

Teacher’s experience in designing for blended learning

Teachers’ workload (Do they have time allowance for redevelopment?)

Teacher’s willingness to try new teaching methods

Peer support and mentoring (Are they provided with peer support and mentoring?)

Institution-Related Criteria

Blended learning alignment with institutional goals (culture)

Time release (Does the institution provide teachers with time release?)

Technical support

Professional development (Does the institution provide professional development?)

Teacher performance evaluation (Is performance evaluation conducted?)

Supporting teaching innovation

Implications for Teaching and Best Practices

The Impact of Disciplines and Academic Levels

Limitations

Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

Ethical Approval

References