Sage Journals: Discover world-class research

Abstract

Interventions are crucial in the learning analysis process. Learning analytics-based interventions are being widely applied in the field of education. However, it is currently unclear whether learning analytics-based interventions effectively enhance students’ learning effects. To conduct a comprehensive review assessing the extent to which interventions contribute to the enhancement or deterioration of learning outcomes. This study employs meta-analysis to quantitatively analyze 34 published empirical research articles related to learning analytics-based interventions. The results obtained from this analysis are as follows: (1) Interventions based on learning analytics can greatly enhance students’ learning outcomes, with a moderate overall effect value; (2) In terms of specific dimensions of learning outcomes, learning analytics-based interventions can effectively improve knowledge acquisition, while the improvements in social emotion and cognitive skill were relatively small; (3) subject areas, learning stage, learning environments, intervention type, and diagnostic assessment tool all serve as important moderating factors affecting the interventions on student learning outcomes.

Keywords

learning analytics interventions effectiveness learning effect meta-analysis moderator variables

Introduction

Learning analytics is gaining popularity among researchers because of its emphasis on quantifying, collecting, analyzing, and reporting data regarding learners and their environments. This is done to understand and optimize learning experiences and the environments (Long & Siemens, 2011 ; Siemens & Baker, 2012). Conversely, learning analytics utilizes data sources, analytical methods, and predictive modeling to inform interventions that aim to improve students’ learning experiences (Kew & Tasir, 2017; Sønderlund et al., 2018; Williams, 2014). Learning analytics encompasses five main elements: data collection, analysis, student learning, audience, and interventions (Brown, 2011). Thus, these interventions are vital in “closing the loop” within the cyclic learning analytic process (Clow, 2012; Knobbout & Van Der Stappen, 2020). The use of data-driven methodologies has guided learning analytics-based interventions in identifying student difficulties and providing them with timely, personalized assistance, leading to beneficial outcomes for both the learners and the entire learning environment (B. T. M. Wong & Li, 2020). This study examines the impact of learning analytics-based interventions on student learning outcomes, focusing on knowledge acquisition, cognitive skills, and social emotion. The framework proposed by Darling-Hammond et al. (2019) serves as the basis for this analysis.

While several studies have conducted comprehensive reviews and analyses of literature about learning analytics-based interventions from different perspectives, there is a scarcity of empirical research specifically investigating the effects of these interventions on learning outcomes (Kew & Tasir, 2017; Tepgeç & Ifenthale, 2022; Zheng et al., 2023). Therefore, the question of how learning analytics-based interventions enhance student learning outcomes remains unanswered, which implies that educators have not yet implemented learning analytics-based interventions more effectively to improve student’s learning outcomes (Rienties, Cross & Zdrahal, 2016 ). For example, Zheng et al. (2023) conducted a meta-analysis of 33 empirical studies on learning analytics-based interventions and their impact on learning successes from 2012 to 2021. The authors discovered that interventions based on learning analytics had a significant and favorable effect on learning achievements. In addition, the findings indicated that the impact of these interventions was greatly influenced by various aspects, including the composition of the sample, the areas of learning, the methods of learning, the technology used for learning analytics, and the metrics used for learning analytics. Nevertheless, this study solely assessed the overall influence of learning analytics-based interventions on learning outcomes without divulging the precise effects of these interventions or determining if notable distinctions existed among these factors. Sønderlund et al. (2018) conducted a comprehensive analysis of 11 research that examined the impact of learning analytics-based interventions on the effectiveness of higher education. The authors found that interventions based on learning analytics had the potential to improve student success and retention. Nevertheless, the study failed to reveal the magnitudes of the effects of interventions based on learning analytics on students’ learning outcomes or identify the most effective intervention methods for improving learning outcomes. Thus, it is yet unclear how interventions based on learning analytics can better encourage changes in learning outcomes. Furthermore, the efficacy of interventions based on learning analytics in promoting gains in learning outcomes has been the subject of mixed or contradicting findings in prior empirical research. To examine how students’ self-directed meta-cognitive prompts affected their navigation behavior and learning performance, Bannert et al. (2015) used a pre-post experimental design. The research conducted by these authors revealed that the group that utilized self-directed meta-cognitive prompts demonstrated superior transfer performance compared to those who did not employ such prompts. Lonn et al. (2014) examined student motivation within a learning analytics-based interventions implemented during a summer bridge program. Their results indicated that self-report interventions launched by teachers had a detrimental effect on the course mastery scores of remedial students.

The studies above indicated the inconsistent effectiveness of learning analytics-based interventions in promoting students’ learning effect. Hence, conducting an extensive review is crucial to assess whether and to what extent learning analytics-based interventions contribute to increasing or decreasing learning outcomes. Meta-analysis is a systematic research method that involves statistically analyzing and synthesizing many individual study outcomes using effect sizes (ES; Glass, 1976). This research method can reduce the uncertainty associated with individual studies and produce more decisive research outcomes (Lipsey & Wilson, 2000).

This meta-analysis aimed to examine the efficacy of interventions based on learning analytics in improving learning outcomes. The goal was to provide guidance for future endeavors in improving learning environments and interventions, as discussed below.

What is the total effect size of learning analytics-based interventions in enhancing students’ learning outcomes and their impact on the three dimensions of learning outcomes (i.e., knowledge acquisition, cognitive skills, and social emotion)?

If there are variations in the effects of different experimental studies included in a meta-analysis, how do various moderating variables impact the disparities in research conclusions?

Consequently, The structure of this paper is as follows: Section “Literature Review” is a literature review that examines the definition of learning analytics-based interventions, the dimensions that determine their effectiveness, and the factors that influence their efficacy. In the following section, we detail our methodology, which includes a literature search strategy, criteria for sample data inclusion and exclusion, data coding and analysis, and an assessment of publication bias and heterogeneity. In the “Results” and “Discussion” sections, we present and discuss the findings of our study. Finally, in the “Implications, Limitations, and Future Directions” section, we summarize the conclusions, discuss the limitations of this study, and suggest directions for future research.

Literature Review

Learning Analytics-Based Interventions Definition and Effectiveness Dimensions

Learning analytics-based interventions have recently become extensively utilized in the education sector. Nevertheless, a precise and unambiguous definition of learning analytic-based interventions is currently lacking. After a comprehensive review and analysis of previous research, the definition of learning analytics-based interventions is broadly categorized into two approaches. The first perspective highlights the comprehensive structure of interventions based on learning analytics. For instance, Wise (2014) defines learning analytics-based interventions as the surrounding frame of activity through which analytic tools, data, and reports are taken up and utilized; Khalil and Ebner (2015) mentioned that “Intervention, as the final stage of the learning analytics cycle, depends on the input from the previous stage to determine the specific issues to be addressed.” The Society for Learning Analytics Research (SoLAR) asserts that Learning analytics-based interventions are designed to facilitate the optimization of learning and learning environments ( Society of Learning Analytics Research, n.d.). On the other hand, the second perspective focuses on the specific operations and applications of learning analytics-based interventions. For example, Brown (2011) noted that learning analytics-based interventions are about analyzing digital data to help educators understand student behavior patterns, predict which students might face difficulties, and provide timely support. Additionally, it can assist in optimizing course design, improving teaching methods, and evaluating the effectiveness of educational programs; B. T. M. Wong and Li (2020) pointed out that learning analytics-based interventions are measures taken to enhance student outcomes by addressing challenges such as reduced retention and inadequate pass rates, utilizing data on student learning progress; Zheng et al. (2023) consider learning analytics-based interventions to refer to interventions based on learning analytics dashboards and interventions informed by learning analytics (e.g., providing feedback, prompts, or suggestions based on the outcomes of learning analytics). However, there is a distinction between the two definitions. . The first category definition emphasizes the overall framework of learning analytics-based interventions, which is rather broad, whereas the second category definition focuses on the specific operations and applications of learning analytics-based interventions, neglecting the importance of learning analytics-based interventions as a whole framework. This study believes that the definition of learning analytics-based interventions should emphasize the overall framework and pay attention to specific operations and applications. In addition, Zheng et al. (2023) pointed out that learning analytics-based interventions involve two learning settings: face-to-face and online learning. This study focuses on online learning interventions. Therefore, this study considers learning analytics-based interventions as the measures taken by employing learning analytics technology to collect and analyze data from learners and their contexts (e.g., providing automated feedback, personalization of course materials, learner support, and sending information for learners, or messages from teacher to learner).

In the field of practical teaching, several researchers have examined the effectiveness of learning analytics-based interventions based on this definition. For instance, Rienties, Boroowa, et al. (2016) manifested that the proactive dispatch of warning emails to students identified as being at risk positively influences their attitudes, behaviors, and cognitive processes; Kim et al. (2016) carried out an experimental research to demonstrate that the use of learning analytics dashboards can enhance students’ academic performance; Lu et al. (2017) found that teachers sending messages to students based on learning analytics reports can enhance their self-regulation abilities; Y. Lee and Specht (2023) found that providing automated feedback in Human-Robot Interaction (HRI) can enhance learners’ self-efficacy. From previous studies, it has been universally acknowledged that in the quest to examine the effectiveness of learning analytics-based interventions, Some researchers have concentrated on on learners’ knowledge acquisition, as evidenced by academic performance, examination scores, course grades, and grade point average (GPA; Kim et al., 2016; Knobbout & Van Der Stappen, 2020; Viberg et al., 2018; Zheng et al., 2023). Others have emphasized the enhancement of cognitive skills, including abilities such as self-regulated learning and collaborative learning (Heikkinen et al., 2023; Lu et al., 2017; Nussbaumer et al., 2015; J. Wong et al., 2019). Additionally, attention is concentrated on the social-emotional dimensions, including learning attitudes, motivation, and satisfaction (D. Lee et al., 2019; Lonn et al., 2014; Rienties, Boroowa, et al., 2016; Smith et al., 2012). This aligns with the broader understanding of learning outcomes as described by Darling-Hammond et al. (2019), who state that learning outcomes are primarily directed toward academic, cognitive, and social-emotional. Building on the classification of learning outcomes by Darling-Hammond et al. (2019), and synthesizing existing research on the effectiveness of learning analytics-based interventions, this study categorizes the outcomes of learning analytics-based interventions into knowledge acquisition, cognitive skill, and social emotion.

Factors in the Effectiveness of Learning Analytics-Based Interventions

The effectiveness of learning analytics-based intervention may vary among different factors (B. T. M. Wong, 2017; B. T. M. Wong & Li, 2020). For example, regarding a subject area, Rienties, Boroowa, et al. (2016) found that the intervention effect in Technology was better than expected, while Psychology performed relatively poorly, although with a minor impact. From the perspective of the learning stage, Zhang et al. (2023) mentioned in their meta-analysis that the intervention effects in higher education are superior to those in primary, middle, and high schools. In terms of intervention duration, Jayaprakash et al. (2014) mentioned that within the first 5 weeks of the course, sending academic alerts to students at risk through learning analytics tools allows both teachers and students enough time to identify and address existing academic issues, thereby improving student academic performance, while Rienties, Boroowa, et al. (2016) adopted a quasi-experimental method to analyze the impact of sending early warning emails to potentially at-risk students using learning analytics on their learning outcomes. From the 12th week onward, the experimental group had a considerably greater level of involvement in the virtual learning environment than the control group, as indicated by the study. From the perspective of the learning environment, Kew and Tasir (2021) noted that the most commonly implemented online learning environments for learning analytics-based interventions mainly comprise four types, namely Learning Management System (LMS; such as Moodle, Blackboard, etc.), Computer-based environment (such as CSCL, desktop application), virtual learning environment (VLE), web-based environment (except LMS, VLE, MOOC), and other. Kim et al. (2016) conducted an experimental study on groups using a dashboard, comparing the experimental group with a control group. These authors found that within the experimental group students who used the dashboard had higher final scores than those in the control group who did not use it. This study indicates that learning analytics dashboards (LAD) can improve student academic achievement. Kim et al. (2016) indicated that high-achieving learners show lower satisfaction with using the dashboard. Knobbout and Van Der Stappen (2020) used open coding to synthesize the categories for intervention types based on a systematic literature review of 62 articles about learning analytics-based treatments. Eight distinct types of learning analytics-based interventions have been identified in the literature so far: automated feedback (AF), dashboards (D), information for learners (IL), information for teachers (IT), messages from teachers to learners (MT), course material personalization (PC), learner support (LS), and visualization (V). The effects of learning analytics-based feedback in massive online courses were investigated experimentally by Lim et al. (2019) . According to these authors, academically, students in the feedback-receiving experimental group outperformed their non-feedback-receiving control group, where learning analytics can help students learn. Conversely, Bannert and Reimann (2012) interviewed students exposed to self-regulation prompts. The findings revealed that half of these students felt these prompts hindered and interrupted their learning. Regarding diagnostic assessment tools, Xu et al. (2023a) systematically coded the experimental literature included in the meta-analysis, considering that the evaluation tool consists of a few categories. Kew and Tasir (2017) pointed out that learning analytics technology can capture, record, and store learners’ login frequency, page views, number of posts, replies, academic performance, and test scores, which is more conducive to standardized test assessments. Lonn et al. (2014) found integrating psychological scales with existing learning analytics-based intervention measures difficult and challenging.

Finally, several variables affect how well interventions based on learning analytics work. The present study investigated the effectiveness of learning analytics-based interventions in enhancing student learning outcomes, particularly regarding the impact of subject area, learning stage, intervention duration, learning environments, intervention type, and diagnostic assessment tool on student learning improvement. It aims to provide evidence regarding whether these factors enhance or reduce student learning outcomes and to what extent.

Method

This research followed Cooper’s (2010) rigid meta-analysis approach for analyzing quantitative data from multiple studies on the same topic, including database searching, identification, screening, eligibility, merging, duplicate removal, and study analysis. Rev-Man 5.4 was used to conduct a meta-analysis. Cohen’s kappa coefficient was used to test the consistency of three researchers’ data, and publication bias and heterogeneity tests were done on the sample data to evaluate this meta-analysis.

Literature Search Strategy

A comprehensive literature search was carried out to gain insight into the present state of research regarding the efficacy of learning analytics interventions in enhancing learning outcomes. The search encompassed several electronic databases, including the Web of Science Core Collection and the Chinese Social Science Citation Index (CSSCI), as well as journal papers included in the China National Knowledge Infrastructure (CNKI). The reason for choosing the LAK conference and these databases is that they maintain stringent standards for the quality of the literature they compile, ensuring authenticity and reliability through mechanisms such as peer review. The study employed Boolean operators in a search query within Web of Science, which are as follows: TS=(“learning analytics” or “learning analysis” and “experiment” or “empirical study” or “control group” or “quasi-experiment”) or (“learning analytics” or “learning analysis” and “pretest” or “post-test”) and (“intervention” or “instructional intervention” or “educational intervention”) and (“learning effect” or “learning outcome” or “learning achievement” or “learning performance”). “Education and Educational Research” was the focus of the study from 2010-01-31 to 2023-12-31. A grand total of 681 items were kept. Utilizing the Boolean operator in the CNKI, the search string SU=(“learning analytics”*“intervention”+“learning analytics”*“instructional intervention”+“learning analytics”*“educational intervention”+“learning analytics”’*“learning effect”+“learning analytics”*“learning outcome”+“learning analytics”*“learning achievement”+“learning analytics”*“learning performance”) was utilized. The research period from January 1, 2010, to December 31, 2023, is known as the AND variable, and it can be defined as an experiment, empirical study, control group, quasi-experiment, pretest, or post-test. We ended up keeping 143 items. After eliminating 47 items duplicated in the search results, a total of 1,002 relevant research studies were found based on the literature search methodologies. The overall number of studies was 955. Afterward, 34 research documents were chosen based on the inclusion criteria after two researchers reviewed the abstracts and titles of the 955 collected studies.

Inclusion Criteria and Exclusion Criteria

This meta-analysis incorporated literature matching the specified criteria: (1) The issue should be the learning analytics-based interventions on students’ learning outcomes. (2) Experimental or quasi-experimental designs with pre-and post-test or comparative experimental data should be used. (3) Must identify particular markers, such as sample size, mean, standard deviation, and other descriptive statistical measures to assess the efficacy in augmenting students’ learning outcomes. Additionally, to ensure the generalizability of the findings, we excluded studies that were (1) written in languages other than English and Chinese; (2) conference abstracts and review articles; (3) did not provide usable effect sizes; (4) studies that published the same data more than once.

Two researchers meticulously examined the complete texts of each item, adhering strictly to the criteria for inclusion and exclusion. They employed a snowball sampling method, leveraging the references and citations from the included publications to achieve thorough coverage. In the end, 34 items were kept.

Coding of Study Characteristics and Moderator Variables

Two researchers independently reviewed the literature to identify relevant studies (Figure 1), achieved a satisfactory agreement (Cronbach’s α = .847), and resolved disagreements through consensus discussions. The two researchers gathered descriptive data in each study, encompassing the paper title, author’s name, and publication year. Furthermore, they retrieved the data encompassing three metrics used to measure the learning impact of learning analytics-based interventions: sample size, mean value, and standard deviation.

Figure 1.

Flowchart illustrating the process of acquiring literature for the meta-analysis.

The variable information from 34 experimental studies is primarily concentrated on two types of variables: dependent variable (learning effect) and moderator variables (subject area, learning stage, intervention duration, learning environment, intervention type, diagnostic assessment tools). As a dependent variable, the learning effect included knowledge acquisition, cognitive skill, and social emotions. The literature identified the following six factors as moderator variables for the purpose of sub-group analysis: subject area, learning stage, intervention duration, learning contexts, intervention type, and diagnostic assessment tools. Table 1 depicts the specific coding scheme, Appendix 1 depicts the list of references included in the study, Appendix 2 depicts the coding information for dependent variable and moderating variables.

Table 1.

The Process of Coding Eigenvalues in Literature.

Descriptive information	Descriptive information			Data information
Descriptive information	Field	Type	Explanation	Data information
Article titleAuthor’s namePublication year	Dependent variable	Learning effect	Knowledge acquisition; Cognitive skill; Social emotion	Sample sizeAverage valueStandard deviation
	Moderating variable	Learning stage	Primary school; Middle school; High school; Higher education
		Subject area	Science; Education; Mathematics; Physics; Psychology; other
		Intervention duration	Less than a week; 1–4 weeks; 4–18 weeks; More than 18 weeks
		Learning environments	Learning Management System; Learning analytics Dashboards; Game-based learning; Intelligent Tutoring System; Integrated development environment; other
		Intervention type	Automated feedback; Information for learner; Message from teacher to learner; Personalization of course materials; Learner support; Visualization
		Diagnostic Assessment tool	Self-complied scale; Self-complied test; Standardized test

In this study, the sample size, average, standard deviation, and other parameters of 34 studies are extracted, and the standardized mean difference (SMD) is selected as the effect size according to the needs of the study to characterize the effect of learning analytics-based interventions on students’ learning outcomes. SMD = difference between group effect means ÷ standard deviation of participant effects (Hedges, 1981 ), and its functional relationship is as follows:

$SMD = \frac{M_{E} - M_{C}}{\sqrt{\frac{{S_{E}}^{2} (N_{E} - 1) + {S_{C}}^{2} (N_{C} - 1)}{N_{E} + N_{C} - 2}}} (1 - \frac{3}{4 (N_{E} + N_{C}) - 9})$

Where M_E and M_C denote the average of the experimental and control groups, respectively; N_E and N_C denote the sample size of the experimental and control groups, respectively; S_E and S_C denote the standard deviation of the experimental and control groups, respectively.

Analysis of Data

A comprehensive meta-analysis regarding the information in relevant literature was conducted using Rev-man 5.4 software, calculated the overall effectiveness of whether learning analytics-based interventions can promote students’ learning effect, and conducted a homogeneity test to explore the influence of moderating variable (subject area, learning stage, intervention duration, learning environments, intervention type, diagnostic assessment tools) on students’ learning outcomes.

Assessment of Publication Bias and Heterogeneity

Publication Bias Test

Publication bias leads to an overestimation of the effect size. Research studies demonstrating statistically significant findings are more prone to being published and referenced (Rothstein, 2008 ). The validity and dependability of meta-analysis findings hinge on the ability to conduct an impartial and thorough examination of publication bias throughout the research endeavor. Hence, it is imperative to do a publication bias assessment of the sample data in the meta-analysis. The most popular methods for assessing publication bias, including funnel plots and Egger’s test, were used in this study. In the funnel plot, the standard error is depicted vertically, and the effect magnitude is represented horizontally. A symmetrical and uniform distribution of the sample data around the mean effect size suggests a reduced probability of publication bias. Out of the 76 effect values examined in this work, most show a dispersed distribution focused in the upper-middle area of the funnel plot. This area corresponds to the dependable section of the dataset, as shown in Figure 2. Further quantitative testing revealed that, with the exception of knowledge acquisition variables, all variables exhibited good funnel plot symmetry in Egger’s test, indicating no signs of publication bias (Egger’s test: p-value > .05), as detailed in Table 2.

Figure 2.

Funnel plot analyzing 76 effect sizes concerning publication bias across 34 studies.

Table 2.

Results of Publication Bias Egger’s Test.

Variables	Bias	SE bias	t	df	p
Overall	1.2615	0.6870	1.84	75	.070
knowledge acquisition	2.6667	1.1078	2.41	37	.021
Cognitive skill	2.9520	1.4586	2.02	25	.054
Social emotion	5.9749	2.9536	2.02	11	.071

Knowledge acquisition variables (Egger’s test: t = 2.41, p < .05) indicate the presence of publication bias in the knowledge acquisition dimension of learning outcomes. After correction using the trim-and-fill method, Figure 3 shows symmetry following the trim-and-fill adjustment, with the pooled effect size (95% CI) being 0.640 [0.428, 0.852]. The results after applying the trim method did not significantly differ from the original results, implying that the data is reliable.

Figure 3.

Funnel plot of knowledge acquisition variables after clipping and patching.

Heterogeneity Test

Homogeneity analysis examines whether all effect sizes estimate the identical population mean. Through this type of analysis, other factors that influence the strength of the relationship can be uncovered (Kyndt et al., 2013). The meta-analysis usually employs Cochrane’s Q and I² values for heterogeneity testing. According to the heterogeneity test principle proposed by Hedges and Vevea (1998), if I² > 75%, Q-value>df, it indicates significant heterogeneity among the study samples. The random-effects model should be chosen for classification and coding; otherwise, if there is homogeneity among the study samples, the fixed-effects model should be selected for classification and coding. The results showed that the Q-value is larger than the degree of freedom (Q-value = 942.68, df = 75, p < .001), and the I² value was 92%, which exceeded 75%, indicating the heterogeneity is significantly high, as depicted in Table 3. Consequently, the total effect size is computed using the random-effects model.

Table 3.

Heterogeneity Test Results.

Effect model	Effect quantities	Effect size	95% Confidence interval		Heterogeneity test
Effect model	Effect quantities	Effect size	Lower limit	Upper limit	Q	df (Q)	p	I ²
Fixed effect model	76	0.37	0.34	0.40	942.68	75	<.001	92%
Random effect model	76	0.46	0.34	0.57	942.68	75	<.001	92%

Results

An Examination of the Total Effect Size

A total of 34 empirical investigations were included in the random effect model, which examined 76 effect quantities. Figure 4 shows the forest plot that this study produced after removing heterogeneity. It has 76 effect values. The map indicates a statistically significant total impact value of 0.46 (Z = 7.75, p < .001). Interventions grounded on learning analytics have the potential to substantially enhance students’ learning effect, with an overall effect value at a lower-moderate level, as determined by Cohen's (1988) criteria for the size of effect value (95% CI [0.34, 0.57]).

Figure 4.

Impact of the forest map.

This study analyzed knowledge acquisition, cognitive skill, and social emotion to better understand how learning analytics-based interventions improve learning outcomes (see Table 4). Learning analytics-based interventions improved knowledge acquisition (ES = 0.55), social emotion (ES = 0.39), and cognitive skills (ES = 0.35) with insignificant differences between groups (Q = 2.48, p > .01). Therefore, learning analytics-based interventions improve three distinct dimensions of learning outcomes. It is worth noting that the improvements in students’ knowledge acquisition were most affected (ES = 0.55, Z = 6.89, p < .001, 95% CI [0.40, 0.71]), while the improvements in social emotion (ES = 0.39, Z = 2.03, p < .05, 95% CI [0.01, 0.76]) and cognitive skill (ES = 0.35, Z = 3.22, p < .05, 95% CI [0.14, 0.56]) was relatively small.

Table 4.

Text of Intervention Strategies Concerning the Three Learning Outcomes.

	Dimensions of learning effects	Effect quantity	Effect size	95% Confidence interval		Heterogeneity test		Two-tailed test Z (p)	Intergroup effect
Variable	Dimensions of learning effects	Effect quantity	Effect size	Lower limit	Upper limit	I ²	df (p)	Two-tailed test Z (p)	Q (p)
Learning effects	knowledge acquisition	38	0.55	0.40	0.71	95%	37 (<.001)	6.89 (<.001)	Q = 2.48p = .29
	Cognitive skills	26	0.35	0.14	0.56	85%	25 (<.001)	3.22 (<.05)
	Social emotion	12	0.39	0.01	0.76	88%	11 (<.001)	2.03 (<.05)
Combined effect size		0.46 [0.34,0.57], Z = 7.74(p < .001)

Analysis of Moderator Effect Size

The heterogeneity test results demonstrated a substantial relationship among the 76 effect quantities (I² = 92%, Z = 7.74, p < .001), suggesting that potential moderator factors may impact the variations in effect sizes. A two-tailed test was performed on these quantities (see Figure 4). This study used subgroup analyses on the following variables: subject area, learning stage, intervention duration, learning environment, intervention type, diagnostic assessment tool, and intervention type to investigate potential factors, as depicted in Table 5.

Table 5.

Text of Moderating Effect.

	Variable type	Effect quantity	Effect size	95% Confidence interval		Heterogeneity test		Two-tailed test Z (p)	Intergroup effect
Moderating Variable	Variable type	Effect quantity	Effect size	Lower limit	Upper limit	I ²	df (p)	Two-tailed test Z (p)	Q (p)
Subject area	Science	40	0.56	0.40	0.72	89%	39 (<.001)	6.83 (<.001)	Q = 62.52p < .001
	Education	6	0.75	0.19	1.30	90%	5 (<.001)	2.64 (<.05)
	Mathematics	6	1.46	−0.20	3.12	87%	5 (<.001)	1.73 (.08)
	Physics	4	0.90	0.67	1.14	78%	3 (<.05)	7.58 (<.001)
	Psychology	8	0.21	0.17	0.24	44%	7 (.09)	12.23 (<.001)
	Other	12	1.29	0.54	2.04	97%	11 (<.001)	3.38 (<.001)
Combined effect size		0.57[0.47,0.67], Z = 11.02(p < .001)
Learning stage	Primary school	6	2.27	0.48	4.06	88%	5 (<.001)	2.48 (<.05)	Q = 26.68p < .001
	Middle school	22	1.01	0.74	1.28	88%	21 (<.001)	7.32 (<.001)
	High school	6	0.19	−0.31	0.68	95%	5 (<.001)	0.74 (.46)
	Higher education	42	0.32	0.21	0.42	93%	41 (<.001)	5.95 (<.001)
Combined effect size		0.55[0.45,0.65], Z = 10.58(p < .001)
Intervention duration	Less than a week	14	0.57	0.33	0.80	96%	13 (<.001)	4.76 (<.001)	Q = 6.37p = .09
	1–4 weeks	21	0.69	0.38	0.99	79%	20 (<.001)	4.46 (<.001)
	4–18 weeks	27	0.41	0.27	0.55	95%	26 (<.001)	5.72 (<.001)
	More than 18 weeks	14	1.25	0.42	2.08	89%	13 (<.001)	2.94 (<.05)
Combined effect size		0.57[0.47,0.67], Z = 11.02(p < .001)
Learning environments	Learning management system	29	0.33	0.23	0.44	94%	28 (<.001)	6.12 (<.001)	Q = 19.51p < .05
	Learning analytics Dashboards	10	0.36	−0.35	1.06	81%	9 (<.001)	0.99 (.32)
	Game-based learning	5	2.11	0.56	3.67	89%	4 (<.001)	2.66 (<.05)
	Intelligent Tutoring System	21	0.88	0.60	1.15	87%	20 (<.001)	6.26 (<.001)
	Integrated development environment	5	0.81	0.09	1.52	97%	6 (<.001)	2.20 (<.05)
	other	6	0.26	−0.23	0.75	1%	5 (.41)	1.04 (.30)
Combined effect size		0.57[0.47,0.67], Z = 11.02(p < .001)
Intervention type	Automated feedback	32	0.52	0.31	0.73	85%	31 (<.001)	4.80 (<.001)	Q = 73.87p < .001
	Information for learner	4	0.03	−0.05	0.10	38%	3 (.19)	0.74 (.46)
	Message from teacher to learner	11	0.67	0.00	1.35	97%	10 (<.001)	1.97 (.05)
	Personalization of course materials	4	2.66	1.88	3.44	69%	3 (<.05)	6.71 (<.001)
	Learner support	12	0.23	0.15	0.30	80%	11 (<.001)	5.98 (<.001)
	Visualization	13	1.36	0.31	2.41	99%	12 (<.001)	2.55 (<.05)
Combined effect size		0.65[0.52, 0.79], Z = 9.42(p < .001)
Diagnostic Assessment tool	Self-complied scale	16	0.12	−0.15	0.38	80%	15 (<.001)	0.86 (.39)	Q = 13.22p < .05
	Self-complied test	42	0.59	0.48	0.70	94%	41 (<.001)	10.83 (<.001)
	Standardized test	18	1.28	0.38	2.18	95%	17 (<.001)	2.79 (<.05)
Combined effect size		0.57 [0.47, 0.67], Z = 11.02(p < .001)

Subject Area

Different subject areas promoted the improvement of learning outcomes, and differences between groups were significant (Q = 62.52, p < .001). Mathematics had the greatest impact on learning outcomes (ES = 1.46, p < .001), followed by other (ES = 1.29, p < .001) and physics (ES = 0.90, p < .001). Education (ES = 0.75, p < .001) and science (ES = 0.56, p < .001) exhibited a moderately high level of impact effects. Psychology was the least effective (ES = 0.21, p < .001). These results showed that natural sciences fields (such as mathematics, physics, and science) may be the most effective subject areas for using applied learning analysis to improve learning outcomes.

Learning Stage

Various learning stages impacted learning outcomes, and their effect sizes were significant at all grade levels (Q = 26.68, p < .001). Among them, the largest for elementary school students (ES = 2.27, p < .001), followed by middle school students (ES = 1.01, p < .001); college students were relatively smaller (ES = 0.32, p < .001), and the smallest for high school students (ES = 0.19, p < .001). It is essential to highlight that in higher education, the number of empirical research on learning analytics-based interventions is relatively high (this accounts for 55% of the total number of studies). The findings of this study show that implementing learning analytics-based interventions during the primary school stage is beneficial for enhancing students’ learning outcomes.

Intervention Duration

Different intervention duration positively impacted learning outcomes, and no significant differences found between groups (Q = 6.37, p = .09). The effect size was categorized in the following order: more than 18 weeks (ES = 1.25, p < .05), 1 to 4 weeks (ES = 0.69, p < .001), less than a week (ES = 0.57, p < .001), 4 to 18 weeks (ES = 0.41, p < .001). These results indicate that while different intervention durations positively facilitate learning outcomes, they cannot be considered the determining factor influencing learning outcomes. Furthermore, an extended duration of learning analytics-based interventions lasting beyond 18 weeks appears to be associated with optimal learning outcomes.

Learning Environments

Different learning environments significantly impacted learning outcomes, with significant differences between groups (Q = 19.51, p < .05). In these online learning environments, the application of game-based learning tools significantly impacted students’ learning outcomes (ES = 1.70, p < .001), followed by an integrated development environment (ES = 0.70, p < .001) and intelligent Tutoring System (ES = 0.69, p < .001), application of learning analytics dashboards (ES = 0.22, p = 1.30) and learning management system (ES = 0.21, p < .001) had little effect on students’ learning outcomes. These results showed that implementing Game-based learning tools for instructional interventions can significantly improve students’ learning outcomes.

Intervention Type

Different intervention type influenced learning outcomes, and differences between groups were significant (Q = 73.87, p < .001). The personalization of course materials (ES = 2.66, p = .02) and visualization (ES = 1.36, p < .001) had attained a high level of impact. The message from teacher to learner (ES = 0.67, p < .001) and automated feedback (ES = 0.52, p < .001) achieved an upper-medium level of effect, the learner support (ES = 0.23, p < .001) reached a relatively modest level of effectiveness, the information for learner (ES = 0.03, p = .24) had no significant positive effect.

Diagnostic Assessment Tool

Different diagnostic assessment tools influenced learning outcomes, and differences between groups were significant (Q = 13.22, p < .05). The standardized test (ES = 1.28, p < .001) exhibited highly significant effects, the self-complied test (ES = 0.59, p < .001) reached an upper-medium degree of effect, and the self-complied scale produced an insignificant positive effect. (ES = 0.12, p < .001). These findings show that the standardized test has a positive promotional effect on students’ learning outcomes.

Discussion

The Impact of Learning Analytics-Based Interventions on Students' Learning Outcomes

This meta-analysis of 34 empirical studies indicates that learning analytics-based interventions can improve student learning outcomes and positively promote the three dimensions of learning outcomes. The results showed that the improvement in knowledge acquisition related to this has a high and significant effect, while the corresponding enhancements of cognitive skills and social emotion are relatively smaller. These findings support the idea that learning analytics-based interventions can enhance academic performance. This is consistent with previous research (Kew & Tasir, 2017; Gong & Liu, 2019; Karaoglan Yilmaz & Yilmaz, 2022). This finding indicates that learning analytics can identify potential learning issues early on by collecting and analyzing data on students’ learning behaviors. Implementing targeted interventions can help students better understand and master course content, encourage active participation, and significantly enhance academic performance. However, the results indicate that learning analytics-based interventions have a relatively small effect on the enhancement of cognitive skills, which is consistent with the findings of Ye and Zhou (2022). Cognitive skills are a complex process, which involves the active construction of new behaviors that affect learning, and it is unlikely that students can improve their cognitive skills through short-term courses (Nussbaumer et al., 2015). Learning analytics-based interventions have limitations in enhancing students’ cognitive skills. To effectively enhance students’ cognitive skills, it is necessary for future research to extend the intervention period. Additionally, while the meta-analysis confirms that learning analytics-based interventions positively promote social emotion, it does not demonstrate the anticipated higher level of effectiveness, which is consistent with the results of Arguedas et al. (2016) and Y. Lee and Specht (2023). Compared to traditional face-to-face learning, students are more likely to experience negative emotional states (e.g., social anxiety) in online learning settings. (Ifenthaler et al., 2023). Negative emotions can have an adverse impact on students’ learning (Rienties, Boroowa, et al., 2016). For example, improper intervention measures can distract students, disrupt their learning, cause anxiety, reduce engagement, and hinder their learning process (Bannert & Reimann, 2012; Clarebout et al., 2013 ). Given the complexity and diversity of human emotions, automated tools face challenges in accurately capturing and interpreting all emotional states, especially the subtle and complex ones (Ekman, 1999). This challenge is particularly evident when students’ emotional states change dynamically and there is a lack of real-time data and feedback, making it difficult for learning analytics tools to identify and respond to students’ emotional changes in a timely and accurate manner (Baker & Inventado, 2014; Davies et al., 2017; Kort et al., 2001; Rienties, Boroowa, et al., 2016; B. T. M. Wong & Li, 2020). For example, if learning analytics fail to recognize students’ negative emotions during critical moments of rapid emotional change, they may not provide timely support, leading to increased feelings of frustration and disengagement (Mayer, 2019; Reis et al., 2018). Learning analytics-based interventions has limitations in enhancing students’ social-emotional. To enhance students’ social-emotional, future research should employ multi-modal data for affective computing to discern learners’ implicit emotional states, thereby effectively identifying and responding to learners’ negative emotions in a timely manner. Such interventions can help maintain a positive emotional state in students, which can, in turn, improve learning outcomes by addressing the emotional aspects of the learning process more directly.

The Moderating Impact of Variables on Learning Analytics Interventions

A subgroup analysis of potential moderating variables included in 34 experimental studies to investigate further the key factors in learning analytics-based interventions that affect the learning effect. Research revealed that learning analytics-based interventions can improve learning outcomes by regulating variables like subject area, learning stage, intervention duration, learning settings, intervention type, and diagnostic assessment tool.

Regarding subject areas, differences between subjects significantly differ in intervention effects. Among them, the impact of natural sciences fields (such as mathematics, physics, and science) on intervention effects is most significant, consistent with the findings by other researchers. For example, learning analytics-based interventions can make invisible processes visible, and through visual representations, they help learners establish connections between the micro and macro levels, deepening students’ understanding of the micro level and thereby promoting the development of natural science disciplines (Zhang & Linn, 2011).

From the perspective of the learning stage, different learning stages show significant variations in the effectiveness of interventions. Among them, the intervention’s impact has the largest comprehensive effect on elementary school students. This finding is consistent with Mohd Syah et al. (2016) study that early intervention should be initiated from the beginning of school enrollment for children at academic risk to address issues hindering cognitive ability enhancement. This is crucial to prevent long-term cognitive deficiencies and psychological impacts, promoting improved learning outcomes. Therefore, its impact on the intervention effect is the most significant.

Different intervention periods positively affect the improvement of learning outcomes, but the inter-group effects are not statistically significant, consistent with the findings of Zheng et al.’s (2023) study that the intervention duration cannot be considered a decisive factor affecting learning outcomes. Notably, designing intervention measures need to consider the long-term nature of the intervention period to allow students enough time to master cognitive skills (Zimmerman, 2000). However, in teaching practice, interventions are usually conducted in the short term (Heikkinen et al., 2023).

Regarding learning environment, different learning environments show significant variations in the effectiveness of interventions. Among them, the overall impact of applying game-based learning tools is maximized. For example, a game-based learning environment aims to enhance student interest in learning through real-time adaptive support while improving performance or knowledge in specific subjects (Emerson et al., 2023; Kuo, 2007; Troussas et al., 2020). And it can provide students with an engaging and effective learning experience (Emerson et al., 2023). Therefore, its impact on intervention effectiveness is the most significant.

From the perspective of the intervention type, different intervention type showed significant variations in the effectiveness of interventions. Among them, the combined effect value of personalizing course materials is the greatest. For example, Kew and Tasir (2017) conducted a systematic review of 13 relevant intervention studies and found that the most commonly used intervention measure is to assist students at academic risk by providing personalized materials. Furthermore, relevant information to students at academic risk can offer them additional assistance based on their learning needs, thereby increasing their likelihood of learning success and academic performance (Corbi & Burgos, 2014; Kew & Tasir, 2017; Van Barneveld et al., 2012). Therefore, the impact of providing personalization of course materials on intervention effectiveness is the most significant.

Concerning diagnostic assessment tools, different diagnostic assessment tools show significant variations in the effectiveness of interventions. Among them, the overall impact of the standardized test is maximized. Researchers unanimously consider standardized testing the most effective and reliable diagnostic assessment tool (Xu et al., 2023a, 2023b). However, only 23.68% utilized this diagnostic assessment tool, while 76.32% used a self-complied test or scale to assess the intervention effects. This result indicates that assessing learning analytics-based intervention effects is not always effectively conducted using standardized measurement tools. Lonn et al. (2014) argued that in the current context, there is a certain difficulty in integrating scale measurement tools and data sources (e.g., grades) with learning analytics-based interventions. Therefore, future research needs to focus on developing learning analytics techniques to effectively combine measurement tools and data sources, such as academic performance and enriching measurement tools, thus meeting the needs of learners.

The Practical Implications of Learning Analytics-Based Interventions

The meta-analysis results show that learning analytics-based interventions can effectively improve student learning outcomes. Our findings are also useful for teaching practice.

First, although teachers can strengthen relationships between them and students, rethink their teaching performance, and accommodate their curriculum content by employing learning analytics-based interventions (Rienties, Boroowa, et al., 2016), they should not ignore the issue of time costs, especially for those with large class sizes (Corrin et al., 2016; B. T. M. Wong & Li, 2020). This is because when teachers have insufficient time and available resources, they may be constrained from taking proactive intervention measures with students. To maximize the benefits of learning analytics, teachers should prioritize tasks, utilize automation, manage their time effectively, and ensure adequate resource allocation to address the time cost challenges associated with learning analytics-based interventions.

Second, Researchers implement learning analytics-based interventions as a predictive tool to identify students at risk of dropping out, increasing their likelihood of continuing in the program (Arroway et al., 2016; Li et al., 2018). However, they encounter technical challenges in obtaining specific data types for learning analytics practices, such as students’ perceptions, emotions, and teachers’ observations (Gaševic et al., 2017; B. T. M. Wong & Li, 2020). To overcome these challenges, interdisciplinary collaboration with data scientists, psychologists, and computer scientists is essential to develop innovative data collection and analysis methods. Such cooperation can enhance the effectiveness of data utilization, improving the quality and impact of learning analytics-based interventions.

Finally, School administrators implement learning analytics-based interventions to improve student attendance, strengthen teacher-student interactions, and boost student retention rates (Li et al., 2018). Nevertheless, they face challenges in evaluating the effectiveness of these interventions (Li et al., 2018; B. T. M. Wong & Li, 2020). Therefore, it is essential to establish an evidence-based framework for assessing learning analytics-based interventions, enabling school administrators to manage, evaluate, and determine which types of interventions are effective under specific conditions.

Limitations and Directions for Future Research

The meta-analysis has limitations, and future studies could aim to improve and refine the following three aspects. Firstly, due to language constraints, this study was limited to coding literature written in English and Chinese that explores the effects of learning analytics-based interventions on learning outcomes, excluding literature not written in English and Chinese. This may limit the generalizability of this meta-analysis research findings. Future research could compare studies from different linguistic and cultural backgrounds to gain a more comprehensive understanding. Secondly, Although there is a scarcity of empirical studies on the impact of learning analytics-based interventions on learning outcomes, this research indicates that learning analytics-based interventions have a positive effect on cognitive skills and social emotions, even though these interventions have not achieved the expected higher level of effectiveness. Future research should concentrate efforts on investigating and analyzing how learning analytics-based interventions can measure and enhance students’ cognitive skills and social emotion. Lastly, this study examined the moderating variables of intervention period and intervention measures, without analyzing other potential boundary conditions. Future research can further explore the moderating variables of interventions, such as intervention targets, intervention scale, and challenges encountered during the intervention process.

Validity Threats

The validity threats include aspects such as internal, external, construct, and conclusion validity (Shadish et al., 2002). In conducting this meta-analysis, this study minimized the potential risks of validity threats by adhering to the recommended mitigation actions suggested by Cooper (2015). Search terms that are inappropriate or not comprehensive can undermine construct and internal validity. Due to our use of subject terms in the literature search, there is a possibility that we may have overlooked studies that are actually related to learning analytics interventions and learning outcomes, simply because they did not use these keywords. As a result, we not only focused on the keywords but also paid close attention to the content to make sure we didn’t miss any relevant research. Statistically significant results are more likely to be published than papers with non-significant findings, which may affect the estimated overall effect size in meta-analyses, leading to an overestimation of the true treatment effect and impacting external validity. In this study, we conducted a publication bias test for the included studies to identify research that might affect the reliability of the results, which helps to reduce the threat to external validity caused by varying study quality. In order to minimize the risk to conclusion validity, we have discarded the duplicate studies. Meanwhile, to achieve a consensus that minimizes subjective bias and reduces threats to both internal and conclusion validity, the authors engaged in extensive discussions.

Conclusion

The primary goal of this meta-analysis was to address how interventions effectively enhance student learning outcomes, grounded in the analysis of quantified data results. The results showed that learning analytics-based interventions can effectively enhance students’ learning outcomes,with an overall effect value at a moderate level. In terms of specific dimensions of learning outcomes, interventions can significantly and effectively improve knowledge acquisition, while the improvements in social emotions and cognitive skills were relatively small. The research results also indicated that the five moderating variables included in the 34 experimental studies significantly and positively impacted interventions. Subject area, learning stage, learning environment, intervention type, and diagnostic assessment tools are all important moderating factors that influence the impact of interventions on student learning outcomes. Therefore, it is recommended that stakeholders involved in learning analytics-based interventions focus on factors such as subject area, learning stage, learning environment, intervention type, and diagnostic assessment tools when implementing interventions. Furthermore, the present study provides recommendations for subsequent research, to emphasize that future empirical studies need to pay closer attention to cognitive skills and social emotions in order to enhance the overall learning outcomes of students. Additionally, stakeholders involved in designing learning analytics-based interventions should consider the long-term nature of the intervention period to provide students with sufficient time to develop cognitive skills.

Footnotes

We,the undersigned authors,hereby solemnly declare that we have diligently discharged our roles and responsibilities in safeguarding the integrity and precision of the research findings presented in this manuscript. We recognize and commend the contributions of each team member throughout the research endeavor and assert with conviction that each author has played an indispensable part in the successful publication and the advancement of the knowledge encapsulated within this scholarly work.

ORCID iDs

Yan Liu

Wei Wang

Enwei Xu

Author Contributions/CRediT

All authors contributed to the study conception and design. Material preparation,data collection and analysis were performed by Yan Liu and Enwei Xu. The first draft of the manuscript was written by Wei Wang,Yan Liu and Enwei Xu and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

The author(s) disclosed receipt of the following financial support for the research,authorship,and/or publication of this article: This work was supported by General Project of Guangdong Provincial Philosophy and Social Science Planning for the Year 2024 in China;Research and Practice on Promoting High-Quality Development in Basic Education through the Construction of New Teacher Training Programs in Guangdong Province for the Year 2023;The 2024 Think Tank Bidding Project of Xinjiang Normal University named “Research on the Practical Model of Educational Digitalization Enabling Teacher Education Students to Collaborate in Deepper Learning” (Grant number No. XJEDU2024J148);The 2023 Xinjiang Uygur Autonomous Region “14th Five Year Plan” Key Discipline Education Tender Project “Research on Collaborative Deepper Learning Model for Undergraduate Education Technology Majors from the Perspective of Teacher Education Certification” (Grant number No. 23XJKD0205);The research on the Design and Application of Collaborative Deep Learning Activities Supported by Intelligent Technology in the Tianchi Talent Program of Xinjiang Uygur Autonomous Region (Youth Doctoral Program).

Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

Arguedas

Daradoumis

Xhafa

(2016). Analyzing the effects of emotion management on time and self-management in computer-based learning. Computers in Human Behavior, 63, 517–529.

Arroway

Morgan

Keefe

Yanosky

M. R.

(2016). Learning analytics in higher education. Research report. ECAR, EDUCAUSE.

Baker

R. S.

Inventado

P. S.

(2014). Educational data mining and learning analytics. In Spector

J. M.

Merrill

M. D.

Elen

Bishop

M. J.

(Eds.), Handbook of research on educational communications and technology (pp. 103–117). Springer.

Bannert

Reimann

(2012). Supporting self-regulated hypermedia learning through prompts. Instructional Science, 40, 193–211.

Bannert

Sonnenberg

Mengelkamp

Pieger

(2015). Short- and long-term effects of students’ self-directed meta-cognitive prompts on navigation behavior and learning performance. Computers in Human Behavior, 52, 293–306.

Brown

(2011). Learning analytics: The coming third wave. Educause learning initiative brief, 1(4), 1–4.

Clarebout

Elen

Collazo

Lust

Lai

(2013). Meta-cognition and the use of tools. In Azevedo

Aleven

(Eds.), In International Handbook of meta-cognition and learning technologies (pp. 187–195). Springer.

Clow

(2012). The learning analytics cycle: Closing the loop effectively [Conference session]. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 134–137).

Cohen

(1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed., pp.289–354). Lawrence Erlbaum Associates.

10.

Cooper

(2010). Research synthesis and meta-analysis:A step-by-step approach (4th ed., pp.145–196). Sage.

11.

Cooper

(2015). Research synthesis and meta-analysis: A step-by-step approach (Vol. 2, pp. 1407–1452). Sage publications.

12.

Corbi

Burgos

(2014). Review of current student-monitoring techniques used in e-learning focused recommender systems and learning analytics: The Experience API & LIME model case study. International Journal of Interactive Multimedia and Artificial Intelligence, 2(7), 44–52.

13.

Corrin

Kennedy

de Barba

P. G.

Lockyer

Gaševic

Williams

Bakharia

(2016). Completing the loop: Returning meaningful learning analytic data to teachers: A Handbook for Educators and Learning Analytics specialists. Australian Government-Office for Learning and Teaching. http://melbourne-cshe.unimelb.edu.au/__data/assets/pdf_file/0006/2083938/Loop_Handbook.pdf.

14.

Darling-Hammond

Flook

Cook-Harvey

Barron

Osher

(2019). Implications for educational practice of the science of learning and development. Applied Developmental Science, 24, 97–44.

15.

Davies

Nyland

Bodily

Chapman

Jones

Young

(2017). Designing technology-enabled instruction to utilize learning analytics. TechTrends, 61(2), 155–161.

16.

Ekman

(1999). Basic emotions. In Dalgleish

Power

M. J.

(Eds.), Handbook of cognition and emotion (pp. 45–60). John Wiley & Sons.

17.

Emerson

Min

Rowe

(2023). Multi-modal predictive student modeling with multi-task [Conference session]. Proceedings of the sixth International Conference on Learning Analytics and Knowledge (pp. 534–535).

18.

Gaševic

Dawson

Pardo

(2017). How do we start? State and directions of learning analytics adoption. International Council for Open and Distance Education.

19.

Glass

G. V.

(1976). Primary, secondary and meta-analysis of research. Educational Researcher, 5(10), 3–8.

20.

Gong

L. L.

Liu

(2019). Design and application of intervention model based on learning analytics under blended learning environment [Conference session]. Proceedings of the 7th International Conference (pp. 225–229). ACM.

21.

Hedges

L. V.

(1981). Distribution theory for glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6(2), 107–128.

22.

Hedges

L. V.

Vevea

J. L.

(1998). Fixed- and random effects models in meta-analysis. Psychological Methods, 3(4), 486–504.

23.

Heikkinen

Saqr

Malmberg

Tedre

(2023). Supporting self-regulated learning with learning analytics interventions: A systematic literature review. Education and Information Technologies, 28, 3059–3088.

24.

Ifenthaler

Cooper

Daniela

Sahin

(2023). Social anxiety in digital learning environments: An international perspective and call to action. International Journal of Educational Technology in Higher Education, 20(1), 18.

25.

Jayaprakash

S. M.

Moody

E. W.

Lauría

E. J.M.

Regan

J. R.

Baron

J. D.

(2014). Early alert of academically at-risk students: An open source analytics initiative. Journal of Learning Analytics, 1(1), 6–47.

26.

Karaoglan Yilmaz

F. G.

Yilmaz

. (2022). Learning Analytics intervention improves students’ engagement in online learning. Technology Knowledge and Learning, 27, 449–460.

27.

Kew

S. N.

Tasir

(2017). A systematic review of learning analytics intervention contributing to student success in online learning [Conference session]. 2017 International Conference on Learning and Teaching in Computing and Engineering (pp. 62–67).

28.

Kew

S. N.

Tasir

(2021). Learning analytics in online learning environment: A systematic review on the focuses and the types of student-related analytics data. Technology Knowledge and Learning, 27, 405–423.

29.

Khalil

Ebner

(2015). Learning analytics: Principles and constraints. Edmedia+ innovate learning. Association for the Advancement of Computing in Education (AACE) (pp. 1789–1799).

30.

Kim

I. H.

Park

(2016). Effects of learning analytics dashboard: analyzing the relations among dashboard utilization, satisfaction, and learning achievement. Asia Pacific Education Review, 17, 13–24.

31.

Knobbout

Van Der Stappen

(2020). Where is the learning in learning analytics? A systematic literature review on the operationalization of learning-related constructs in the evaluation of learning analytics interventions. IEEE Transactions on Learning Technologies, 13(3), 631–645.

32.

Kort

Reilly

Picard

R. W.

(2001). An affective model of interplay between emotions and learning: reengineering educational pedagogy—Building a learning companion [Conference session]. Proceedings IEEE international conference on advanced learning technologies (pp. 43–46). IEEE.

33.

Kuo

M. J.

(2007). How does an online game based learning environment promote students' intrinsic motivation for learning natural science and how does it affect their learning outcomes? [Workshop]. 2007 First IEEE International Workshop on Digital Game and Intelligent Toy Enhanced Learning (DIGITEL'07) (pp. 135–142).

34.

Kyndt

Raes

Lismont

Timmers

Cascallar

Dochy

(2013). A meta-analysis of the effects of face-to-face cooperative learning. Do recent studies falsify or verify earlier findings? Educational Research and Reviews, 10, 133–149.

35.

Lee

Watson

S. L.

Watson

W. R.

(2019). Systematic literature review on self-regulated learning in massive open online courses. Australasian Journal of Educational Technology, 35(1), 28–41.

36.

Lee

Specht

(2023). Can we empower attentive e-reading with a social robot? An introductory study with a novel multimodal dataset and deep learning approaches [Conference session]. Proceedings of International Conference on Learning Analytics and Knowledge (pp. 520–530).

37.

K. C.

C. J.

Wong

B. T.

(2018). Status of Learning analytics in asia:perspectives of higher education stakeholders. In Cheung

S. K. S.

J. L.

W. K.

W. S.

(Eds.), In Technology in education: Innovative solutions and practices (pp. 267–275).

38.

Lim

L. A.

Gentili

Pardo

Kovanovica

Whitelock-Wainwrightb

Gaševicc

Dawson

(2019). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction, 72, 1–11.

39.

Lipsey

Wilson

(2000). Practical meta-analysis (pp. 92–160). International Educational and Professional.

40.

Long

P. D.

Siemens

(2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 31–40.

41.

Lonn

Aguilar

S. J.

Teasley

S. D.

(2014). Investigating student motivation in the context of a learning analytics intervention during a summer bridge program. Computers in Human Behavior, 47, 90–98.

42.

O. H. T.

Huang

J. C. H.

Huang

A. Y. Q.

Yang

S. J. H.

(2017). Applying learning analytics for improving students engagement and learning outcomes in an MOOCs enabled collaborative programming course. Interactive Learning Environments, 25, 220–315.

43.

Mayer

R. E.

(2019). Searching for the role of emotions in e-learning. Learning and Instruction, 70, 101213.

44.

Mohd Syah

N. E.

Hamzaid

N. A.

Murphy

B. P.

Lim

. (2016). Development of computer play pedagogy intervention for children with low conceptual understanding in basic mathematics operation using the dyscalculia feature approach. Interactive Learning Environments, 24(7), 1477–1496.

45.

Nussbaumer

Hillemann

E. C.

Gütl

Albert

(2015). A competence-based service for supporting self-regulated learning in virtual environments. Journal of Learning Analytics, 2(1), 101–133.

46.

Reis

R. C. D.

Isotani

Rodriguez

C. L.

Lyra

K. T.

Jaques

P. A.

Bittencourt

I. I

. (2018). Affective states in computer-supported collaborative learning: Studying the past to drive the future. Computers & Education, 120, 29–50.

47.

Rienties

Boroowa

Cross

Farrington-Flint

Herodotou

Prescott

Mayles

Olney

Toetenel

Woodthorpe

(2016). Reviewing three case-studies of learning analytics interventions at the Open University UK [Conference session]. Proceedings of the sixth International Conference on Learning Analytics and Knowledge (pp. 534–535).

48.

Rienties

Cross

Zdrahal

(2016). Implementing a learning analytics intervention and Evaluation Framework: what works? In Motidyang

Butson

(Eds.), Big data and learning analytics in higher education: Current theory and practice (pp. 147–166). Springer.

49.

Rothstein

H. R.

(2008). Publication bias as a threat to the validity of meta-analytic results. Journal of Experimental Criminology, 4, 61–81.

50.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Company.

51.

Siemens

Baker

R. S. J.

(2012). Learning analytics and educational data mining: Towards communication and collaboration [Conference session]. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 252–254).

52.

Smith

V. C.

Lange

Huston

D. R.

(2012). Predictive modeling to forecast student outcomes and drive effective interventions in online community college courses. Journal of Asynchronous Learning Networks, 16(3), 51–61.

53.

Society of Learning Analytics Research. (n.d). About [Web page]. http://www.solaresearch.org/mission/about/.

54.

Sønderlund

A. L.

Hughes

Smith

P. J.

(2018). The efficacy of learning analytics interventions in higher education: A systematic review. British Journal of Educational Technology, 50, 2594–2640.

55.

Tepgeç

Ifenthaler

(2022) Learning analytics based interventions: A systematic review of experimental studies. International Association for Development of the Information Society, 327–330.

56.

Troussas

Krouska

Sgouropoulou

(2020). Collaboration and fuzzy-modeled personalization for mobile game-based learning in higher education. Computers & Education, 144, 1–18.

57.

Van Barneveld

Arnold

K. E.

Campbell

J. P

. (2012). Analytics in higher education: Establishing a common language. EDUCAUSE learning initiative, 1(1), l–ll.

58.

Viberg

Hatakka

Bälter

Mavroudi

(2018). The current landscape of Learning Analytics in Higher Education. Computers in Human Behavior, 89, 98–110.

59.

Williams

(2014). Squaring the circle: A new alternative to alternative-assessment. Teaching in Higher Education, 19(5), 565–577.

60.

Wise

A. F.

(2014). Designing pedagogical interventions to support student use of learning analytics [Conference session]. Proceedings of the 4nd International Conference on Learning Analytics and Knowledge (pp. 203–211).

61.

Wong

B. T. M.

(2017). Learning analytics in higher education: An analysis of case studies. Asian Association of Open Universities Journal, 12(1), 21–40.

62.

Wong

B. T. M.

K. C.

(2020). A review of learning analytics intervention in higher education (2011-2018). Journal of Computers in Education, 7(1), 7–28.

63.

Wong

Baars

Davis

Van Der Zee

Houben

G. J.

Paas

(2019). Supporting self-regulated learning in online learning environments and MOOCs: A systematic review. International Journal of Human-Computer Interaction, 35(4–5), 356–373.

64.

Wang

(2023a). A meta-analysis of the effectiveness of programming teaching in promoting K-12 students’ computational thinking. Education and Information Technologies, 28, 6619–6644.

65.

Wang

(2023b). The effectiveness of collaborative problem solving in promoting students’ critical thinking: A meta-analysis based on empirical literature. Humanities and Social Sciences Communications, 10, 1–11.

66.

J. M.

Zhou

(2022). Exploring the relationship between learning sentiments and cognitive processing in online collaborative learning: A network analytic approach. Internet and Higher Education, 55, 1–11.

67.

Zhang

Z. H.

Linn

M. C.

(2011). Can generating representations enhance learning with dynamic visualizations? Journal of Research in Science Teaching, 48(10), 1177–1198.

68.

Zheng

Fan

Gao

Huang

(2023). The impact of learning analytics interventions on learning achievements: A meta-analysis of research from 2012 to 2021. Interactive Learning Environments, 32, 6267–6316.

69.

Zimmerman

B. J.

(2000). Attaining self-regulation: A social cognitive perspective. In Boekaerts

Pintrich

P. R.

Zeidner

(Eds.), Handbook of self-regulation (pp. 13–40). Academic Press.

The Effectiveness of Learning Analytics-Based Interventions in Enhancing Students’ Learning Effect: A Meta-Analysis of Empirical Studies

Abstract

Keywords

Introduction

Literature Review

Learning Analytics-Based Interventions Definition and Effectiveness Dimensions

Factors in the Effectiveness of Learning Analytics-Based Interventions

Method

Literature Search Strategy

Inclusion Criteria and Exclusion Criteria

Coding of Study Characteristics and Moderator Variables

Analysis of Data

Assessment of Publication Bias and Heterogeneity

Publication Bias Test

Heterogeneity Test

Results

An Examination of the Total Effect Size

Analysis of Moderator Effect Size

Subject Area

Learning Stage

Intervention Duration

Learning Environments

Intervention Type

Diagnostic Assessment Tool

Discussion

The Impact of Learning Analytics-Based Interventions on Students' Learning Outcomes

The Moderating Impact of Variables on Learning Analytics Interventions

The Practical Implications of Learning Analytics-Based Interventions

Limitations and Directions for Future Research

Validity Threats

Conclusion

Footnotes

ORCID iDs

Author Contributions/CRediT

Funding

Conflicting Interests

Data Availability

References