Abstract
Keywords
Introduction
Worldwide, mental health problems affect 10–20% of children and adolescents (Kieling et al., 2011). Yet, while effective mental health interventions for youth are available (Weisz et al., 2006), the majority of affected adolescents do not receive treatment (Merikangas et al., 2011). As adolescents spend a substantial part of their time at school (Eccles & Roeser, 2011), the school environment seems particularly suitable to reach and support adolescents with emerging mental health problems. In the last decades, school-based intervention programs have become increasingly popular, and meta-analytic reviews have demonstrated that school-based intervention programs—particularly targeted programs aimed at adolescents in need—have the potential to improve adolescent mental health, including physiological and psychological stress, social and emotional skills, internalizing problems, and behavior problems (Durlak et al., 2011; Feiss et al., 2019; van Loon et al., 2020, 2022; Werner-Seidler et al., 2017).
Although targeted school-based intervention programs promoting mental health seem worthwhile to support adolescents, successful implementation and evaluation of such programs are considered challenging (Gee et al., 2021). Previous literature showed challenges and issues in the evaluation and implementation of school-based mental health intervention programs relating to logistics (e.g., constraints regarding timetables, physical school environment), recruitment of schools and participants, parental consent, privacy, and stigma (Gee et al., 2021; Jaycox et al., 2006). However, these studies did not randomly allocate individuals to either an intervention or control group. Randomized controlled trials (RCTs) are considered the “gold standard” for effectiveness studies because randomization reduces bias (Hariton & Locascio, 2018). Several studies examined the implementation and evaluation process of RCTs, demonstrating general and specific challenges, including resistance to random allocation, challenges regarding recruitment and retention of organizations and participants, and high staff turnover (Asscher et al., 2007; Prinz et al., 2001). Yet, these RCTs were not conducted in a school setting. Notably, despite the advantages of RCTs, educational researchers often choose less robust designs (Brady et al., 2023). This warrants the need for more information about (overcoming) the challenges of implementing and evaluating school-based intervention programs with RCT designs. In this paper, we address exactly this gap in the literature.
Some of these implementation and evaluation challenges may apply to effectiveness studies in general, such as creating and maintaining productive and positive collaborations with involved parties, participant recruitment and retention, and successful program implementation. Other challenges are more “intervention at school” specific, such as scheduling the intervention and the accompanied effectiveness study or “RCT,” such as random assignment of students. Overall, these challenges can threaten the robustness of study findings and thereby hamper our knowledge of what works and for whom. Furthermore, few studies pay attention to the implementation process of intervention programs (Perepletchikova et al., 2007), even though successful implementation may be essential for program effectiveness (Lane et al., 2004). More knowledge about the implementation and evaluation process of intervention studies is thus necessary.
Between 2018 and 2021, we conducted two RCTs to investigate the effectiveness of two targeted school-based skills-training programs promoting adolescent mental health, by addressing either school or social stress. Even though RCTs are considered the “gold standard” for effectiveness studies (Hariton & Locascio, 2018), a recent meta-analysis demonstrated that only half of included studies examining the effectiveness of school-based intervention programs conducted (cluster) RCTs (van Loon et al., 2020). Practical reasons (e.g., cooperation with involved parties, randomization process) might demotivate researchers to use an RCT design to evaluate the effectiveness of school-based intervention programs. Because we believe our experiences may be useful and relevant for others planning to study the effectiveness of school-based intervention programs, to ultimately improve the evidence base for supporting adolescents with emerging mental health needs in the school environment, we present the challenges encountered and lessons learned. Moreover, we hope to motivate others to conduct an RCT, which may subsequently pave the way for future RCTs (Asscher et al., 2007), thereby advancing the quality of knowledge on program effectiveness.
Background of Our Project
As part of a larger project aimed at strengthening the link between secondary education and youth mental health care, we implemented two school-based skills-training programs in nine secondary schools in the Netherlands and conducted two RCTs to examine their effectiveness (see van Loon et al., 2019 for the study protocol). This project was performed in the context of a Response to Intervention (RtI) model—a three-tiered approach with universal, targeted, and intensive interventions—focusing on identifying vulnerable adolescents and providing them with suitable (preventive) interventions (Kearney & Graczyk, 2014). Stress was chosen as an overarching topic, as stress has been associated with various mental health problems (Snyder et al., 2017). Moreover, the schools involved in the project expressed a need for supporting their students in adaptively coping with school stress (e.g., exams) and social stress (e.g., issues with peers), which were frequently observed. Recent research has also demonstrated that the most salient stressors of adolescents are related to school and social situations (Anniko et al., 2019; Núñez-Regueiro & Núñez-Regueiro, 2021). Adolescents in the first, second, and third years (equivalent to USA 7th, 8th, and 9th-grade students) of the participating secondary schools first received an educational program about stress (i.e., tier 1 of the RtI model: universal intervention for all students). After these three lessons, which were delivered to the entire class, adolescents were offered the opportunity to participate in one of the skills-training programs (i.e., tier 2 of the RtI model: targeted intervention for a selection of students), targeting either school or social stress by addressing skills to deal with performance anxiety or social skills, respectively (van Loon et al., 2019). Adolescents self-selected into one of the programs, and after obtaining active informed consent from both adolescents and parents, they were randomly assigned to an intervention (i.e., experimental) or waitlist control group. The programs were evaluated in a mixed-ethnic identity sample of adolescents attending different educational levels (i.e., from practical to preuniversity education). The programs were offered by three external youth care organizations and consisted of seven weekly sessions of 45 min. This project (i.e., tier 2) was conducted in the school years 2018–2019, 2019–2020, and 2020–2021. The effectiveness results of the programs are reported elsewhere (van Loon et al., 2023). Tier 3 of the RtI model contains intensive (individual) interventions directed at adolescents with severe and/or complex problems, which was outside the scope of our project.
Encountered Challenges and Lessons Learned
Maintaining Cooperation and Participation of Involved Parties: Challenges
Creating and maintaining positive relationships with external facilitators of the intervention programs and school staff, as well as between them, is important for successful program implementation (Gee et al., 2021). However, establishing productive and positive collaborations with involved parties can be complicated, especially with many parties, as each party has its own priorities and interests. In our project, some schools invested more in this project than others, for example by allowing teachers to spend time to support the project (e.g., by helping the researchers to collect informed consent forms). Furthermore, diversity in terms of school population, size, and policies necessitated taking sufficient time to attend to each school’s characteristics. Because of the large number of participating schools and youth care organizations, investing enough time in establishing relationships was a challenge. Another challenge we encountered was the unfamiliarity of involved parties, particularly schools, with (randomized controlled) research. Some contact persons did not fully grasp the goals and requirements of a controlled research design (e.g., were not convinced of the necessity of having an experimental and control group) and randomization process (e.g., believed that adolescents with more pressing problems should get the program first). Unfortunately, due to a combination of factors, one school dropped out of the project.
Maintaining Cooperation and Participation of Involved Parties: Lessons Learned
To establish cooperative collaborations while taking the diversity among schools into account, we realized that investing in a customized approach for each school was required. Each school selected a youth care organization they believed was the best match for their student population and entered into a partnership. We collaborated with the matched schools and youth care organizations to develop school-specific protocols for informing school staff, adolescents, and parents about the programs and project logistics (e.g., when and how to send information, when and where to schedule program sessions and assessments). Hence, school-specific protocols differed depending on the needs of the (contact persons of the) schools. For instance, in some schools, parents were informed about this project with information letters sent by email, whereas in other schools, the project was also introduced on information evenings or in school newsletters. This way, we aimed to attune as much as possible to the regular methods for working with adolescents at the specific school within the boundaries of our standardized approach for this project. In the end, only minor aspects differed between schools (e.g., number of emails sent to parents, information in school newsletter), ensuring internal and external validity of the project (Wagner et al., 2004). The partnership and co-creation in the development of school-specific protocols also strengthened the collaboration between schools and youth care organizations. For example, some schools mentioned that because of this project, they knew better how to find their way to youth mental health care and contacted the organization for other issues (e.g., help for individual students, other programs).
To create support for adhering to the design of our project, we invested time in explaining and discussing the controlled research design and the significance of the randomization process. We experienced that making plans together and discussing important topics stimulated the support and active contribution of the involved parties to all the steps of the process. Nevertheless, at one school this was not sufficient, given that the school stopped participation; we were unable to reach mutual agreement on various aspects of the implementation process (possibly reinforced by the COVID-19 pandemic).
To maintain close contact as well as foster informal contact, the primary researcher was on-site at least once a week. In addition, contact persons were asked to contact the primary researcher as early as possible when problems occurred, and the researchers made sure to be available to discuss potential solutions and act immediately if needed. Moreover, clear communication and keeping short lines of communication, by informing school staff and youth care organizations about every step of the process (i.e., keeping them up-to-date) and providing them with the appropriate information on time, were important for creating and maintaining positive and fruitful collaborations. We encountered that by being helpful, available, flexible, and communicative, the required time investment for schools (and youth care organizations) was minimized, which is important, as many secondary school teachers present high levels of burnout symptoms (García-Carmona et al., 2019). Overall, it is important to invest sufficient time in establishing positive relationships with involved parties to improve support and participation in the project, by keeping tight management and up-to-date administration about the process at each specific school.
Reaching the Target Group: Challenges
Effective recruitment is essential for successful program implementation, as recruitment is the first opportunity to contact and engage participants (Bruzzese et al., 2009). Furthermore, to properly investigate the effectiveness of an intervention program, it is necessary to include a sufficient number of participants to ensure there is enough power to detect potential statistical differences (Umscheid et al., 2011). Yet, reaching and recruiting participants is difficult to achieve. We encountered several challenges in attempting to reach and recruit adolescents (and parents) to enroll in the programs and participate in research. First, one challenge was the proper distribution of information and making sure that the provided information was clearly understood by adolescents and parents. We found that parents did not always receive or read the provided information, and therefore missed crucial details. Furthermore, the provided information was not always clearly understood by adolescents. That is, adolescents thought the programs were similar to the educative lessons (i.e., Tier 1) or had not understood the focus of the programs. Furthermore, language issues, as well as the relatively long and complex information form (required part of consent procedure) might have hindered parents’ (and adolescents’) clear understanding of the provided information.
Second, we encountered that involving parents as informants in the research (i.e., data collection) was a major challenge, and due to low response rates, we decided to remove this component. Although parent involvement is desirable, as it provides additional perspectives besides that of the adolescent, it is often difficult to achieve (Gee et al., 2021).
Third, motivating adolescents to participate was challenging, as reflected by low recruitment rates in some schools. Low levels of recruitment might be related to stigma associated with (mental health) programs (Gee et al., 2021) or mistrust or unfamiliarity regarding research in some populations (e.g., families with lower socioeconomic status; Bonevski et al., 2014). Study participation was conditional upon provided informed consent by adolescents and parents. Yet, for the collection of signed consent forms, we were dependent on the willingness of adolescents and parents to return the forms, but above all on the dedication and determination of school staff to collect them. Some schools were more successful than others, as these school liaisons, mentors, or teachers were more dedicated to the project. For instance, some school personnel contacted students (and their parents) multiple times about this project, while others did not.
Reaching the Target Group: Lessons Learned
In our project, to reach and motivate adolescents (and parents) to enroll in the programs and participate in research, we used multiple strategies. First, to increase the chance that adolescents received and understood the information, they received information in different ways and from multiple distributors. Adolescents received information on paper from researchers or school personnel as well as a presentation in their classroom from their mentor and/or one of the researchers or trainers including a short movie of the trainers explaining the content of the programs. Parents also received information in multiple ways. We designed a project website (targeting parents in particular), with information about the programs provided in text and in a video as well as with a list of frequently asked questions. Moreover, we organized general information evenings at school, gave adolescents written information (to take home), and emailed parents (via the school). Additionally, in some schools, parents were contacted by telephone to explain the value, aims, and purpose of the study (if necessary, family members were asked to translate the information to the parent), which increased recruitment rates. To prevent misunderstandings due to language issues, we tried to make the information form as simple as possible, including making the documents short and brief and adding pictograms next to the written text.
Second, this project was performed in the context of an RtI model, and students received psychoeducation about stress (“Stress Lessons”) before they were offered the programs. The main goal of the Stress Lessons was to teach students about stress (e.g., how to recognize stress and how to reduce stress), to increase awareness about experiencing stress, and to encourage students to talk about stress. Hence, in these lessons, we tried to frame experiencing stress as something that happens to everyone and to convey the importance of expressing feelings of stress. We aimed—by talking and teaching about stress—to “normalize” talking about stress and worries and to prevent stigma about signing up for a follow-up (mental health) intervention to reduce stress. Previous research showed that psychoeducation programs reduced the stigma around mental health problems in educational settings (Han & Chen, 2014; Waqas et al., 2020). Making adolescents more aware of mental health issues (such as stress and its consequences) as well as increasing their knowledge about mental health issues might reduce mental health stigma and increase help-seeking. Additionally, by framing the programs as skills-training programs focusing on school stress and enhancing social skills, we tried to minimize stigma and make the programs as relevant, approachable, and low-threshold as possible. This way, we aimed to increase the motivation of adolescents to participate (and parents to consent).
Third, to enhance recruitment and the collection of signed consent forms, we organized meetings with mentors and teachers to ask for their help in motivating and enthusing adolescents to participate and to emphasize the importance of their involvement. It is important to solicit the involvement of mentors and teachers in the project, as they can help to motivate adolescents to participate. Furthermore, to make giving consent as easy as possible, adolescents and parents could give informed consent digitally as well as on paper. In most schools, we received more digital than paper forms, which suggest that giving digital consent is preferred by adolescents and parents.
Retaining Participants: Challenges
Once participants are recruited, the next challenge is making sure they actually attend the intervention and data collection sessions, because along with the way, some participants might lose interest, become less motivated, or even drop out. Retention of participants is important, as high attrition can negatively affect the intervention as well as the quality of an effectiveness study (e.g., by creating nonrepresentative groups or reducing statistical power; Prinz et al., 2001). In our case, attendance regarding the intervention programs and research required constant attention. With regard to the programs, we experienced that not all adolescents were (intrinsically) motivated to attend (all) the sessions, possibly because some information was not properly understood by adolescents or parents. For example, some adolescents had different expectations, had registered because they believed participation was obligatory, or were registered by their parents without their full support for that decision. Regarding the research, the absence of adolescents during the assessments due to illness, other obligations, or lack of motivation, posed a challenge. Furthermore, we noted that the questionnaires were long and difficult, especially for younger adolescents or adolescents with lower educational levels, possibly lowering motivation to continue participation.
Retaining Participants: Lessons Learned
In our project, to prevent dropout regarding intervention and research, several strategies were used. With regard to the programs, before adolescents started the first session, we explained the relevance of the programs and answered questions from adolescents (i.e., during the preintervention measurement). During program implementation, trainers strived to make the sessions as attractive as possible (e.g., variety of exercises, including psychoeducation and skill-building, interactive format) and motivated adolescents throughout the sessions (e.g., by investing in building relationships with the participants). Furthermore, the primary researcher kept a log to monitor program attendance, and if adolescents were not present, they were contacted and alerted by school staff or researchers, which increased program attendance. Regarding the assessments, adolescents were invited in small groups, with at least one researcher present, so they could receive support and help at an individual level. Moreover, adolescents filled out the questionnaire online (i.e., on a phone, tablet, or laptop) and during school hours (i.e., not in their own free time). In addition, we found that at least one week (i.e., dependent on the number of participants) needed to be scheduled for the assessments per school and that a sufficient number of researchers must be available to come (back) to that school during this period. This way, adolescents who were absent during the planned assessment could be reached at another moment later that week, thereby preventing dropout.
Overcoming Logistical and Planning Issues: Challenges
To facilitate effective implementation and evaluation of targeted school-based intervention programs, working within constraints imposed by the school environment (e.g., timetables, holidays, or school exam weeks, outings, or other events), is important (Gee et al., 2021). However, accommodating logistical issues can be challenging. First, in our case, planning the program sessions and assessments was complicated due to the different schedules of adolescents (i.e., adolescents from different classes participated), the availability of sufficiently private spaces in school, and the timing of the program sessions (i.e., during or after school). Second, timely communication about planning and logistical issues is also a key. In our project, trainers indicated that they sometimes received the schedule of the program sessions late or were unaware in which room they had to give the session, which could hinder the program sessions. Additionally, adolescents sometimes missed a program session because of the cancelation of class lessons that day (and they did not want to wait for the session) or because the session was planned parallel to a class lesson they did not want to miss. Third, a challenge was the high staff turnover among the student research assistants, who were involved in the data collection. Some student research assistants became less motivated over time or had to quit because they graduated and obtained a job.
Overcoming Logistical and Planning Issues: Lessons Learned
First, to minimize problems regarding logistics and planning, we drafted the logistic procedure in collaboration with the involved parties weeks before the start of the recruitment phase. Second, we strived to inform adolescents (via an electronic learning environment for schools) and trainers well in advance about the location and schedule of the program sessions. Involved parties were kept up-to-date about the actual status of the schedule at least once a week, in person or via phone, during the course of the whole project. Moreover, if there were any obstacles (e.g., absence of a trainer, change in schedule), the involved parties were notified immediately, and a new date was planned and communicated. Third, to prevent understaffing and to provide flexibility during the data collection, we hired a large team of student research assistants to support the data collection assessments.
COVID-19 Pandemic: Challenges
An unexpected challenge in the implementation and evaluation of the skills-training programs in schools was the worldwide outbreak of COVID-19. Due to government measures, secondary schools in the Netherlands were suddenly closed to prevent the spreading of the virus (i.e., March 2020). The majority of schools in our project were in the middle of the recruitment or implementation phase and had to be canceled. We decided to (re)start the project six months later (i.e., from September 2020), necessitating rescheduling of the programs and assessments again, which demanded a lot of the adolescents, schools, youth care organizations, and researchers. Moreover, participants who already completed the preintervention assessment needed to complete the questionnaire again to obtain accurate preintervention data. Unfortunately, restarting the programs resulted in dropout from the intervention (not the research). Half of the participants for which the program was postponed did not attend any session of the performance anxiety program (48%) and a quarter of participants did not attend any session of the social skills program (23%). It is possible that participants dropped out because they were not motivated anymore, felt they did not need the support anymore, or did not want to miss more classes.
COVID-19 Pandemic: Lessons Learned
The high dropout after a postponement of the intervention programs emphasizes the necessity of registering and offering the program in a timely matter. That is, programs should be offered as soon as possible after participants sign up. The longer the time between registration and the start of the program, the greater the chance that participants will not want to attend the program anymore (e.g., forget what they signed up for, no longer have the time). Furthermore, in light of the COVID-19 pandemic, both program providers and researchers should have online or hybrid options for intervention programs and data collection, respectively, readily available. This enables them to flexibly respond to lockdowns with school closings and, for program providers, to provide timely help to those who were willing to participate. The ability to flexibly shift from offline to online modalities and back may be necessary to make studies more resilient in the face of sudden environmental challenges, such as the COVID-19 pandemic. An alternative online option for school-based intervention programs and research may prevent cancelation and participant dropout. Yet, although online programs can have a positive effect on adolescent mental health, moderate to high rates of noncompleters have been found (Clarke et al., 2015). Hence, further research examining the effectiveness of and adherence to online programs is warranted.
Discussion
In the present paper, we describe the implementation process of two school-based skills-training programs and accompanying RCTs in a diverse (i.e., mixed ethnic identity and educational level) sample of adolescents. Based on our experiences, we have several recommendations for researchers planning to implement and evaluate targeted school-based intervention programs promoting adolescent mental health (Table 1). Our experiences, as well as those of involved parties, underscore that it is possible to effectively implement and evaluate skills-training programs with RCT designs in schools, but that it requires investing time, maintaining close contact and clear communication with and between involved parties (e.g., adolescents, school personnel, parents, trainers), and a research team characterized by flexibility and a problem-solving attitude. This means that researchers need to be “on-call” to detect potential problems during the whole process, which requires tight management and up-to-date administration at each specific school and routine communication with individual participants. In case of any obstacle, researchers should immediately contact the involved parties and propose solutions. Overall, flexibility and close contact are the key ingredients of successful program implementation and effectiveness research. In addition, as no school is the same, and every school presents unique opportunities as well as challenges (Wagner et al., 2004), it is important to provide tailored plans regarding implementation and evaluation, adjusting to the needs of each individual school and its unique population.
Recommendations for Successful Implementation and Evaluation of Targeted School-Based Intervention Programs Promoting Adolescent Mental Health.
Although RCTs are considered the “gold standard” for intervention research, as randomization minimizes the risk of confounding bias (Hariton & Locascio, 2018), there are some limitations. Practitioners might be resistant to (individual) randomization, for instance, because it reduces their autonomy or because they feel it is unethical to withhold an intervention from a group of participants. Furthermore, RCTs are time- and energy-intensive and expensive (Tomlinson et al., 2015). Consequently, RCTs are not always the best option, depending on the context and quality of the research (Deaton & Cartwright, 2018). That is, a poorly done or failed RCT might not generate more robust results than a quasi-experimental study (Deaton & Cartwright, 2018). Alternative options may be considered when randomization is not possible (e.g., nonrandomized cohort study) (Tomlinson et al., 2015). Yet, it is important that school-based intervention programs are examined using high-quality research designs to assess their effectiveness (Deaton & Cartwright, 2018; Tomlinson et al., 2015). Our experiences indicate that it is possible to examine school-based intervention programs with RCTs and that a randomized controlled design is an option in school-based intervention research. Successful implementation and evaluation of school-based intervention programs require clear communication, close and regular contact, flexibility, and the development of school-specific protocols tailored to the needs of each school and their unique population. We would like to stress the importance of paying attention to the implementation process of intervention programs in effectiveness research. Although successful implementation may be crucial for program effectiveness, few studies present explicit and comprehensive information about this. Despite the challenges, it is important that school-based intervention programs are effectively implemented to improve adolescents’ mental health. Schools are an ideal environment to reach large groups of adolescents and offer preventive care, particularly for adolescents who are reluctant to search for care outside of school. Moreover, tackling (and thus limiting) challenges of program implementation is essential for learning what works and for whom (i.e., improving care and support for adolescents) as well as for facilitating the sustainability of such programs.
Footnotes
Declaration of conflicting interests
Funding
Ethical approval
.ORCID iD