Sage Journals: Discover world-class research

Abstract

Background. Integration of autism screening into primary care practice in Saudi Arabia is not well established. Objectives. To evaluate the feasibility and effectiveness of implementing the Arabic Modified Checklist for Autism in Toddlers (M-CHAT) in a primary care practice at John Hopkins Aramco Healthcare Center in Saudi Arabia. Method. The Arabic version of M-CHAT was distributed to caregivers of 1207 toddlers (16-32 months) from January to December 2014. Feasibility was assessed by measuring the proportion of visits with M-CHAT completed, and reports of workflow challenges and provider satisfaction. The effectiveness of screening was evaluated based on the number of referrals for autism evaluation and autism identification rates. Results. Total M-CHAT completion rate was 89% (1078 out of 1207 child-specific visits). Those identified as low risk (n = 951; 88%) were reassured and followed routinely. Those screening positive (n = 127; 12%) were referred for diagnostic assessment. Twelve (1% of toddlers screened) were diagnosed with autism at a mean age of 24 months. In addition, positive M-CHAT detected speech delay and social anxiety. Providers acknowledged their satisfaction with the M-CHAT implementation process; the main challenge was communicating to families the importance of screening. Referrals for diagnostic evaluations increased from 23 to 43 cases in the first year, and 35 in the second year. Conclusion. Implementation of the autism screening using the Arabic M-CHAT is feasible and effective in a primary care setting in Saudi Arabia. Sustaining the implementation of developmental screening in practice requires staff engagement and systematic monitoring of the impact of change.

Keywords

Autism Saudi Arabia M-CHAT Autism screen Primary care

Introduction

The American Academy of Pediatrics (AAP) recommends that all children receive autism-specific screening at 18 and 24 months of age, in addition to broad developmental screening at 9, 18, and 24 months.^1,2 Since this recommendation, there has been an increase in the routine use of developmental screening in primary care.^3,4 Challenges to the implementation of developmental screening in primary care have been reported.^3,4 Systemic challenges include time constraints and inadequate reimbursement.^5,6 Also, appropriate training regarding the importance of developmental screening influences the willingness of providers to implement routine screening.⁷ Although standard developmental screening tools increase detection of concerns, many primary care providers continue to rely on surveillance alone, trusting their clinical acumen and parental concerns.⁷ Therefore, the implementation of developmental screening in primary care practice requires a systematic and comprehensive approach that addresses barriers to implementation.⁸ Appropriate training regarding the importance of developmental screening influences the willingness of providers to implement routine screening.⁷

The use of developmental screeners in non-English primary language cultures has further limitations due to linguistic and cultural differences.^9,10 Adoption of developmental screening in practice requires awareness of culture and setting, in addition to the appropriate translation of screening measures.^11,12

The Modified Checklist for Autism in Toddlers (M-CHAT) is one of the most widely used tools for early detection of autism.⁵ While the M-CHAT has been translated into multiple languages including Arabic, it is not commonly used in Arab countries. One study involving 9 Arab countries concluded that M-CHAT’s sensitivity, specificity, and positive predictive value were similar to those reported in the initial M-CHAT validation study.¹³ A study conducted in Egypt using the Arabic M-CHAT noted concern among pediatricians that the high number of false positives would have a negative impact on resource-limited health care systems.¹⁴ Furthermore, the feasibility of its implementation in primary care settings in Arabic-speaking countries has yet to be described.

As part of a quality improvement (QI) initiative, the Pediatrics Department at John Hopkins Aramco Healthcare (JHAH), Saudi Arabia, evaluated the feasibility and effectiveness of implementing the M-CHAT as a routine screening tool for autism in a low-risk pediatric population for the year of 2014. Feasibility was reflected by the proportion of visits with M-CHAT completed, and reports of workflow challenges and provider satisfaction. The effectiveness of screening as applied in the primary care setting by staff in that setting was evaluated based on the number of referrals for autism evaluation and rates of autism identification.¹⁵ We describe in this article the QI implementation process and report on the feasibility and effectiveness of implementing M-CHAT screening in a large primary care setting in Saudi Arabia.

Methods

Setting

The Primary Care Pediatric Clinic at JHAH serves the employees of Saudi ARAMCO (Saudi Arabian American Oil Company) and their dependents, who are from multiple provinces of Saudi Arabia. In 2014, JHAH served approximately 30 000 children between 0 and 14 years.

Participants

The M-CHAT was distributed to the caregivers of all toddlers between 16 and 32 months attending JHAH’s pediatric clinic between January and December 2014 regardless of visit type (ie, well-visit, sick-visit, or nurse-only immunization visit). If a child had multiple visits during this period, only the results from the first M-CHAT screening were included in analyses, even if the screen was re-administered on subsequent visits.

Project Implementation

We identified the following key steps for project implementation: (1) assessing office protocol and workflow; (2) identifying a physician champion; (3) addressing key drivers to project success; (4) conducting staff orientation and training; and (5) sharing process and outcome data at regular intervals with the staff.⁴

Through assessment of the office workflow, we were able to formulate the most suitable process to efficiently distribute M-CHAT screens without significant impact on the workflow. A physician champion led the project and was responsible for training, data collection, and analyses. Key drivers to project implementation were defined, and physician and nursing staff were engaged in training and regular sharing of process and outcome data. Monitoring of the impact of changes in M-CHAT completion rate, the accuracy of scoring, and referral procedures were conducted monthly through charts review.

Addressing Key Drivers

We developed a key driver diagram (Figure 1), highlighting the primary drivers that contribute to the achievement of the project aim: (1) physician engagement, (2) nursing engagement, (3) workflow challenges, and (4) parent/family engagement. We theorized on potential secondary drivers that were used to test change ideas.

Figure 1.

Key driver diagram.

Project Timeline

Approval was obtained from the Chiefs of the Pediatric and Nursing Departments prior to project implementation. Nurses implemented a simultaneous QI project involving proper documentation of M-CHAT distribution. Institutional review board approval was not required for purposes of project implementation and reporting; authorization to review medical records of patients was obtained.

Educational training was conducted in December 2013 with physicians and nursing staff on the latest AAP developmental screening recommendations and the diagnostic criteria for autism spectrum disorder. Also, physicians attended educational workshops on the scoring of M-CHAT and referral procedures, as well as how to discuss the results with families.^15,16

Following a review of the first quarter data, we noted the frequency of incomplete/missing M-CHATs and erroneous scoring. A change cycle was implemented that included providing more 1:1 guidance on the scoring of M-CHAT with physician providers. During the second quarter, changes to the formatting of the M-CHAT were made following feedback from nurses and families to improve readability. Following the second-quarter review, it was noted that a change in formatting, such that the screening questions were on both front and back of the page, led to an increased number of incomplete M-CHAT screeners. Another PDSA cycle was implemented adding an instruction to “turn the page.” We also created a dedicated referral link on the electronic health system for positive M-CHAT scores. In July 2014, the Arabic M-CHAT-R (Modified Checklist for Autism in Toddlers–Revised) replaced the M-CHAT, scoring was adjusted accordingly, and providers were again given 1:1 guidance on scoring. For purposes of reporting, we continue to use M-CHAT as the collective term to describe both the original M-CHAT and the revised version.

M-CHAT Scoring and Referral Processes

For purposes of our project, M-CHAT scores were classified as low risk (ie, negative score), medium risk, or high risk, as described in the M-CHAT standard instructions. Table 1 notes the action recommended for each score/category on the M-CHAT and M-CHAT-R.

Table 1.

M-CHAT and M-CHAT/R Score, Risk Categories, and Associated Actions.

Score		Category		Action
M-CHAT	M-CHAT/R	Category		Action
0-2 (non-critical items)	0-2	“Low risk”	Negative score	Reassured and given regular follow-up
3-6	3-7	“Medium risk”	Positive score	Referred to Neurodevelopmental Clinic
>6 (or ≥2 critical items on M-CHAT)	8-20	“High risk”	Positive score	Referred to Autism Multidisciplinary Team Diagnostic Clinic for further evaluation

Abbreviations: M-CHAT, Modified Checklist for Autism in Toddlers; M-CHAT/R, M-CHAT–Revised

Children with a medium risk score were referred to the Neurodevelopmental (ND) Clinic, run by a consultant pediatrician who reviewed and verified M-CHAT responses using the follow-up interview, a method shown to increase the positive predictive value of M-CHAT screening.¹⁷ This was particularly relevant because some of the M-CHAT questions were hard to interpret in Arab culture; some questions created confusion either because the question was not a familiar activity (eg, Q1: Does your child enjoy being swung, bounced on your knee, etc?) or the meaning was not understood (eg, Q18: Does your child make unusual finger movements near his/her face?) Similar difficulties understanding certain M-CHAT questions were also noted in other cultures.¹⁶

The pediatrician in the ND clinic conducted a full history and developmental observation. Children who did not meet criteria for autism were reassured. When needed, referral to an autism-specific treatment program was made, and information/resources were provided to the parents. Few were further referred to the Autism Multidisciplinary Team Diagnostic Clinic (AMTDC) for a definite diagnosis. Thus, the ND clinic served as an intermediate level of evaluation.

Children whose M-CHAT scores were in the high-risk range were referred directly to AMTDC. The AMTDC consisted of a multidisciplinary team including a consultant pediatrician, child psychiatrist, clinical psychologist, speech therapist, occupational therapist, and a social worker. Comprehensive diagnostic evaluations were in keeping with recommendations for autism diagnosis, including (1) detailed health, developmental and behavioral history, eliciting core diagnostic symptoms of autism (DSM-5); (2) comprehensive physical examination for dysmorphic features and neurologic abnormalities; (3) assessments of language, cognitive, and motor functioning; and (4) use of a recognized autism assessment tool, the Childhood Autism Rating Scale.¹⁸

Measures

All charts were reviewed monthly and reported quarterly. The following items were assessed on the chart review:

Completion. M-CHAT fully completed and, in the chart.

Scoring accuracy. Including correction of inaccurate scoring.

Referrals. Documentation of action taken based on score (eg, reassurance, referral to ND clinic, AMTDC clinic, or other referrals).

Diagnoses. Review of charts for patients seen in ND Clinic and/or AMTDC clinic to determine any final diagnoses.

Patient demographic information.

In addition, following project completion, we continued to review the referral and diagnostic rates to AMTDC using data from the clinic itself.

Primary Outcome Measures

Feasibility measurement was based on the percentage of visits with M-CHAT completed. The impact of systematic screening on the number of diagnostic referrals and autism identification rates were estimated by chart review.

Secondary Outcome Measures

To explore the engagement and satisfaction of the providers, a survey was e-mailed to providers in April 2016. The survey consists of 10 questions that targeted providers’ engagement, and their perceptions of barriers, benefits, and potential improvements.

Results

The Effectiveness of M-CHAT at Identifying Children At Risk for Autism

Of the 1207 M-CHAT screens distributed, 129 (11%) were missing or incomplete, and 1078 were completed (Figure 2). Of those completed, the majority of toddlers (88%) scored negative (low risk); these parents were reassured and given regular follow-up appointments. In our study, of those who were screened, a total of 12% were positive (two thirds medium risk and one third high risk).

Figure 2.

Flow chart of M-CHAT screening results.

Toddlers with positive M-CHAT in the medium- and high-risk range were evaluated in the ND clinic and/or AMTDC (Figure 3). Of the initial 127 positive M-CHAT scores, 40 (31%) were lost to follow-up. Of those evaluated (n = 87), 41 toddlers (47%) were judged to be developing typically. Twelve were diagnosed with autism (14% of all positive M-CHAT, and 1% of all children screened) at a mean age of 24 months. In addition to autism, positive M-CHAT detected other developmental disorders in 34 toddlers (39% of those evaluated): speech delay, social anxiety, and subthreshold autism symptoms not meeting diagnostic criteria. Of the toddlers who had a speech delay (without autism), 85% were males. On the other hand, of toddlers with social anxiety, 80% were females.

Figure 3.

Referral to AMTDC and number of cases diagnosed with autism.

The Effectiveness of M-CHAT Screening in Increasing Autism Diagnostic Referrals

There was a sizable increase in the number of referrals to AMTDC in the year of project implementation and the year after (2014 and 2015). In 2013, there were 21 referred patients seen at AMTDC, while in 2015 there were 43 patients, with the largest increase noted in the year immediately following project implementation. While there was a slight decline in the referrals seen in 2016, the referral numbers remained higher than in the period before M-CHAT implementation.

Feasibility of M-CHAT in Practice

Study findings indicated that 89% of eligible toddlers received M-CHAT screening within the study year, suggesting that overall M-CHAT implementation in primary care practice in Saudi Arabia is feasible. Also, there was a sizeable decrease in the number of missing/incomplete screeners from the first half of the year to the second half.

Figure 4 represents the number of missing/incomplete screeners and inaccurate scoring (errors) over time. It is noteworthy that the implementation of individual guidance on scoring after the first quarter was associated with a reduced number of errors. There was a spike in the number of incomplete and erroneously scored screeners following formatting changes during the second quarter, and again after the M-CHAT-R was initiated in the second half of the implementation. Improvements made to the formatting and further guidance on scoring resulted in the decline in the number of incomplete/missing screeners and inaccurate scoring over time.

Figure 4.

Effect of interventions on the rate of missing/incomplete charts, and errors in scorning.

Feasibility of M-CHAT: Provider Engagement and Satisfaction

All pediatric physicians participated in the M-CHAT implementation, but only 4 out of 13 physicians (30%) and 6 out of 8 nurses (75%) completed the survey. All providers reported satisfaction with the process of M-CHAT implementation; they felt involved in the implementation, decision-making, and modifications processes. The majority felt that their concerns were met. Forty percent found it challenging to their workflow, suggesting time associated barriers, even though they were overall satisfied with the project implementation. Eighty percent agreed that M-CHAT improved patient’s care and increased their knowledge about autism. Qualitative responses indicated the following future suggestions:

Send M-CHAT to parents before the clinic visit to ease the clinic flow and allow for more accurate information.

Use the electronic version of M-CHAT which is directly incorporated into electronic medical record.

Change 18-month-old nurse-only immunization visit into a physician visit to complete M-CHAT screening, as specifically recommended by the AAP.

Increase parental awareness and acceptance of screening by providing information regarding the importance of screening and early detection and use email blasts that describe screening process.

Discussion

To our knowledge, this is the first study to evaluate the feasibility and effectiveness of implementing routine screening of toddlers using the Arabic M-CHAT in a primary care setting in Saudi Arabia. In our study, of those children screened, a total of 12% were noted to have concerns on M-CHAT screening (two thirds medium risk and one third high risk). As has been reported based on the use of the M-CHAT in the United States, positive M-CHAT in our study also detected other delays (eg, speech delay) and psychological difficulties (eg, social anxiety).¹⁹ This highlights that children with false-positive screening results are an at-risk group for which diagnostic evaluation draws attention to their need for services.¹⁹

Referral rates for diagnostic assessments increased during the implementation period suggesting the effectiveness of routine M-CHAT screening on identifying those children at risk for autism and other developmental concerns. AMTDC was established as part of JHAH in 2012. Following its establishment, awareness of neurodevelopmental disorders among providers at JHAH increased resulting in an initial spike in 2013 referral rates. The AMTDC clinic received referrals from the primary care clinic, as well as several other subspecialty clinics within JHAH. The increase in referrals during and following M-CHAT screening implementation was above what was expected as a direct result of screening, suggesting that this trend may have partly been contributed by generally increasing awareness and knowledge of neurodevelopmental disorders among pediatric health care providers.

The initial increase in referrals to AMTDC was maintained (with only a slight decline) in the years following initial project implementation. The decline in referral rate might be explained by changes in when M-CHAT was distributed. During project implementation, M-CHAT was distributed to all toddlers (16-32 months) regardless of visit type; however, in subsequent years, the M-CHAT was given only at 18 and 24 months, in keeping with the AAP recommendation for autism-specific screening. This suggests that regular developmental surveillance is needed in addition to the use of formal screening instrument.

Within our sample, 32% of children who screened positive on M-CHAT were later felt to be typically developing following evaluation. Two thirds of those screening positive on M-CHAT (medium risk) were evaluated in ND clinic, which served as an intermediate step by reviewing the questions with parents based on the M-CHAT follow-up interview and by addressing the cultural interpretation of questions. Using a screening tool needs it to be adapted to the cultural norms and familiarities of the studied population.^12,20 The majority of the children with a positive M-CHAT were therein determined not to have specific developmental concerns. Other studies have indicated the importance of the M-CHAT follow-up interview in increasing the positive predictive value of concerning results.²¹ The M-CHAT follow-up interview can be done on the same visit, or a close follow-up visit. Using this 2-step screen can reduce the number of false positive and the referral rate to the specialized clinic.^22,23 Pediatricians might be trained on using 2-step screening procedures to assure appropriate referrals for comprehensive evaluations.²⁴

Study results also demonstrated that implementation of M-CHAT in primary care practices in an Arabic-speaking country is feasible, with a very high rate of properly completed M-CHAT screeners and a high level of provider satisfaction. We found that incorporation of QI strategies helped ensure the feasibility and fidelity of M-CHAT implementation. The number of missing, incomplete, and erroneously scored screeners diminished over time. Providers’ knowledge of correct scoring rules, the format of the survey, and parental awareness of the purpose of screening contributed to successful implementation.

In the first half of the QI cycle, regular individual scoring guidance with providers improved rates of correct scoring as well as decreasing the number of missing/incomplete charts. This supports studies that suggest that providers’ lack of comfort with screening instruments is a barrier to successful screening in primary care practices.⁷ Through the QI process, nursing staff noted that lack of parental awareness of the purpose of screening influenced survey completion. Therefore, an essential change strategy included increasing family/parental engagement by providing information regarding the importance of screening.

In the second half of the QI cycle, with the use of M-CHAT-R, we noticed less error in scoring, more completed forms, and less parental confusion answering the screen. Also, the scoring is easier in the M-CHAT-R, with fewer questions, which were more clearly stated and included added examples.¹⁷ Even minor details such as survey formatting affected the completion rate of the questionnaire.

It has been noted before that after the success of a QI project, it is important to implement strategies to sustain the change.²⁵ Four areas have been identified that lead to sustainability: awareness of the practice, agreement with it, adoption of it, and adherence.²⁶ Our QI process concluded by the routine policy of screening using the M-CHAT at the 18- and 24-months visits. To sustain the use of the M-CHAT for autism screening, systematic changes that directly incorporate the procedure into the workflow and staff engagement continue to be vital to its ongoing success.²⁷

Limitations

We were not able to follow-up on the toddlers who scored in the low risk or those apparently “typically developing.” As such, calculation of positive predictive value, sensitivity, and specificity are not within the scope of this study. Some of the toddlers who scored positive on the M-CHAT were lost to follow-up. These toddlers may be at particular risk, and cultural barriers may impede recognition of developmental concern or willingness to pursue further evaluation.

Implications and Conclusions

Implementation of autism screening using the Arabic M-CHAT is effective and feasible among a low-risk toddler population in Saudi Arabia. While it was not within the scope of the study to evaluate the psychometric properties of M-CHAT in Arabic population, study findings highlight the need to support the cultural interpretation of screening instruments and to have in place an adequate referral and evaluation system. In addition, sustaining implementation within a primary care setting requires engagement of the entire clinic staff, initial and ongoing training, as well as the incorporation of ongoing change cycles with monitoring of effect.

Footnotes

Author Contributions

Dr. Amel Alawami conceptualized and designed the study,collected,analyzed and interpreted the data;drafted the initial manuscript and reviewed and revised the manuscript.

Dr. Christina Sakai conceptualized the study design,contributed to analyses and interpretation of data,drafting and revising the manuscript for critically important intellectual content.

Dr. Ellen Perrin supported analyses and interpretation of data and revised the manuscript critically for intellectual content.

All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author(s) received no financial support for the research,authorship,and/or publication of this article.

ORCID iD

Amel Hussain Alawami

References

Council on Children With Disabilities; Section on Developmental Behavioral Pediatrics; Bright Futures Steering Committee; Medical Home Initiatives for Children With Special Needs Project Advisory Committee. Identifying infants and young children with developmental disorders in the medical home: an algorithm for developmental surveillance and screening. Pediatrics. 2006;118:405-420. doi:10.1542/peds.2006-1231

Myers

Johnson

; American Academy of Pediatrics Council on Children With Disabilities. Management of children with autism spectrum disorders. Pediatrics. 2007;120:1162-1182. doi:10.1542/peds.2007-2362

Radecki

Sand-Loud

O’Connor

Sharp

Olson

LM.

Trends in the use of standardized tools for developmental screening in early childhood: 2002-2009. Pediatrics. 2011;128:14-19. doi:10.1542/peds.2010-2180

Earls

Hay

SS.

Setting the stage for success: implementation of developmental and behavioral screening and surveillance in primary care practice—the North Carolina assuring better child health and development (ABCD) project. Pediatrics. 2006;118:e183-e188. doi:10.1542/peds.2006-0475

Miller

Gabrielsen

Villalobos

et al . The each child study: systematic screening for autism spectrum disorders in a pediatric setting. Pediatrics. 2011;127:866-871. doi:10.1542/peds.2010-0136

Pinto-Martin

Dunkle

Earls

Fliedner

Landes

Developmental stages of developmental screening: steps to implementation of a successful program. Am J Public Health. 2005;95:1928-1932. doi:10.2105/AJPH.2004.052167

Morelli

Pati

Butler

et al . Challenges to implementation of developmental screening in urban primary care: a mixed methods study. BMC Pediatr. 2014;14:16. doi:10.1186/1471-2431-14-16

American Academy of Pediatrics. Developmental surveillance and screening of infants and young children. http://pediatrics.aappublications.org.ezproxy.library.tufts.edu/content/pediatrics/108/1/192.full.pdf. Accessed May 14, 2019.

Heo

Squires

Yovanoff

Cross-cultural adaptation of a pre-school screening instrument: comparison of Korean and US populations. J Intellect Disabil Res. 2008;52(pt 3):195-206. doi:10.1111/j.1365-2788.2007.01000.x

10.

Cross

. Towards a culturally competent system of care: a monograph on effective services for minority children who are severely emotionally disturbed. https://eric.ed.gov/?id=ED330171. Published March 1989. Accessed May 14, 2019.

11.

Kara

Mukaddes

Altnkaya

Guntepe

Gokcay

Ozmen

Using the modified checklist for autism in toddlers in a well-child clinic in Turkey: adapting the screening method based on culture and setting. Autism. 2014;18:331-338. doi:10.1177/1362361312467864

12.

Peña

. Lost in translation: methodological considerations in cross-cultural research. Child Dev. 2007;78:1255-1264. doi:10.1111/j.1467-8624.2007.01064.x

13.

Eldin

Habib

Noufal

et al . Use of M-CHAT for a multinational screening of young children with autism in the Arab countries. Int Rev Psychiatry. 2008;20:281-289. doi:10.1080/09540260801990324

14.

Mohamed

Zaky

Youssef

et al . Screening of Egyptian toddlers for autism spectrum disorder using an Arabic validated version of M-CHAT; report of a community-based study (Stage I). Eur Psychiatry. 2016;34:43-48. doi:10.1016/j.eurpsy.2016.01.2421

15.

Pagoto

Lemon

SC.

Efficacy vs effectiveness. JAMA Intern Med. 2013;173:1262-1263. doi:10.1001/jamainternmed.2013.6521

16.

Jacob

Srinath

et al . Toddlers at risk for autism spectrum disorders from Kerala, India—a community based screening. Asian J Psychiatr. 2018;31:10-12. doi:10.1016/J.AJP.2017.12.016

17.

Robins

Casagrande

Barton

Chen

CMA

Dumont-Mathieu

Fein

Validation of the Modified Checklist for Autism in Toddlers, Revised with Follow-up (M-CHAT-R/F). Pediatrics. 2014;133:37-45. doi:10.1542/peds.2013-1813

18.

Johnson

Myers

; American Academy of Pediatrics Council on Children With Disabilities. Identification and evaluation of children with autism spectrum disorders. Pediatrics. 2007;120:1183-1215. doi:10.1542/peds.2007-2361

19.

Robins

DL.

Screening for autism spectrum disorders in primary care settings. Autism. 2008;12:537-556. doi:10.1177/1362361308094502

20.

Soto

Linas

Jacobstein

Biel

Migdal

Anthony

BJ.

A review of cultural adaptations of screening tools for autism spectrum disorders. Autism. 2015;19:646-661. doi:10.1177/1362361314541012

21.

Blenner

Braden

Gabovitch

et al . Considering culture in autism screening. Learn the signs act early. https://www.maactearly.org/uploads/9/2/2/3/9223642/considering_culture_asd_screening_1-68_version.pdf. Accessed May 14, 2019.

22.

Chlebowski

Robins

Barton

Fein

Large-scale use of the modified checklist for autism in low-risk toddlers. Pediatrics. 2013;131:e1121-e1127. doi:10.1542/peds.2012-1525

23.

Khowaja

Robins

Adamson

LB.

Utilizing two-tiered screening for early detection of autism spectrum disorder. Autism. 2018;22:881-890. doi:10.1177/1362361317712649

24.

Sturner

Howard

Bergmann

et al . Autism screening with online decision support by primary care pediatricians aided by M-CHAT/F. Pediatrics. 2016;138:e20153036. doi:10.1542/peds.2015-3036

25.

Virani

Lemieux-Charles

Davis

Berta

Sustaining change: once evidence-based practices are transferred, what then?

Healthc Q. 2009;12:89-96. doi:10.12927/hcq.2009.20420

26.

Pathman

Konrad

Freed

Freeman

Koch

GG.

The awareness-to-adherence model of the steps to clinical guideline compliance. The case of pediatric vaccine recommendations. Med Care. 1996;34:873-889.

27.

Bray

Cummings

Wolf

Massing

Reaves

After the collaborative is over: what sustains quality improvement initiatives in primary care practices?

Jt Comm J Qual Patient Saf. 2009;35:502-508. doi:10.1016/S1553-7250(09)35069-2