Sage Journals: Discover world-class research

Abstract

Background

Clinical practice guidelines are systematically developed statements that assist clinicians in making evidence informed decisions regarding patient care. Within pediatric concussion, the Ontario Neurotrauma Foundation released the Guidelines for Diagnosing and Managing Pediatric Concussion in 2014. The purpose of this study was to evaluate the 2014 guidelines using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) evaluation tool, in addition to a brief knowledge translation survey, and to utilize the collected feedback from end users to inform improvements to support an updated version. An integrated knowledge translation approach was employed using clinical experts as guideline appraisers.

Methods

A purposive sample of researchers, physicians, allied health professionals, policy makers, educators and knowledge translation experts involved in updating the guidelines (N = 31) completed the AGREE II Likert scale survey regarding the 2014 guideline, and provided written justifications for their ratings. Domain and item AGREE II scaled scores were reported stratified by demographic factors, and written justifications were synthesized using content analysis to determine areas of improvement for the 2014 guideline.

Results

Appraisers scored the editorial independence (88.9%) and scope and purpose (80.8%) domains the highest, indicating high quality. The guidelines scored the lowest in the applicability domain (69.3%). Participants with less than 10 years of experience in their respective disciplines, as well as physicians and allied health professionals consistently provided higher ratings across domains compared to other professions.

Conclusions

The process of evaluating the 2014 guideline resulted in these important outcomes: (1) identified areas of the guideline that may have affected the lack of previous clinical uptake while abiding by a clinical practice guideline development framework; (2) shared and informed decision making regarding content and format of the revised clinical practice guideline; and (3) targeted content, clinical questions and dissemination strategies, which are key to clinical uptake.

Keywords

Pediatric concussion clinical practice guidelines AGREE II integrated knowledge translation

Background

Pediatric concussion is considered a serious public health concern in Canada.¹ Between 2003 and 2013, there was a 4.4-fold increase in pediatric concussion visits in Ontario, with nearly 35,000 visits in 2013.² The identification, assessment and management of pediatric concussion have evolved rapidly over the past decade, yet emerging evidence is not always integrated into practice.³ Thus, it is critical that the clinical community is provided with evidence informed clinical guidelines to ensure that their concussion practices reflect a high level of care and consistency.

Clinical practice guidelines (CPGs) are systematically developed statements that assist clinicians in making evidence informed decisions regarding patient care. CPGs cover an array of patient-related domains such as screening, diagnosis, and management, in addition to informing health policy.⁴ The overall goal of a CPG is to reduce unnecessary variations in service delivery by informing the management of a certain health condition.⁵ Unnecessary variations in service delivery may be due to a variety of demographic factors that influence if and how a CPG is used. For example, factors such as a healthcare professional’s discipline, experience, age, and even sex, have the potential to influence the management of concussion, but have yet to be investigated in the field of pediatric concussion.⁶

Within pediatric concussion, the Ontario Neurotrauma Foundation (ONF) released the Guidelines for Diagnosing and Managing Pediatric Concussion in 2014 (herein referred to as the 2014 CPG) with the aim to clarify and standardize concussion recommendations, anchored in empirical evidence. At the outset, the ONF committed to update the guidelines (proposed for 2019).⁷ The 2014 CPG is one of four CPGs specific to pediatric concussion available, with one study reporting that 71.4% of appraisers indicated that they would recommend the 2014 CPG with revisions.⁸ To date, no research has examined the various barriers to the 2014 CPG.

Updating the 2014 CPG presented a unique opportunity to assess the guidelines to inform current development activities. Beyond reviewing new knowledge generated by empirical evidence, it was key to employ a structured and systematic approach to evaluate the 2014 CPG so its evolution aligned rigorously with CPG development methodology, and to facilitate optimal dissemination and uptake of updated versions. Therefore, the use of an integrated knowledge translation (iKT) approach, defined as a co-production of knowledge through a relationship between researchers and decision makers,⁹ was integral to improving clinical uptake of the CPG given that uptake is influenced by a myriad of factors beyond the evidence alone.

Though CPGs have the potential to improve practice and support clinical decision-making, these benefits have not consistently been achieved due to barriers to use. In addition to demographic factors, limited impact of a guideline can be attributed to vagueness,¹⁰ lack of awareness, complexity of the guidelines, accessibility to resources and tools, and the lack of end-user involvement in development.¹¹ For instance, it has been reported that 16–66% of pediatricians were aware of the existence of various pediatric CPGs, but only 19–28% reported a change in their practice behaviour.¹² This supports that guidelines that are clear to understand and do not require specific resources are more likely to be implemented.¹³ Therefore, it is evident that an iKT approach is warranted to glean the barriers limiting clinical uptake from target users presumed to use CPGs in their daily clinical practice.

Evaluating previous CPGs can be an effective and inexpensive way to increase the impact a CPG will have on clinical practices.¹⁴ The validated Appraisal of Guidelines for Research and Evaluation II (AGREE II) tool by Brouwers et al.,¹⁵ the most commonly used guideline appraisal tool,¹⁶ was developed to assess CPG quality and to provide a methodological framework for the development and/or revision of CPGs.¹⁵ The original AGREE tool was developed in 2003 through a multi-staged process and was tested on 100 guidelines from 11 countries.¹⁷ In 2010, the AGREE II tool replaced the original AGREE project with improvements to its measurement properties, usefulness to stakeholders, and ease of implementation.¹⁵ The AGREE II tool evaluates CPGs through six domains: (1) scope and purpose; (2) stakeholder involvement; (3) rigour of development; (4) clarity of presentation; (5) applicability; and (6) editorial independence through a seven-point Likert scale.¹⁸ According to Appenteng et al.,⁸ the highest quality CPGs are targeted guidelines for a specific population, developed by a group with clinical, research and methodological expertise, and have undergone iterations of previous editions. The process of evaluating and developing CPGs should be viewed as progressively adaptive where quality is informed by the composition and expertise of the guideline developers.¹⁹

The current study is anchored in an iKT approach and uses clinical experts as appraisers of the 2014 CPG given their scope of responsibility to invoke practice change. The purpose of this study was to rigorously evaluate the 2014 CPG using the AGREE II tool and utilize the feedback collected from end users to inform quality improvements across content, format, development processes and dissemination to support the development of an updated CPG. The study objectives include: (1) describe the quality of the 2014 CPG through AGREE II domain scores and frequency analysis of knowledge translation (KT) questions; (2) evaluate whether AGREE II ratings of the 2014 CPG are impacted by demographic factors; and (3) interpret written responses/justifications to AGREE II scores to describe potential areas for improvement beyond frequency analysis.

Evaluating AGREE II ratings by demographic factors may inform whether the CPG meets the needs of its target users, its content is well disseminated to end users, and whether there are barriers to use for subgroups of target users. Further, engaging end users in a CPG evaluation is a valuable method to revising a CPG as all end users are aware of the updating process as appraisers, purpose of the CPG as guideline developers, and carry out the recommendations in their own practice. The outcomes of this study are critical to inform the development of an impactful and evidence-informed knowledge product that has been co-created by concussion and knowledge translation experts.

Methods

Context

The ONF was a health research organization focusing on the practical applications of research to improve the lives of individuals with acquired brain injury or spinal cord injury, and the prevention of neurotrauma injuries.²⁰ In 2014, the ONF funded and released the Guidelines for Diagnosing and Managing Pediatric Concussion, a CPG intended for use by healthcare professionals in diagnosing and managing pediatric concussion (5–18 years). The 2014 CPG presents evidence informed recommendations, with levels of evidence indicative of the strength of research to date. The guideline was housed on the ONF website as a downloadable 140-page PDF document. The 2014 CPG contained sections regarding using the guidelines; recommendations for primary users (i.e., healthcare providers), secondary/tertiary users (i.e., parents and/or caregivers, schools and/or community sports organizations), and by injury timeline; background on pediatric concussion, rationale for the guidelines and areas of further research; information about the project team; a rigorous description on the methodology of creating the guidelines; results of an external review; derivative products, costs, updates and audits; notices; and a list of tools. At its release, three external reviewers (two physicians, one researcher) provided an overall quality rating of 90% for the guideline using the AGREE II tool.

In January 2018, the ONF initiated the process to update the 2014 CPG as per their commitment. In October 2018, leaders in concussion research convened for a pediatric concussion guideline consensus meeting, where limitations of the 2014 version were discussed, and a consensus on the new guidelines and format were formed. Engaging end users in the evaluation and revision of the 2014 CPG was a critical component to the consensus meeting as end users could readily reflect on the 2014 CPG when proposing revisions (i.e., content, format, development, dissemination) for the 2019 CPG. Thus, this study served as an essential tool to the process and end product of revising the CPG, in line with an iKT framework.

Study design

This iKT study employed a mixed-methods design. Quantitative data was gathered though surveys. Qualitative data was gathered through written justifications of quantitative survey responses.

Participants

A purposive sample of participants was recruited via email from the list of invitees attending the October 2018 pediatric concussion guideline consensus meeting. The participants comprised researchers, physicians, allied health professionals (i.e., occupational therapy, physical therapy, and speech language pathology), policy makers, educators and knowledge translation (KT) experts. As this was a purposive sample, there were no explicit exclusion criteria, with the exception that participants would be excluded if they could not read or write in English.

Measures

Appraisal of Guidelines for Research and Evaluation II (AGREE II) tool¹⁸

The AGREE II tool is a 23-item questionnaire, spanning six domains: (1) scope and purpose; (2) stakeholder involvement; (3) rigour of development; (4) clarity of presentation; (5) applicability; and (6) editorial independence. A description of the assessment focus for each domain is provided in Table 1. AGREE II items are rated on a seven-point Likert scale (1 = strongly disagree to 7 = strongly agree). The AGREE II tool recommends at least two, but preferably four external reviewers.⁴ The study aim was not to increase the amount of external reviewers and deviate from AGREE II evaluation methodology, rather the goal was to engage the perspective of a diverse group of end users, as reviewers are typically content and methodology experts.

Table 1.

Assessment description for AGREE II domains.¹⁸

Domain		No. of items	Description
1	Scope and purpose	3	Assesses the overall aim of the guideline, specific health questions and the target population
2	Stakeholder involvement	3	Assesses the extent to which the guideline was developed by the appropriate stakeholders and represents the views of its intended user
3	Rigour of development	8	Examines the process of evidence retrieval and synthesis, as well as the methods to formulate and update the recommendations
4	Clarity of presentation	3	Assesses the language, structure and format of the guidelines
5	Applicability	4	Examines the barriers and facilitators to implementation, strategies to improve uptake, and resource implications of applying the guideline
6	Editorial independence	2	Assesses the formulation of recommendations not being unduly biased with competing interests

The AGREE II tool is the most commonly used guideline appraisal tool,¹⁶ and it is well suited for evaluating the 2014 CPG as it is able to target any step along the healthcare continuum (i.e., identification, assessment, management, intervention) and is intended for healthcare providers and guideline developers. The AGREE II user manual does not specify thresholds that distinguish a high-quality versus low-quality guideline at this time; however, higher scores indicate higher quality CPGs.¹⁸

AGREE II score comments

Appraisers were asked to provide comments and/or justifications in response to their rating for each AGREE II item. This was an opportunity to share strengths and constructive criticism/areas of improvement to consider for the 2019 CPG.

Demographic information

Demographic information such as age, sex, discipline and years of experience were collected.

Knowledge translation appraisal

This brief survey, purposefully developed by the research team including clinicians and knowledge translation specialists included 11 knowledge translation questions, which included participant self-reported previous use and familiarity of the 2014 CPG; influence of the 2014 CPG on concussion identification, assessment and management; facilitators and barriers to the use of the 2014 CPG; and recommended tools to enhance future clinical uptake. These KT questions utilized Likert scale responses.

Procedure

The study was approved in concordance with the Research Ethics Board at Holland Bloorview Kids Rehabilitation Hospital in Toronto, Canada. Participants were recruited via email and informed of the option to consent to have their data be part of this research study or decline their participation in research (but agree to have the broader team utilize their data for the development of the updated version of the CPG). Participants completed an online survey containing the measures. An electronic version of the 2014 CPG for healthcare professionals was provided to all participants.

The survey was created, distributed and housed on REDCap™, a secure web application used to build and manage online surveys and databases.^21,22

Analysis of data

Frequency analysis of the self-reported demographic factor data was conducted. To assess quality across the AGREE II domains in accordance with the AGREE II user manual,¹⁸ scaled domain scores were derived by summing the scores of the individual items in a domain, and scaling the total as a percentage of the maximum possible score for that domain. Descriptive statistics (i.e., mean, median, frequency percentages and 95% confidence intervals) captured how the 2014 CPG was rated according to the AGREE II criteria. Scaled AGREE II domain scores were stratified by meaningful demographic factor groups.

Qualitative data (i.e., comments/scoring justifications) related to the AGREE II tool were evaluated using a conceptual content analysis,²³ a common method for analyzing written answers to open-ended questions. An inductive approach was taken with explicit terms coded to determine the occurrence of common themes within the data. Critiques or strengths of the 2014 CPG were of interest during the content analysis. One research team member, independent of the development of either pediatric concussion CPGs, conducted this analysis. Continuous reflection and self-criticism by the research team member was practiced.^24–26 Frequencies of concepts were reported. Lines-of-argument (synthesizing arguments) were generated by integrating the results of the content analysis into the update of the CPG. For this study, an ONF guideline developer provided details regarding how the synthesizing arguments were addressed in the development of the updated CPG.

Results

Participants

A total of 31 participants (males = 13, females = 18) enrolled in this study. Thirteen participants had made a previous contribution to the development of the 2014 CPG. While an array of healthcare professionals were represented in this study, 13 participants were physicians. Participants were equipped with notable experience in their respective fields (Median = 25 years, IQR = 14–30 years) as well as pediatric concussion (Median = 12 years, IQR = 8–20 years).

Appraisal of the 2014 CPG Using the AGREE II Tool

The results of the AGREE II appraisal stratified by demographic variables and KT survey items are provided in Table 2. Table 3 contains a comprehensive overview of the AGREE II ratings by domain and item, a summary of the appraiser justifications, and CPG updates provided by an ONF guideline developer. Note that the N did not remain consistent throughout the survey, as there was some attrition with survey participation (i.e., number of responses ranged from 21 to 26 per AGREE II item).

Table 2.

Mean scaled domain scores (as percentages) of the AGREE II tool, stratified by demographic factors.

Mean Scaled Domain Score, % (95% CI)
Demographic variables	N	Scope and purpose	Stakeholder involvement	Rigour of development	Clarity of presentation	Applicability	Editorial independence
Sex
Male	13	81.3 (57.8–100.0)	72.8 (41.9–100.0)	82.8 (65.6–100.0)	86.1 (54.7–100.0)	80.7 (48.0–100.0)	99.0 (60.4–100.0)
Female	18	80.4 (47.6–100.0)	73.8 (40.2–100.0)	75.3 (60.4–90.3)	72.2 (50.4–94.1)	62.2 (43.7–80.7)	82.7 (76.9–88.5)
Age, years (M = 50.0 ± 9.6 years)
<40	3	87.0 (61.1–100.0)	83.3 (52.5–100.0)	83.3 (66.9–99.7)	79.2 (47.7–100.0)	75.0 (38.2–100.0)	94.4 (81.1–100.0)
40–49	11	71.7 (35.8–100.0)	65.3 (27.0–100.0)	77.1 (59.3–94.9)	81.5 (61.5–100.0)	75.0 (56.5–93.5)	86.1 (54.0–100.0)
≥50	17	86.3 (69.0–100.0)	76.1 (48.4–100.0)	77.6 (61.2–94.0)	75.0 (42.1–100.0)	64.9 (33.0–96.8)	88.9 (52.3–100.0)
Primary discipline
Physician	13	83.9 (51.3–100.0)	79.4 (48.7–100.0)	87.8 (76.1–99.5)	88.2 (67.3–100.0)	83.3 (67.3–99.4)	99.0 (93.1–100.0)
Psychologist/Neuropsychologist	4	77.8 (47.0–100.0)	63.6 (33.2–94.1)	76.4 (62.0–90.9)	63.9 (29.1–98.6)	59.4 (26.0–92.8)	85.4 (56.0–100.0)
Researcher	6	75.6 (47.2–100.0)	72.2 (41.4–100.0)	59.1 (37.2–81.0)	73.6 (36.8–100.0)	56.3 (13.7–98.8)	70.8 (14.7–100.0)
Nurse Practitioner	1	66.7 (N/A)	33.3 (N/A)	54.2 (N/A)	66.7 (N/A)	37.5 (N/A)	58.3 (N/A)
AHP^a	7	84.3 (70.1–98.4)	75.6 (40.8–100.0)	84.3 (72.6–96.1)	76.4 (58.6–94.2)	71.9 (54.2–89.5)	97.9 (88.5–100.0)
Years of experience in discipline, years (M = 21.9 ± 10.3 years)
<10	3	90.7 (64.8–100.0)	90.7 (64.8–100.0)	81.2 (58.8–100.0)	94.4 (83.6–100.0)	86.1 (64.0–100.0)	100.0 (N/A)
10–19	10	71.0 (34.8–100.0)	61.9 (24.8–99.0)	79.7 (66.0–93.5)	72.2 (50.0–94.4)	66.7 (40.4–92.9)	83.3 (53.5–100.0)
≥20	18	84.9 (66.2–100.0)	75.4 (47.9–100.0)	77.2 (60.7–93.7)	75.4 (43.4–96.9)	66.4 (35.8–96.9)	88.1 (52.6–100.0)
Years of experience in pediatric concussion, years (M = 14.1 ± 7.7 years)
<10	8	80.6 (55.0–100.0)	73.0 (37.9–100.0)	70.4 (48.8–91.9)	77.8 (46.2–100.0)	60.7 (21.2–100.0)	79.8 (34.7–100.0)
10–19	13	79.8 (46.0–100.0)	72.8 (46.0–100.0)	79.0 (64.1–93.8)	83.3 (62.6–100.0)	76.2 (58.6–93.8)	86.9 (57.2–100.0)
≥20	10	82.5 (65.9–99.2)	74.6 (54.1–95.1)	85.1 (74.0–96.3)	71.4 (36.6–100.0)	70.8 (45.6–96.0)	100.0 (N/A)
Setting of use
Acute care	13	87.2 (73.1–100.0)	72.2 (48.5–100.0)	82.4 (68.3–96.6)	82.7 (61.4–100.0)	76.4 (55.5–97.3)	93.5 (73.3–100.0)
Rehabilitation	8	69.1 (30.6–100.0)	53.7 (18.5–88.9)	76.8 (60.7–93.0)	56.7 (27.8–85.5)	61.7 (31.0–92.3)	95.0 (83.5–100.0)
Policy	3	87.0 (61.1–100.0)	87.0 (59.7–100.0)	79.7 (60.9–98.5)	81.5 (53.4–100.0)	81.9 (59.3–100.0)	88.9 (59.1–100.0)
Education	7	76.2 (50.7–100.0)	76.9 (41.4–100.0)	82.5 (66.7–98.3)	78.9 (57.6–100.0)	76.7 (58.6–94.7)	85.0 (50.4–100.0)
Research	13	88.9 (67.3–100.0)	79.6 (50.9–100.0)	75.8 (56.1–95.6)	84.0 (56.0–100.0)	70.4 (37.6–100.0)	82.4 (38.1–100.0)
Contribution to 2014 CPG
Yes	13	75.5 (43.2–100.0)	73.2 (41.3–100.0)	74.6 (57.2–92.1)	70.1 (36.3–96.4)	63.5 (30.7–96.4)	84.4 (41.8–100.0)
No	18	85.3 (63.5–100.0)	73.5 (42.3–100.0)	80.7 (64.8–96.7)	82.1 (57.0–100.0)	72.8 (45.5–100.0)	91.7 (67.8–100.0)
Overall	31	80.8 (53.5–100.0)	73.4 (41.9–100.0)	78.3 (61.9–94.7)	77.5 (48.6–100.0)	69.3 (40.2–98.4)	88.9 (56.7–100.0)

^aAllied health professional.

Table 3.

AGREE II score results (%), score comment summaries and improvements in updated CPG. If no n is indicated in the comments, one appraiser provided the feedback.

Scaled item score (Median)	Summary of qualitative comments	Future improvements for updated CPG
Domain 1. Scope and Purpose, Mean scaled domain score = 80.8%
1. The overall objective(s) of the guideline is (are) specifically described (N = 26)
83.3 Med= 6	• Objective is well written/clear/concise (n = 16) • Hyperlinks make CPG easy to navigate• Concern that purpose is medically based with no education implications• Objective should be closer to beginning of CPG (n = 2) • Consider changing headings (ex., “Target Users and Population” to “Who are the guidelines for?”)• More assistance with persistent symptoms would be appreciated	• Content is now organized into a website, which enables ease of accessing specific information (e.g., objectives). A search function will be available to tailor content inquiry. • Hyperlinks will be embedded throughout the website to allow for easy access and flow to documents.• Headers have been changed (e.g., “Target Users and Population”) so that they are more intuitive and specific with respect to content presented.
2. The health question(s) covered by the guideline is (are) specifically described (N = 26)
78.2 Med= 6	• CPG clear/well described/concise given vagueness of concussion (n = 14) • Purpose and context for each recommendation need more detail (n = 3)• More assistance for family physicians• When guideline is targeting different stakeholders, it is difficult to address each person’s exact needs• Adequate if for healthcare only• No case studies • Hyperlinks good for specific questions, but cumbersome for cluster symptoms; algorithms better for short appts (n = 2)• Tip sheets useful, but repetitive• Presentation of who, where, why are a bit confusing	• The domains have been organized according to three sections within the CPG. An introductory paragraph has been written for each section to provide context regarding the domains covered in each section.• The updated guideline targets only healthcare providers (HCPs), the primary target audience.• Derivative products are being made for parents, and the sports and education communities to capture content from the CPG that is important and relevant to them.• The presentation of who, where, why has been removed from this guideline. Rather, recommendations have been written more concisely.
3. The population to whom the guideline is meant to apply is specifically described (N = 26)
80.8 Med= 6.5	• Population is clear/exclusions and age ranges described/easy to find (n = 15) • Specific age groups not well delineated from developmental perspective (i.e., school info–grade school vs. high school)• More assistance for family doctors required• Need to add speech-language pathologists & educational specialists as part of the concussion team• Might be better to separate primary users in one guideline and secondary/tertiary to another• Could be better formatted to be more clear• Should be moved up before “Target population” section	• A speech language pathologist and an expert in special education were part of the expert panel. • The updated guideline targets HCPs only as the primary target audience for this guideline.• Derivative products are being made for parents, and the sports and education communities to capture content from the CPG that is important and relevant to them.• The CPG is being translated into a website format, which will help to enhance the clarity of the presented content. A KT specialist and graphic designer have been engaged to enhance the clarity and presentation of the content.
Domain 2. Stakeholder Involvement, Mean scaled domain score = 73.4%
4. The guideline development group includes individuals from all relevant professional groups (N = 24)
75.0 Med= 6	• Clear and sufficiently detailed/diverse (n = 12) • Good location in guideline• Would be better if diversified expert panelists outside Ontario (n = 3), all specialists from urban areas• Should include: methodology expert, youth, family leader, family medicine, primary care provider, social work, clinical psychologist, developmental pediatrics, educators, sports coaches, CCC• Knowledge translation to general community could be improved	• There was national and international representation on the expert panel. • The following were represented on the expert panel: youth advisor, family medicine, primary care provider, social work, clinical neuropsychology, and special educators. Members of the expert panel have expertise in developing CPGs, and so provide methodology expertise.• KT support has been bolstered by having a KT Specialist on the expert panel. Supports created to foster KT into the community include a document to support sharing and using the guideline in clinical practice.
5. The views and preferences of the target population have been sought (N = 24)
63.2 Med= 5	• Clear (n = 7) and easy to find• Clear that the CPG is primarily for doctors to use and merely a resource for parents to further recognize symptoms/signs• Should include school personnel in updated CPG• Front-line family physicians not considered, unclear if teachers were considered• Family, youth, lived-experience not considered creating a lack of clear description of stakeholders (n = 5)• Knowledge translation to general community could be improved	• Special education was represented in the new CPG.• KT support has been bolstered by having a KT Specialist on the expert panel. • Derivative products are being made for parents, and the sports and education communities to capture content from the CPG that is important and relevant to them. Family leaders, coaches and teachers actively participated in the creation of these products.
6. The target users of the guideline are clearly defined (N = 24)
81.9 Med= 6	• Clearly defined (n = 14)• Need more guidance for family physicians• Should include pediatricians and athletic therapists• Somewhat difficult to navigate at times• Document not friendly for secondary/tertiary users such as parents/caregivers (n = 3)	• Athletic therapists and pediatricians were included in the expert panel.• Navigation of content will be improved by having the content translated into website format.• The guideline is for HCPs only as they are the primary audience. • Derivative products are being made for parents, and the sports and education communities to capture content from the guideline that is important and relevant to them.
Domain 3. Rigour of Development, Mean scaled domain score = 78.3%
7. Systematic methods were used to search for evidence (N = 22)
84.1 Med= 6	• Rigorous and well-described (n = 16)• Search terms not included (n = 2), no clear methodological description• Search not reproducible• Simply not many systematic reviews on the topic	• Scope of the search and inclusion/exclusion criteria is clearly outlined.
8. The criteria for selecting evidence are clearly described (N = 22)
80.3 Med= 6	Clearly described (n = 13)• Level of evidence pyramid used unclear• Description could have been more clear (n = 4) by providing justification, inclusion/exclusion criteria• Difficult to find in the document	• The CPG will be housed on braininjuryguidelines.org. This site is the main hub where all ONF guidelines specific to concussion and m/TBI are being housed.• A communication strategy will be put into place to promote the dissemination of the CPG.
9. The strengths and limitations of the body of evidence are clearly described (N = 22)
77.3 Med= 6	• Well written without bias, covers basic strengths and limitations (n = 10)• No definition of level of evidence scheme• Strengths and weaknesses not clear (n = 3), strengths and weaknesses stated but not specific• Additional description for secondary/tertiary users may be warranted• More information needed in some domains (social/behavioural, neuroimaging) and for some ages (youngest/oldest in population)	• The information provided reflects the current state of the evidence. This CPG will now operate as a living guideline, in which the recommendations and domains will be updated as the evidence emerges.• There are no longer secondary/tertiary users of the guideline. HCPs are the primary and only guideline users.
10. The methods for formulating the recommendations are clearly described (N = 21)
84.9 Med= 7	• Well described (n = 17)• Clear and concise (n = 8)• Not described at all• Absence of solid evidence• Possibly too much detail/could be put in an appendix (n = 2)	• A clear and stepwise methodology section has been added to the CPG.• Algorithms were developed in an effort to decrease the quantity of information and better guide clinical decision-making.
11. The health benefits, side effects and risks have been considered in formulating the recommendations (N = 21)
66.7 Med= 5	• More examples of benefits and risks/more explicitly defined (n = 5) • Unclear/needs to be updated (n = 3)• Option of A vs. B, standard vs. more conservative approach could include more context
12. There is an explicit link between the recommendations and the supporting evidence (N = 21)
74.6 Med= 6	• Yes, explicit link (n = 12)• No link (n = 2)• Must be updated/content was good at the time (n = 3)• Strong attempt	• The information provided reflects the current state of the evidence. The CPG will now operate as a living guideline, the recommendations and domains will be updated as the evidence emerges.
13. The guideline has been externally reviewed by experts prior to its publication (N = 21)
78.6 Med= 6	• Appropriate reviewers (n = 13)• Could not find/unclear if completed (n = 3)• Should have consulted general population (n = 2)• Could have used a few more external reviewers	• The guideline has been externally reviewed by two experts; this is stated explicitly in the guideline.
14. A procedure for updating the guideline is provided (N = 21)
79.4 Med= 6	• Description provided (n = 16)• Noted not on time (n = 3)• Not done	• This CPG is now going to be a living guideline; the recommendations and domains will be updated as the evidence emerges.
Domain 4. Clarity of Presentation, Mean scaled domain score = 77.5%
15. The recommendations are clear and unambiguous (N = 21)
76.2 Med= 6	• Straightforward and clear (n = 12)• Vague (n = 5)• Recommendations regarding when to refer to multi-disciplinary clinics would be helpful• Statements clear, but disagree with who is listed as primary “go to persons” and believe the who needs to change in some sections	• Prompts for multidisciplinary referral are now based on the HCPs scope of practice.
16. The different options for management of the condition or health issues are clearly presented (N = 21)
77.8 Med= 6	• Clearly presented/easy to find (n = 17), broadness is good so physician can use numerous approaches• Not found• Algorithms provided are useful, however they seem embedded in the document• Put the specific recommendations near the front of the CPG since many clinicians might only review the topline recommendations and miss the details further on in the document• Well-presented but could use more clarity• Too much reliance on user knowing when/where to seek additional information• Needs to be updated	• Recommendations will now be presented in web based format in addition to a search/find function to look for specific content.• A PDF version of the recommendations will also be made available for download.
17. Key recommendations are easily identifiable (N = 21)
78.6 Med= 6	• Clearly presented (n = 16)• In a few instances, school personnel are not adequately included• Some content no longer relevant• Suggest an executive/lay summary instead of algorithm• Too much repetition–could be more concise• Could improve language, font, graphics (n = 2)• All recommendations have same weighting, suggest that CPG highlights most important points	• A graphic designer and KT specialist have been engaged to improve the language, font, and graphics of the CPG and the website.• A derivative product for the education community is being developed, where school personnel will be engaged to inform its content.• Content has been written in such a way to minimize repetition and enhance clarity.
Domain 5. Applicability, Mean scaled domain score = 69.3%
18. The guideline describes facilitators and barriers to its application (N = 21)
70.6 Med= 6	• Well described (n = 13)• Length of guideline may limit usage• Information on how to overcome barriers is lacking• Too broad (n = 2)• Not easy to find	• CPG has been transformed into a website, which will enhance accessibility of content. • The CPG will be housed on braininjuryguidelines.org. This site is the main hub where all ONF guidelines specific to concussion and m/TBI are housed.• A communication strategy will be put into place to promote the dissemination of the CPG.
19. The guideline provides advice and/or tools on how the recommendations can be put into practice (N = 21)
82.5 Med= 7	• Practical recommendations (n = 16)• Need to provide more resources for family physicians in smaller communities that lack resources• Yes, but some info in the tools provides slightly different information which is confusing. Would be better to stick with a few good sources than include everything• Lots of tools, but who and how to use and interpret the results are not clear• Could be organized better to improve access	• Links to resources are provided throughout the recommendations for community HCPs to access.• Guideline has been transformed into a website, which will enhance accessibility of content, in addition to a search/find function to look for specific content.
20. The potential resource implications of applying the recommendations have been considered (N = 21)
53.2 Med= 5	• Agree (n = 3) • Not relevant for many of the guidelines• Not specifically addressed, not emphasized (n = 12)• Would be helpful to consider time constraints and level of training of end-users	• CPG includes language highlighting the context for which recommendations are best applied (e.g., use of multidisciplinary concussion teams when managing persistent post-concussion symptoms)
21. The guideline presents monitoring and/or auditing criteria (N = 21)
70.6 Med= 6	Included/clearly indicated (n = 14) • Somewhat brief• No tools for measuring the application of guidelines themselves (n = 3) • Desired outcomes for long term community participation may need additional consideration for students with persistent symptoms	• ONF will be collaborating with CPG co-leads to establish implementation initiatives.
Domain 6. Editorial Independence, Mean scaled domain score = 88.9%
22. The views of the funding body have not influenced the content of guideline (N = 21)
88.9 Med= 7	• No evidence of undue influence (n = 19) • Statement about the goals of ONF aligning with the guidelines—is this contradictory?	• CPG developers have employed the same standard to ensure there is no undue influence of guideline content. For example, funders were welcome to attend the consensus conference but were not included in the appraisal of research or in the revision of guidelines.
23. Competing interests of the guideline development group members have been recorded and addressed (N = 21)
88.9 Med= 7	• Clearly stated (n = 17) • Did not see• Unclear if this is the case	• Conflicts of interest were sought from all expert panel members. • A specific section that addresses conflicts of interest was added to the website.

Domain 1: Scope and purpose

Participants provided a mean overall rating of 80.8% (95% CI, 53.5–100.0%) in Domain 1, reflective of the 2014 CPG’s ability to demonstrate adequate scope and purpose. Physicians and allied health professionals provided the highest ratings (83.4% and 84.3%; 95% CI, 51.3–100.0% and 70.1–98.4%, respectively), while other professions (psychologists/neuropsychologists, researchers, one nurse practitioner) provided ratings approximately 6–17% lower. Participants with less than 10 years of experience in their discipline provided a higher rating (90.7%; 95% CI, 64.8–100.0%) than those with more than 10 years of experience who provided mean domain ratings between 71.0–84.9%. Additionally, participants using the CPG in rehabilitation (69.1%; 95% CI, 30.6–100.0%) and education (76.2%; 95% CI, 50.7–100.0%) settings rated this domain lower compared to those in acute care, research and policy settings (range of mean scores, 87.0–88.9%). Many appraisers (n = 16) stated that the scope and purpose were well described although they noted that more assistance with persistent symptoms was necessary (i.e., concussion management) and that the health questions of all stakeholders were not met as there were too many stakeholders considered.

Domain 2: Stakeholder involvement

The mean Stakeholder Involvement score across all participants was 73.4% (95% CI, 41.9–100.0%). Again, physicians and allied health professionals provided the highest overall scores in this domain (79.4% and 75.6%; 95% CI, 48.7–100.0% and 40.8–100.0%, respectively), while other professions (psychologists/neuropsychologists and one nurse practitioner) provided mean scores between a range of 33.3–63.6%. Participants with less than 10 years of experience in their respective discipline provided the highest rating (90.7%; 95% CI, 64.8–100.0%), while those with greater than 10 years of experience provided a rating that was approximately 15–28% lower. Use of the 2014 CPG in a rehabilitation setting provided the lowest rating of 53.7% (95% CI, 18.5–88.9%). Appraisers commented that the guideline development group could be more diverse, both disciplinarily and geographically, and that there was a clear lack of consideration from families and youth with lived experience (n = 5). Common feedback in this domain was that the CPG was not friendly for secondary/tertiary users.

Domain 3: Rigour of development

The mean domain score for Rigour of Development across all participants was 78.3% (95% CI, 61.9–94.7%). Physicians and allied health professionals provided the highest overall score in this domain (87.8% and 84.3%; 95% CI, 76.1–99.5% and 72.6–96.1%, respectively), while other professions (researchers and one nurse practitioner) provided mean domain scores ranging between 54.2–59.1%. Participants with less than 10 years of experience in concussion rated this domain at 70.4% (95% CI, 48.8–91.9%) while those with greater than 10 years of experience in concussion provided a mean scoring range between 79.0–85.1%. In terms of rigour of development, many appraisers commented that the methods and evidence were well-described (n = 16). Appraisers (n = 13) stated that the external reviewers were appropriate although some thought that the general population should have been consulted (n = 2).

Domain 4: Clarity of presentation

The mean Clarity of Presentation score across all participants was 77.5% (95% CI, 48.6–100.0%). Males provided a higher rating (86.1%; 95% CI, 54.7–100.0%) than females (72.2%; 95% CI, 50.4–94.1%). Physicians provided the highest rating (88.2%; 95% CI, 67.3–100.0%) while psychologists/neuropsychologists and one nurse practitioner provided the lowest ratings (63.4% and 66.7%; 95% CI, 29.1–98.6% and N/A, respectively). Participants with less than 10 years of experience in their primary discipline provided the highest rating (94.4%; 95% CI, 83.6–100.0%), while those with greater than 10 years of experience provided ratings approximately 22–25% lower. Further, those with greater than 20 years of pediatric concussion experience provided the lowest rating of 71.4% (95% CI, 36.6–100.0%) compared to those with less experience. Finally, participants from a rehabilitation setting provided the lowest rating of 56.7% (95% CI, 27.8–85.5%), compared to other settings which provided higher mean scores ranging between 78.9–84.0%. Appraisers (n = 17) stated that the presentation of the recommendations was straightforward and clear, but five appraisers believed the recommendations to be too vague. As well, two appraisers noted that the language, font and graphics could be improved for better clarity.

Domain 5: Applicability

The mean score across all participants for Domain 5 was 69.3% (95% CI, 40.2–98.4%). Males provided a higher rating (80.7%; 95% CI, 48.0–100.0%) compared to females (62.2%; 95% CI, 43.7–80.7%). Physicians and allied health professionals provided higher ratings compared to ratings by psychologists/neuropsychologists (59.4%; 95% CI, 26.0–92.8%), researchers (56.3%; 95% CI, 13.7–98.8%), and one nurse practitioner (37.5%; 95% CI, N/A). Participants with less than 10 years of experience in their primary discipline provided the highest applicability rating (86.1%; 95% CI, 64.0–100.0%), yet those with less than 10 years of pediatric concussion experience provided the lowest applicability rating (60.7%; 95% CI, 21.2–100.0%). Finally, participants from a rehabilitation setting provided the lowest rating of 61.7% (95% CI, 31.0–92.3%), compared to other settings, which provided higher mean ratings ranging between 70.4–81.9%. Appraisers reported that while the 2014 CPG describes the facilitators and barriers to use (n = 13), additional barriers such as the CPG’s length exists. Appraisers identified that while many tools were provided to support implementing the recommendations into practice, some content between tools conflicted. Additionally, who should use the tools and how to interpret the results of the tools was unclear.

Domain 6: Editorial independence

The mean score across all participants for Domain 6 was 88.9% (95% CI, 56.7–100.0%). Females provided a lower mean score (82.7%; 95% CI, 76.9–88.5%) compared to males (99.0%; 95% CI, 60.4–100.0%). Researchers and one nurse practitioner rated editorial independence (70.8% and 58.3%; 95% CI, 14.7–100.0% and N/A, respectively) lower than that of the remaining professions. Interestingly, those with less than 10 years of pediatric concussion experience provided the lowest rating of 79.8% (95% CI, 34.7–100.0%), while those with greater than 10 years of concussion experience provided higher mean ratings ranging between 86.9–100.0%. Most appraisers (n = 19) noted that there was no undue influence by the funding body and that any competing interests were clearly stated (n = 17).

Use and influence of CPG 2014

According to the Knowledge Translation Appraisal survey responses, participants reported minimal use of the 2014 CPG in their daily practice. More specifically, 41.9% (n = 23) of appraisers indicated that they never or rarely used the 2014 CPG compared to 51.6% (n = 16) that stated occasional use. More frequent use of the 2014 CPG did not translate into higher AGREE II domain ratings. Notably, those who reported rarely using the 2014 CPG rated both the Clarity of Presentation and Applicability domains approximately 13% higher than those who reported occasional use of the 2014 CPG. The vast majority of appraisers (83.8%, n = 26) indicated that they were at least somewhat familiar with the 2014 CPG. Appraisers with no familiarity of the 2014 CPG rated the Scope and Purpose and Stakeholder Involvement domains the highest compared to appraisers with any other level of familiarity of the 2014 CPG. In all other domains, ratings increased with familiarity.

Participants revealed that the greatest barriers to using the 2014 CPG included lack of knowledge of the guideline (9.7%, n = 3), length of the guideline (12.9%, n = 4), and that the recommendations could not be applied in their practice setting (19.4%, n = 6). Those that reported that the recommendations could not be applied to their practice setting were neuropsychologists and researchers, who are not primary end users of the guideline. These results provide clear feedback to consider during CPG updates.

Participants also rated how influential they perceived the 2014 CPG to be on concussion identification, assessment, and management. Twenty-four (77.4%) participants reported the 2014 CPG to be slightly or somewhat influential in identifying concussions, 22 (71.0%) participants stated that the 2014 CPG was somewhat to very influential on concussion assessment, and 23 (74.2%) participants indicated that the 2014 CPG was somewhat to very influential on concussion management. The majority of physicians (76.9%, n = 10) rated the 2014 CPG as somewhat to very influential on concussion identification, assessment and management. Also, 85.7% (n = 6) of allied health professionals rated the 2014 CPG somewhat influential on concussion identification, assessment and management. However, most appraisers in other disciplines (i.e., research, neuropsychology/psychology, nursing) believed the 2014 CPG to be only slightly influential on concussion identification, assessment and management.

CPG updates based on appraisal outcomes

Guideline developers considered AGREE II scaled scores in combination with appraiser feedback (Table 3, Column 2) to identify limitations of the 2014 CPG and conceptualize improvements to enhance implementation of the updated CPG (Table 3, Column 3). Common feedback included that the level of evidence was lacking in some areas, the evidence and recommendations were difficult to interpret for secondary/tertiary users, and the guideline was too long and often repetitive. As the field of pediatric concussion research is relatively new and ever expanding, establishing “gold standard” evidence is ongoing. As such, areas of pediatric concussion rehabilitation remain vague/less directive, which poses a challenge to end users looking for guidance in their everyday clinical practice.¹¹ To align with the progression in the field, the new CPG will be a living guideline in a website format. Content will be updated as the evidence emerges to ensure recommendations are rigorously developed and current. The website will make content more accessible and increase clarity, with a filter function to streamline end user needs. Additionally, to increase uptake and improve clarity, the updated CPG was created for healthcare providers as the single, primary target user. Derivative products were created for parents, as well as the sports and education communities, containing relevant information from the updated CPG. All areas of the updated CPG and associated products have been created by a multi-disciplinary team bolstered by KT specialists and graphic designers for optimal content quality, clarity and dissemination.

Discussion

The primary objective of the study was to evaluate the ONF’s Guidelines for Diagnosing and Managing Pediatric Concussion using the AGREE II tool in order to inform its update. Overall, the findings revealed that the 2014 CPG was acceptable across domains, with variations within domain scores suggested to be accounted for by various demographic factors examined in this study. The findings not only support the use of the AGREE II tool to ensure standards for a high-quality CPG were being met through critical appraisal by end users, but also to inform the development of the updated CPG to optimize implementation of of the revised 2019 CPG. While evaluating CPGs using the AGREE II tool is a widespread practice, the engagement of end users within an iKT lens has not been carried out in the field of pediatric concussion. This study makes a notable contribution to understanding how end user demographic factors influence CPG use and acceptability. The iKT framework engaged end users in the evaluation of the 2014 CPG, which enabled maximum participation in both the conception and future implementation of the 2019 CPG. The implication of this work is to prompt clinical communities to expand how they appraise and revise a CPG. As evidenced in this study, a high external reviewer score does not necessarily reflect optimal clinical uptake or acceptability to end users. Thus, it behoves the pediatric concussion community to take a more systematic approach to investigate factors limiting CPG usage.

The AGREE II appraisal of the 2014 CPG was in line with other pediatric concussion CPGs across the world. A study by Appenteng et al.⁸ reported mean domain AGREE II scores for Scope and Purpose, Stakeholder Involvement, Rigour of Development, Clarity of Presentation, Applicability, and Editorial Independence of 86.1%, 55.7%, 46.7%, 81.0%, 36.2%, and 43.2%, respectively for CPGs that addressed only pediatric concussion or mild traumatic brain injury.^7,27–29 In the current study, the quality ratings from appraisers were in line with or higher than these reported ratings in the literature.

The KT appraisal survey identified some key areas to consider for the updated version of the CPG. Firstly, over 90% of appraisers reported that they did not regularly use the 2014 CPG, which is of concern but may be improved. For example, clinician adherence to CPGs in brain injury rehabilitation in a brain injury rehabilitation unit in Melbourne, Australia with no active KT interventions was approximately 39%. With a >12-month audit and feedback intervention, adherence increased to 84%.³ Thus, removing identified barriers to use (i.e., knowledge of the guideline, length of the guideline, and relevance) and implementing active KT interventions may lead to more frequent use by healthcare professionals. For the updated ONF guidelines, the use of KT specialists with dissemination and implementation expertise will help with widespread distribution of the updated CPG. The length barrier will be removed as the updated CPG will be on a website with sub-sections, making the information more digestible and less repetitive. Relevance of recommendations in practice settings will be improved based on more explicitly defining the target population of healthcare professionals to have more specific and relevant recommendations. Further, from the KT appraisal survey, an important finding was that healthcare professionals (i.e., physicians and allied health professionals), the new target users of the updated CPG, reported that the 2014 CPG was highly influential in identifying, assessing and managing concussions. This suggests that the guideline developers are well positioned to create an impactful, evidence-informed CPG for their target audience.

The combined analysis of the quantitative AGREE II domain ratings and qualitative appraiser feedback informed necessary updates to the 2014 CPG as outlined previously through a systematic approach. The use of end users focussed and streamlined the process. The Scope and Purpose domain of the 2014 CPG appeared to effectively convey objectives in a clear and concise manner. In terms of Stakeholder Involvement, there was some concern regarding specifically outlining the target population, which has now been explicitly addressed by expanding the guideline development team, addressing healthcare professionals as the primary user, and creating derivative products specific to secondary/tertiary users (i.e., parents, sport and education communities) previously included in the 2014 CPG. Rigour of Development has been improved by having the CPG operate as a living document on the ONF website, in which recommendations and domains will be updated as the evidence emerges. Having a living CPG on the ONF website addresses concerns about difficulty finding information with the addition of a search function, having too much or too little detail as information is now on easily digestible sub-pages, and the information being up-to-date as the CPG is a living document and will be updated as the evidence emerges. Clarity of Presentation has been enhanced by engaging a graphic designer and KT specialist to improve the language and design of the website and derivative products. Low Applicability ratings have been addressed; the new website will enhance accessibility of the guideline content and include language highlighting the context for which recommendations are best applied. Regarding Editorial Independence, guideline developers provided an unbiased environment to develop the guidelines, which set a high standard to ensure no undue influence.

In line with the literature, the Applicability domain had the lowest rating,^8,30–32 where low ratings are often attributed to a lack of noted facilitators or barriers to CPG use and vague implementation guidelines.³⁰ CPGs in other healthcare settings such as evidence-based traditional Korean medicine,³¹ as well as nutrition³² reported mean scaled Applicability domain ratings of the CPGs within their fields between 20–63% and 59–99%, respectively. One must note that it is unknown if the variability in scores can be solely attributed to the actual quality of the CPGs within these different healthcare settings since the current study revealed differences in AGREE II scores between rater demographic factors. However, the mean Applicability domain scores for the 2014 CPG fall above or within other CPGs in the healthcare field.

Unique to the field of pediatric concussion was the use of appraisers from various disciplines that were both guideline developers and end users, as well as the high number of appraisers engaged. The engagement of these stakeholders brings value of learning from the use and development of prior iterations. Results from this unique sample were stratified according to experts’ demographic factors and experience. Similar trends were found across domains. For example, physicians and allied health professionals consistently provided higher ratings across domains compared to other professions. This finding may be due to the recommendations in the 2014 CPG being aligned more explicitly with these professions compared to other professions (i.e., neuropsychologists, nurse practitioner, researchers), in which recommendations may have been perceived as ambiguous. Interestingly, less years of experience in an appraiser’s respective discipline revealed stronger ratings across the Scope and Purpose, Stakeholder Involvement, Clarity of Presentation, and Applicability domains compared to more years of experience. These domains (aside from Stakeholder Involvement) involve the use of the 2014 CPG rather than its development. It is suspected that more experience is associated with access to a greater breadth of knowledge for which to compare the current state of evidence and temporal patterns of clinical care resulting in less relevance of the scope/purpose and applicability of the contents of the 2014 CPG to more experienced appraisers. The remaining domains had high scores across all experience groups. Additionally, those practicing in a rehabilitation setting provided much lower domain ratings compared to those in other settings such as primary care, education, policy and research. Here, appraisers’ comments alluded to an inadequate representation of other experts, which have now been included in the development of the updated CPG. This finding may also merely reflect the difference in clinical care offered by primary care versus rehabilitation settings. Primary care settings may see pediatric concussion patients less frequently and in the acute phase of concussion, readily employing the decision-making matrices to assess the injury. Conversely, rehabilitation settings likely observe the complexities of concussion recovery (i.e., physical, social, psychological/emotional and cognitive sequalae), which may enhance the difficulty of deciding what ‘best practice’ intervention to apply resulting in more difficulty of use of the 2014 CPG. The vagueness of the target users in the 2014 CPG may have accentuated the lack of alignment with the objectives, stakeholder involvement, rigour of development, clarity of instructions and applicability for appraisers using the 2014 CPG in rehabilitation settings.

As stated by Brouwers et al.,¹⁵ the potential benefits of CPGs are only as good as the quality of the guideline itself, which is why it is critical to not only appraise CPGs but to involve key stakeholders in an integrated approach. Engaging in an iKT approach to CPG appraisal, as done in this study, brings greater accountability and credibility to practice recommendations, especially given that primary users of the guideline play a role in its development.⁹

Limitations

While the study provides a rigorous approach to revising a CPG, it is not without limitations. While the use of appraisers who were also end users was a unique strength of this study, some appraisers had contributed to the development of the 2014 CPG, which may enter bias into their scoring. Wilcoxon Rank-Sum tests were conducted for each domain between groups to investigate this bias, however no difference between the groups was measured. This study presented AGREE II domain ratings; however, the AGREE II user manual does not specify thresholds that distinguish a high-quality versus low-quality guideline, making it difficult to objectively confirm the quality of the evaluated CPG.¹⁸ The main purpose of the AGREE II tool is to assess the quality of structure and content,³⁰ but it does not explicitly evaluate the clinical utility/appropriateness of the recommendations themselves. Thus, high quality CPG content and structure does not guarantee better health outcomes for the target clinical population,³⁰ although it acts as a useful tool to inform usability while adding context that unless used, better health outcomes cannot be achieved.

Conclusion

High-quality CPGs reflect the perspectives of end users and other relevant stakeholder groups, are based in empirical evidence, are critically reviewed, and reflect specific recommendations.¹⁵ This study provided a foundational iKT framework to appraise a CPG, while honing in on key factors that make a strong CPG in the context of pediatric concussion. Poor uptake of CPGs continues to be a critical barrier to utilizing evidence informed information to make clinical decisions. Through an iKT approach, the critical appraisal of CPGs and subsequent KT strategies provide a foundation for which to revise and implement an updated CPG to more appropriately reach and be used by the target audience of healthcare providers. Further, CPG recommendations need to be simple and clear to minimize the cognitive load and increase uptake. This study supported that ease of access, succinct information presentation and a user-friendly approach are key to enabling target users to implement the CPG. Demographic factors within end users are important to consider given their impact on domain score ratings and CPG use. Thus, engaging end users in the CPG appraisal resulted in these important outcomes: (1) identified areas of the guidelines that may have affected the lack of previous clinical uptake through a mixed-methods, integrated knowledge translation study design while abiding by a clinical practice guideline development framework; (2) shared and informed decision making regarding content and format of revised clinical practice guidelines; and (3) targeted content, clinical questions and dissemination strategies, which are key to clinical uptake.

Footnotes

Acknowledgements

Thank you to the participants for their time and energy contributing to this study.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research,authorship,and/or publication of this article: RZ is supported by a Clinical Research Chair by the University of Ottawa Brain and Mind Research Institute. His research has been funded by competitive grants from the Ontario Neurotrauma Foundation,Ontario Brain Institute,Ontario SPOR Support Unit,Physician Services Incorporated,Children's Hospital of Eastern Ontario Research Institute,CIHR,and the National Football League. All other authors declare that there is no conflict of interest.

Funding

The authors thank project funders,the Ontario Neurotrauma Foundation for their support.

ORCID iD

Melissa Paniccia

References

Tator

CH.

Concussions and their consequences: current diagnosis, management and prevention. Can Med Assoc J 2013; 185: 975–979.

Zemek

Grool

Rodriguez Duque

, et al. Annual and seasonal trends in ambulatory visits for pediatric concussion in Ontario between 2003 and 2013. J Pediatr 2017; 181: 222–228.

Jolliffe

Morarty

Hoffmann

, et al. Using audit and feedback to increase clinician adherence to clinical practice guidelines in brain injury rehabilitation: a before and after study. Plos One 2019; 14: e0213525.

Makarski

Brouwers

MC.

The AGREE enterprise: a decade of advancing clinical practice guidelines. Implement Sci 2014; 9:103.

Institute of Medicine (US) Committee to Advise the Public Health Service on Clinical Practice Guidelines. Clinical practice guidelines: Directions for a new program. Washington (DC): National Academies Press (US), www.ncbi.nlm.nih.gov/books/NBK235751/ (1990, accessed 23 April 2018).

Amorin-Woods

Beck

Parkin-Smith

, et al. Adherence to clinical practice guidelines among three primary contact professions: a best evidence synthesis of the literature for the management of acute and subacute low back pain. J Can Chiropr Assoc 2014; 58: 220–237.

Reed N, Zemek R, Dawson J, et al. Ontario Neurotrauma Foundation. Guidelines for diagnosing and managing pediatric concussion. Toronto, ON: Ontario Neurotrauma Foundation, https://doi.org/10.17605/OSF.IO/3VWN9

Appenteng

Nelp

Abdelgadir

, et al. A systematic review and quality analysis of pediatric traumatic brain injury clinical practice guidelines. Plos One 2018; 13: e0201550–17.

Gagliardi

Berta

Kothari

, et al. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci 2015; 11: 38.

10.

Grol

Dalhuijsen

Thomas

, et al. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. Bmj 1998; 317: 858–861.

11.

Eady

Moreau

Horsley

, et al. Bridging the gap in paediatric concussion management. Paediatr Child Health 2016; 21: 6–8.

12.

Christakis

Rivara

FP.

Pediatricians’ awareness of and attitudes about four clinical practice guidelines. Pediatrics 1998; 101: 825–830.

13.

Francke

Smit

de Veer

, et al. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak 2008; 8: 38.

14.

Kastner

Estey

Perrier

, et al. Understanding the relationship between the perceived characteristics of clinical practice guidelines and their uptake: protocol for a realist review. Implement Sci 2011; 6: 69.

15.

Brouwers

Kho

Browman

, et al.; for the AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ Can Med Assoc J J Assoc Medicale Can 2010; 182: E839–842.

16.

Hoffmann-Eßer

Siering

Neugebauer

EAM

, et al. Guideline appraisal with AGREE II: systematic review of the current evidence on how users handle the 2 overall assessments. Plos One 2017; 12: e0174831.

17.

AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003; 12: 18–23.

18.

AGREE Next Steps Consortium. The AGREE II Instrument. http://www.agreetrust.org (2017).

19.

Eccles

Grimshaw

Shekelle

, et al. Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implement Sci 2012; 7: 8.

20.

Ontario Neurotrauma Foundation, https://onf.org/ (accessed 3 May 2021).

21.

Harris

Taylor

Thielke

, et al. Research electronic data capture (REDCap) – a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42: 377–381.

22.

Harris

Taylor

Minor

, et al. The REDCap consortium: building an international community of software partners. J Biomed Inform 2019; 95: 103208.

23.

Krippendorff

Content analysis. In: Internation encyclopedia of communication. New York, NY: Oxford University Press, 1989, pp.403–407.

24.

Elo

Kääriäinen

Kanste

, et al. Qualitative content analysis: a focus on trustworthiness. SAGE Open 2014; 4: 215824401452263.

25.

Pyett

PM.

Validation of qualitative research in the “real world”. Qual Health Res 2003; 13: 1170–1179.

26.

Thomas

Magilvy

JK.

Qualitative rigor or research validity in qualitative research: scientific inquiry. J Spec Pediatr Nurs 2011; 16: 151–155.

27.

Schutzman

Barnes

Duhaime

A-C

, et al. Evaluation and management of children younger than two years old with apparently minor head trauma: proposed guidelines. Pediatrics 2001; 107: 983–993.

28.

Cook

Schweer

Shebesta

, et al. Mild traumatic brain injury in children: Just another bump on the head? J Trauma Nurs 2006; 13: 58–65.

29.

Kamerling

Lutz

Posner

, et al. Mild traumatic brain injury in children: practice guidelines for emergency department and hospitalized patients. Pediatr Emerg Care 2003; 19: 431–440.

30.

Birken

Ellis

Walker

, et al. Guidelines for the use of survivorship care plans: a systematic quality appraisal using the AGREE II instrument. Implement Sci IS 2015; 10: 63.

31.

Choi

T-Y

Choi

Lee

, et al. The quality of clinical practice guidelines in traditional medicine in Korea: appraisal using the AGREE II instrument. Implement Sci 2015; 10: 1–10.

32.

Dedios

Esperati

De-Regil

, et al. Improving the adaptability of WHO evidence-informed guidelines for nutrition actions: results of a mixed methods evaluation. Implement Sci 2017; 12: 1–12.

Engaging target users to appraise and refine clinical practice guidelines in pediatric concussion: An integrated knowledge translation approach

Abstract

Background

Methods

Results

Conclusions

Keywords

Background

Methods

Context

Study design

Participants

Measures

Appraisal of Guidelines for Research and Evaluation II (AGREE II) tool 18

AGREE II score comments

Demographic information

Knowledge translation appraisal

Procedure

Analysis of data

Results

Participants

Appraisal of the 2014 CPG Using the AGREE II Tool

Domain 1: Scope and purpose

Domain 2: Stakeholder involvement

Domain 3: Rigour of development

Domain 4: Clarity of presentation

Domain 5: Applicability

Domain 6: Editorial independence

Use and influence of CPG 2014

CPG updates based on appraisal outcomes

Discussion

Limitations

Conclusion

Footnotes

Acknowledgements

Declaration of conflicting interests

Funding

ORCID iD

References

Appraisal of Guidelines for Research and Evaluation II (AGREE II) tool¹⁸