Sage Journals: Discover world-class research

Abstract

Purpose: Many elements contribute to practice excellence within child welfare services, yet there are limited measures available to assess these elements. This article describes the process of developing and pilot-testing a measure of child welfare practice excellence. Method: The Elements of Child Welfare Practice (ECWP) measure was developed following an extensive literature review, with input from child welfare research experts and an anti-colonial practitioner. The ECWP was part of a comprehensive online survey administered during site visits with three Indigenous child welfare agencies in Canada. Results: Polychoric correlations and ordinal alpha revealed the ECWP had strong internal consistency and convergent validity. The ECWP had three subscales related to the importance and delivery of practice elements, and workers’ perception of their practice. Conclusions: This measure showed the potential to be useful in assessing the degree to which child welfare workers intend to and actually engage in elements associated with practice excellence.

Keywords

child welfare practice scale development and testing ordinal alpha indigenous child welfare

Child welfare workers have a critical role in helping and supporting the needs of children and families, but few assessments clearly measure the essential elements of high-quality child welfare practice. The quality of child welfare practice is often discussed in terms of family engagement in the process (Toros et al., 2018), the use of empirically-supported practice skills (D’Aunno et al., 2014), client satisfaction (Tilbury & Ramsay, 2018; Mundy et al., 2016), and child and family outcomes (Forrester et al., 2019; Marsh et al., 2012). Excellence in practice may result in positive outcomes in these areas, but are not themselves descriptive of the quality of child welfare practice (Wells et al., 2015). In striving for excellence in child welfare service delivery, how the worker arrives at desired outcome for children and families, including the skills used, is equally as important as the outcomes themselves.

Current frameworks of child welfare practice elements include broad, comprehensive lists of work competencies (Drake, 1996) or outline general child welfare worker skills (Olin et al., 2014; Keys, 2009; Trotter, 2002; 2004). However, the abstract nature of the skills identified (e.g., empathy, listening) is often difficult to quantify and measure in direct practice with clients. Attempts have also been made to identify and measure the skills and abilities necessary to engage in high-quality direct practice with clients. In the extant literature, there are a number of measures of general social worker skills and competencies designed for specific worker populations (e.g., students, social workers; Jensen et al., 2020; Rowe et al., 2020) or environments (e.g., group work, healthcare settings; Davis et al., 2019; Macgowan et al., 2018), as well as measures that assess specific aspects of quality practice (e.g., cultural competence or relational skills; Leung & Cheung, 2013; Krentzman & Townsend, 2008; Poulin & Young, 1997).

Measures of Child Welfare Practice Skills

A review of measures related to child welfare worker skills revealed three notable scales: (1) The Self-Efficacy in Child Welfare Tasks Scale (SECWT; Ellett, 2009); (2) The Practice Skills Inventory (PSI; O’Hare et al., 1998); and (3) the Case Skills Scale, as part of a child welfare decision-making measure (CSS; Dettlaff et al., 2015). The SECWT (Ellett, 2009 measures the degree to which child welfare workers believe in their capacity to work within their agency, as well as persevere and affect client outcomes. The PSI (O’Hare et al., 1998) measures the frequency in which social workers engage in important practice elements (e.g., client support, insight (e.g., relational skills), therapeutic skills, and case management (e.g., practice skills). Finally, the CSS (Dettlaff et al., 2015) measures child welfare workers perceived skill level across a set of concrete direct practice elements (e.g., interpersonal and case management skills).

Each of these measures has unique strengths, such as clearly operationalized items of measurement (CSS; Dettlaff et al., 2015), confidence in practice skills (SECWT; Ellett, 2009), and assessment of the frequency of skill usage (PSI; O’Hare et al., 1998). However, there are also some important measurement limitations, such as a lack of coverage of cultural competence (CSS; Dettlaff et al., 2015 and PSI; O’Hare et al., 1998), a reliance on workers’ assessment of their own skill level (CSS; Dettlaff et al., 2015 and SECWT; Ellett, 2009), abstract items of measurement (PSI, SECWT and Ellett, 2009), or a focus on general social work competencies above and beyond direct practice with children and families (PSI; O’Hare et al., 1998 and SECWT; Ellett, 2009). We were unable to identify a single self-report measure that: (1) had clearly operationalized concepts, (2) measured the workers’ cultural competence, and (3) measured the workers’ direct practice activities with clients. Notably, some measures ask workers to assess their own skill level, but workers may lack the training to understand how to do so. In the present study, we discuss the development of a measure that unites the strengths of these scales and addresses their limitations, resulting in a comprehensive and concise measure of the elements of child welfare practice (ECWP) excellence.

Purpose of the Current Study

In a larger mixed-methods project examining the organizational environment and approaches to practice with Indigenous child welfare agencies, we sought to develop and test the ECWP measure, a self-report measure assessing relational skills, CPS practice skills, and cultural competence among child welfare agency workers. While this project was not an instrument development study, the process of scale development generally followed the phases outlined by Boateng et al. (2018), which included: (1) developing a conceptual framework and item development, (2) scale development, and (3) scale evaluation. In this article, we review the process of developing a conceptual framework of child welfare practice excellence, including item and scale development, discuss the administration of the measure with three Indigenous child welfare agencies in Canada, and detail the process of scale testing for internal consistency and convergent validity. Notably, while this measure was tested with workers at three Indigenous child welfare agencies in Canada, the underlying conceptual framework applies to child welfare practice across diverse populations.

Method

Conceptual Framework for ECWP Excellence

Child welfare practice excellence encapsulates two primary aspects: (1) the major tasks workers are required to perform and (2) the ability to succeed in those tasks. Workers are required to perform the tasks of working effectively within their organization in ways that align with the agency's mission, value, and vision (Drake, 1996), navigate interorganizational relationships and requirements (Drake, 1996; Trotter, 2004), and engage in direct practice with children and families (Drake, 1996; Trotter, 2004). Based on a comprehensive review and Wells et al., 2015, we developed a conceptual model related to the evaluation of the quality of direct practice with children and families. Figure 1 presents the conceptual model for child welfare practice excellence. In addition to the literature review, we used key informants in the field, including researchers at the National Child Welfare Workforce Institute, the Quality Improvement Center for Workforce Development, and the Institute for the Advancement of Social Work Research, to identify self-report measures of child welfare workers’ quality of direct practice with clients. However, the intent of this scale was not to measure all aspects of child welfare practice quality, but instead, to focus on the skills and abilities of workers engaged in practice with children and families.

Figure 1.

Conceptual model for evaluating child welfare practice excellence.

Direct practice with children and families includes three key aspects of practice excellence related to: (1) relational skills, (2) CPS practice skills, and (3) cultural competence (Carnochan et al., 2019; Mundy et al., 2016). Relational skills refer to the worker's ability to empathize with clients and build supportive working relationships that engage children and families in services (Barth et al., 2012; Lundahl et al., 2020; Sinai-Glazer, 2020; Wampold, 2015). Workers foster trusting relationships by actively listening, empathizing with clients’ concerns, and being accessible (Barth et al., 2012; Fuller et al., 2015; Gladstone et al., 2014; Schreiber et al., 2013; Wampold, 2015). CPS practice skills directly facilitate change for clients, such as goal setting and service planning (Barth et al., 2012; Carnochan et al., 2019; Wampold, 2015). Workers must be able to define their roles and expectations, provide knowledge and resources to clients, solve problems collaboratively, and serve as coaches, mediators, and advocates (Chapman et al., 2003; Damiani-Taraba et al., 2017; Fuller et al., 2015; Gladstone et al., 2014; Olin et al., 2014; Schreiber et al., 2013; Trotter, 2002). Finally, cultural competence ensures that workers will treat each client with respect and consideration while offering culturally-appropriate services that incorporate the beliefs and customs guided by the families served (Chapman et al., 2003; Damashek et al., 2012; DeSilva et al., 2018; Wampold, 2015).

Item Development. Item development involves the identification of conceptual domains (i.e., theoretical framework) and item generation, taking into consideration the content and face validity of items. Following our examination of existing measures, a review of the literature, and consultation with child welfare experts, we developed a pool of preliminary scale items related to the conceptual framework and reduced items to fully cover concepts without being redundant. A list of practice elements and preliminary items are presented in Table 1. Items for the ECWP scale were deductively developed by the team, which included an expert in child welfare research and practice, an anti-colonial practitioner with experience working in child welfare, and an investigator with research and practice experience in child welfare, along with a team of research assistants. Items were specifically developed to align with the three main concepts identified in the conceptual framework, which served as the measure's subscales. Items were included after consensus among the investigators, with significant input from those with child welfare practice experiences.

Table 1.

Elements of Direct Practice Excellence and Possible Scale Items Based on the Extant Literature.

Element of practice	Worker skills and abilities	Empirical support	Possible scale items
Relational skills	Worker treats clients with respect	Chapman et al., 2003; Drake, 1996; Fuller et al., 2015; Horwitz & Marshall, 2015; Jensen et al., 2020; Mundy et al., 2016	…show respect for my client's views.
	Worker takes non-judgmental approach	D’Aunno et al., 2014; Drake, 1996; Gladstone et al., 2014; Olin et al., 2014; Schreiber et al., 2013	… uses non-judgmental language to communicate acceptance.
	Worker asks clients for their views	Harris et al., 2000; Horwitz & Marshall, 2015; Kapp & Vela, 2004; Mundy et al., 2016	… ask my clients what is important to them.
	Worker is empathetic and supportive	Fuller et al., 2015; Keys, 2009; Palmer et al., 2006; Schreiber et al., 2013; Trotter, 2004	… understand my clients’ situations from their perspectives.… take the time to listen to my clients’ concerns.… be supportive of my clients under the circumstances.
	Worker is honest and trustworthy	Jensen et al., 2020; Mundy et al., 2016; Trotter, 2004	… be upfront with my clients. … be open with my clients about my concerns.
	Worker partners with clients	Chapman et al., 2003; Drake, 1996; Fuller et al., 2015; Horwitz & Marshall, 2015; Jensen et al., 2020; Mundy et al., 2016	… my relationship with my clients is a partnership.
	Worker spends time with clients	Harris et al., 2000; Mundy et al., 2016	… take the time to get to know my clients. … see clients regularly.
	Worker is dependable and available to clients	Becker et al., 2016; Gladstone et al., 2014; Jensen et al., 2020; Mundy et al., 2016; Schreiber et al., 2013; Trotter, 2004	… if I tell my clients I will do something for them, I do it.… I am easily available to my clients by email or phone.… meet with my clients at the agreed-upon time.
CPS practice skills	Worker meets with clients to develop a service plan to meet their needs	Chapman et al., 2003; Jensen et al., 2020; Olin et al., 2014	… meet with my clients to help them create a service plan.
	Worker gives clients a copy of their service plan	Mundy et al., 2016	… give my clients a copy of their service plan.
	Worker uses the service plan as discussed with clients	Mundy et al., 2016	… use the case plan to guide my activities with the client.
	Worker involves clients in planning	D’Aunno et al., 2014; Gladstone et al., 2014; Jensen et al., 2020; Mundy et al., 2016; Olin et al., 2014	… incorporate my clients’ perspectives into their service plan. … involve my clients in each stage of planning.
	Worker monitors clients’ progress	Olin et al., 2014	… keep my clients up to date on their progress.
	Worker addresses adequacy of existing service plan as necessary	Olin et al., 2014	…work with my clients to update their service plan as necessary.
	Intensity & duration of service meets clients’ needs	D’Aunno et al., 2014	…match services to my clients’ unique needs.
	Worker's expectations of clients are reasonable	Harris et al., 2000; Kapp & Vela, 2004; Mundy et al., 2016	… meet my clients “where they are at” in the change process.… help my clients set realistic goals.… help my client set manageable goals.
	Worker identifies safety concerns	Olin et al., 2014	… identify safety concerns.
	Worker is clear about agency or province concerns and expectations	Horwitz & Marshall, 2015; Mundy et al., 2016	…I am clear with my clients about what the agency expects from them.…I am clear with my clients about what it will take to close their file.
CPS practice skills	Worker is clear about their role with clients	Keys, 2009; Olin et al., 2014; Palmer et al., 2006; Trotter, 2002	… I am able to be clear about my role with clients.… set clear boundaries about what I am able to do and not able to do for clients.
	Worker explains to options client, given their current situation	Chapman et al., 2003	… help my clients understand the options available to them.
	Worker clearly explains decisions that are made	Becker et al., 2016; Mundy et al., 2016; Schreiber et al., 2013	… make sure my clients understand each decision that is made.
	Worker uses strengths-based language to talk about what's going well with clients	Horwitz & Marshall, 2015; Mundy et al., 2016; Olin et al., 2014; Schreiber et al., 2013	… make sure my clients know when they are making good progress. … remind my clients of their strengths.
	Worker has knowledge of supports and evidence-based practice	Fuller et al., 2015; Gladstone et al., 2012; Olin et al., 2014; Schreiber et al., 2013	… I am well-connected in the network of community service providers. … I am knowledgeable about the most effective way to help my clients.
	Worker identifies barriers to positive change	Horwitz & Marshall, 2015	… identify my clients’ challenges and barriers to making positive change.
	Worker identifies clients’ immediate concrete needs	D’Aunno et al., 2014; Horwitz & Marshall, 2015; Jensen et al., 2020	…identify client's concrete needs (e.g., food, housing, job).
	Worker helps meet clients’ concrete needs	D’Aunno et al., 2014; Fuller et al., 2015; Horwitz & Marshall, 2015; Palmer et al., 2006; Schreiber et al., 2013	… provide my clients with material resources if they are in need.… provide my clients with financial resources if they are in need.
	Worker expands clients’ social support systems	Horwitz & Marshall, 2015; Mundy et al., 2016; Olin et al., 2014	… encourage my clients to involve their friends and extended families in the change process.
Cultural competence	Worker asks about the client's culture	Damashek et al., 2012; D’Aunno et al., 2014; Drake, 1996; Mundy et al., 2016	…ask my clients about their cultural views.… take the time to understand my client's culture.
	Worker uses culturally relevant clinical skills in practice with clients	D’Aunno et al., 2014; Fuller et al., 2015; Keys, 2009; Mundy et al., 2016; Olin et al., 2014; Trotter, 2002	…help my clients incorporate aspects of their culture into their service plan.… help my clients develop skills they may be missing or need.… help my clients make better decisions for themselves.… help my clients become independent.
	Worker respects client's culture	Damashek et al., 2012; D’Aunno et al., 2014; Drake, 1996; Mundy et al., 2016	…ask my clients about their cultural views…connect my clients with culturally relevant services… I am mindful of my clients’ cultural values.… I am mindful of sensitive cultural matters.… I try not to impose my values on my clients.

Note. Items in bold font were used to inform items for the ECWP. Future iterations of the scale may include additional items and constructs.

The investigators and team independently rated the full pool of preliminary items for their theoretical importance to direct child welfare practice excellence. The research team then narrowed down the item pool to items that met the following criteria: (1) items deemed most essential to child welfare practice excellence, based on empirical literature and expertise of the investigators, (2) items that could be behaviorally operationalized and were concrete enough to be commonly understood across respondents in the child welfare field, and (3) items expected to produce variability in responses. For example, identifying safety concerns are a key element of child welfare practice; thus, would be expected to show a ceiling effect and little variability. Indicators were behavioral and spanned the aforementioned framework of relational skills, CPS practice skills, and cultural competence. Elements identified have been linked to child and family outcomes, family engagement, client satisfaction, and workers’ perceptions of quality practice (D’Aunno et al., 2014; Forrester et al., 2019; Marsh et al., 2012; Mundy et al., 2016; Tilbury & Ramsay, 2018; Toros et al., 2018). Table 1 outlines the theoretical foundations of the literature review and item development for future researchers who wish to continue the development of the ECWP, and provides a list of important child welfare worker skills and abilities, literature sources, and a sample of related scale items.

Additionally, the extant literature has shown that what workers considered to be important to their practice does not always align with what they were able to do in their practice (Aarons et al., 2011). Thus, we wanted to examine which elements workers deemed to be important to their practice and the frequency with which these elements were delivered in their practice. Finally, to fully capture all aspects of practice, we wanted to include items that measured the workers’ perceptions of their practice, similar to measures of practice self-efficacy (Ellett, 2009). Of note, this self-report measure asks workers to report the degree to which elements of practice excellence are important for their practice, how often those elements are delivered, and how they feel about their practice. These items are suitable for self-report, whereas questions about skill level may present a challenge, particularly for new child welfare workers. This scale has the potential to offer the most precise picture of direct child welfare practice excellence, including the elements workers think are important to practice, the elements delivered in their practice, and the perception of their practice.

The resulting 38-item pilot measure represented a first attempt to develop and evaluate such a scale for future use with child welfare workers. This newly developed scale aimed to assess ECWP excellence among child welfare workers and includes items that address the importance of specific elements of practice (Importance Subscale), how often elements were delivered (Delivery Subscale), and workers’ perceptions of their practice (Perceptions Subscale). Responses for the ECWP scale were given on 5-point Likert-type scales. The Importance and Delivery subscale items had the same 17 elements related to practice excellence (e.g., relational skills, CPS practice skills, and cultural competence), but asked either “how important is it to” (1 = Not at all to 5 = Extremely) or “how often are you able to” (1 = Seldom to 5 = Always). The Perceptions Subscale asked respondents their level of agreement with four items about their practice (1 = Strongly disagree to 5 = Strongly agree). The full ECWP scale is presented in Table 2.

Table 2.

Full ECWP Items and Responses (N = 53).

	How important is it to… \| How often are you able to…	Important						Often
		Not	Some	Mod	Very	Extre	M	Sel	Some	Often	Very	Alw	M
Relational	1. be upfront with my clients.	1	2	4	13	33		2	6	9	12	24
Relational	2. take the time to get to know my clients.	0	3	3	16	31		2	10	10	16	15
Relational	3. ask my clients what is important to them.	0	1	3	17	32		1	7	10	11	24
Relational	4. be open with my clients about my concerns.	1	1	4	13	34		3	5	10	10	25
CPS practice	5. help my clients understand the options available to them.	0	3	3	22	25		3	8	18	10	13	1
CPS practice	6. meet with my clients to help them create a service plan.	2	3	5	23	19	1	3	9	21	8	11	1
CPS practice	7. incorporate my clients’ perspectives into their service plan.	2	2	3	18	27	1	2	7	12	9	22	1
CPS practice	8. involve my clients in each stage of planning.	1	2	5	18	27		2	10	11	13	17
CPS practice	9. make sure my clients understand each decision that is made.	0	3	1	20	29		2	7	10	15	19
CPS practice	10. make sure my clients know when they are making good progress.	0	1	6	18	28		1	7	14	9	22
CPS practice	11. remind my clients of their strengths.	0	1	2	17	33		1	4	7	18	23
CPS practice	12. give my clients a copy of their service plan.	2	2	3	17	28	1	5	9	10	11	17	1
CPS practice	13. encourage my clients to involve their friends and extended families in the change process.	0	3	5	15	29	1	1	9	10	10	22	1
CPS practice	14. see clients regularly.	0	3	9	18	23		2	10	16	14	11
Cultural	15. ask my client questions about their cultural values and views.	0	3	1	16	33		1	8	11	10	23
Cultural	16. help my clients incorporate aspects of their culture into their service plan.	1	2	1	15	34		3	8	8	16	18
Cultural	17. connect my clients with culturally-relevant services.	1	0	5	17	30		3	9	16	9	16
	I often feel that…	SD	D	N	A	SA	M
Perceptions	18. I am well-connected in the network of community service providers.	1	5	13	25	8	1
Perceptions	19. my relationship with my clients is a partnership.	0	1	4	25	25	1
Perceptions	20. I am knowledgeable about the most effective way to help my clients.	0	5	8	29	10	1
Perceptions	21. I am able to be clear about my role with clients	0	2	2	28	20	1

Note. Importance Scale response set: not at all (Not), somewhat (Some), moderately (Mod), very (Very), extremely (Extre); Delivery Scale response set: seldom (Sel), sometimes (Some), often (Often), very often (Very), always (Alw); Perceptions Scale response set: strongly disagree (SD), disagree (D), neither agree nor disagree (N), agree (A), strongly agree (SA); missing response = M.

Scale Development and Evaluation. The second phase of instrument development includes pre-testing questions, sampling and survey administration, item reduction, and extraction of latent factors (Boateng et al., 2018). Sampling and survey administration are discussed in detail in the procedures and participants section. To evaluate the ECWP, we conducted univariate analyses and polychoric correlations to determine the statistical utility of individual items. Items showing a ceiling effect or limited variability were considered for removal. Additionally, within scales, we examined the level of correlation to determine the redundancy of items, then used the theoretical framework to determine if the item represented a unique construct or if it should be omitted. The final phase of instrument development involves scale evaluation, which includes testing of dimensionality, reliability, and validity of scales (Boateng et al., 2018). Given the small sample size, it was not feasible to conduct a confirmatory factor analysis. Instead, we utilized polychoric correlations to examine the dimensionality of the scales and then calculated ordinal alpha as a test of reliability (Gadermann et al., 2012; Stene et al., 2020; Zumbo et al., 2007). Lastly, we examined the convergent validity of the ECWP scale with the CSS (Dettlaff et al., 2015). Although the CSS (Dettlaff et al., 2015) relies on workers’ perceptions of self and lacks measurement of cultural competence in direct practice, we elected to include it in the survey as a validation measure. The CSS (Dettlaff et al., 2015) is closely conceptually related to the ECWP and has a number of strengths, such as its operationalization of indicators of quality, strong internal consistency (Cronbach's α > 0.94), the large sample used in development (N = 1,125), and a conceptual linkage to the measurement of interpersonal skills (i.e., relational skills).

Procedure and Participants

Participant included workers and supervisors from three delegated Indigenous child welfare agencies in Western Canada. All workers and supervisors in the agency were invited to participate in an online survey containing the ECWP scale during the site visit. Of note, the Canadian Child welfare system has the overarching goal of preventing and protecting children and youth from maltreatment and provides a variety of family support services and child protection services, including risk assessments and out-of-home placements (e.g., kinship care, foster care). Child welfare agencies in Canada are responsible for completing safety assessments, making care plans, and providing support services for families. However, child welfare service provisions in Canada are guided by policies at the provincial or territorial level and vary across the country (Public Health Agency of Canada [PHAC], 2019).

Regarding Indigenous child welfare in Canada, the rights and responsibilities for the provision of child and family welfare services were forcibly removed from Indigenous communities through the Indian Act (1951;1985). The recently passed Bill C-92: An Act respecting First Nations, Inuit, and Métis children, youth, and families (2019), represents the first federal law that affirms the rights of Indigenous communities across Canada to have jurisdiction over child and family services thus, restoring the rights and responsibilities for the provision of child and family welfare services to Indigenous communities. To reclaim these rights, many Indigenous communities have moved forward in the development of their own child welfare agencies and have sought out delegated status (referred to as Delegated Aboriginal Agencies (DAAs); Blackstock & Trocmé, 2005; Haight et al., 2018; MCFD, 2021). Delegation indicates the agency has an agreement with the Provincial Ministry of Children and Families (MCFD) to provide services under the Child, Family, and Community Service Act (1996), serve Indigenous communities both on- and off-reserve, and are often Indigenous-led. As a result, the approach to child welfare practice is often agency developed with culturally-relevant practices and directed by provincial standards. However, data collection and engagement with the three DAAs occurred prior to the passing of Bill C-92.

Workers and supervisors were given paid work time to complete the survey during site visits as part of the larger mixed-methods research project examining organizational environment and approaches to practice. The online survey began with a consent form, and included the newly developed ECWP scale and the CSS, an established scale related to practice skills (Dettlaff et al., 2015), along with multiple measures of organizational environment and learning (Potter et al., 2016; Watkins & Marsick, 1997) and a demographic and worker characteristics questionnaire. Participants were anticipated to need 60 min to complete the online survey. A total of 91 participants were invited to complete the survey, with an average response rate of 62.2% (range: 51.3%–73.1%) across the three agencies.

In total, 65 workers and supervisors consented to participate in the survey, with 53 participants completing the ECWP scale. These 53 cases were used for data analysis. Participants were mostly female (84.9%) and between the ages of 30–49 (52.8%). Participants self-identified as Indigenous (45.3%), non-Indigenous (45.3%), or did not respond (9.4%). Participants included caseworkers (50.9%), agency workers (32.1%, e.g., family support workers, kinship workers), and team leaders (15.1%), with (1.9% (n = 1) did not respond). Few participants (5.6%) had less than one year of child welfare experience, 35.9% had 1–4 years, 20.8% had 5–9 years, and 37.7% had >10 years of child welfare experience (2% (n = 1) did not respond). Participants had been with their current agency for an average of 3.4 years (SD = 3.6), with 43.4% reporting one year or less with the agency, 35.9% reporting 2–5 years, and 20.8% reporting more than five years with their current agency. At the time of the survey, 20.8% of participants had no open active cases, 11.3% had 1–10 cases, 26.4% had 11–20 cases, 26.4% had 21–30 cases, and 15.1% had more than 30 cases. To protect the confidentiality of the participants, we only calculated aggregate descriptive data across all three sites. This study received approval from the University of British Columbia—Okanagan Behavioral Research Ethics Board, from the three partnering agencies’ Boards, and participants completed consent forms.

Analytic Approach

First, response frequencies for the ECWP scale were assessed for missingness and lack of variability, with no items excluded based on these criteria. Polychoric correlation analyses were used to support item reduction and testing of dimensionality for the ECWP scale, with ordinal alpha and inter-scale correlations used as a test of internal consistency. Ordinal data from Likert-type items are often treated as continuous, which may lead to spurious results and the overestimation or underestimation of true relationships (Gadermann et al., 2012; Zumbo et al., 2007). To accurately estimate the relationship between items, polychoric correlations (the equivalent of Pearson's correlations) and ordinal alphas (the equivalent of Cronbach's alpha) were employed. Additionally, for smaller sample sizes, polychoric correlations and ordinal alpha are more appropriate to determine item reduction and to test dimensionality (Gadermann et al., 2012; Stene et al., 2020; Zumbo et al., 2007). Items with a polychoric correlation <|.30| were marked as weak correlations and items with a polychoric correlation >|.90| were marked as very highly correlated, indicating the items may be redundant (Cohen, 1988). For items falling into either category, the decision to retain or disregard was based on the conceptual model and discussions among the team members. Thus, in some cases, items showing very high or very low interitem correlations were not excluded based on theoretical support and the expert opinion of investigators. The ECWP subscales related to the Importance, Delivery, and Perceptions of practice were compared to each other, and also compared to the CSS (Dettlaff et al., 2015). While these scales measure different constructs, we hypothesized that the subscales would be positively correlated with each other and with the CSS. Analyses were completed in Stata/MP 15 (StataCorp, 2017) and Rstudio (RStudio Team, 2020).

Results

Item and Scale Descriptions and Reduction

In total, the ECWP measures which elements of practice were considered important and how often workers delivered those elements, specifically related to three aspects of practice: (1) relational skills, (2) CPS practice skills, and (3) cultural competence, as well as the workers’ perception of their practice. Of the original 38 items (17 importance items, 17 delivery items, and 4 perception items), all items were retained. Of note, there were no discernible missing data patterns, and it appeared items were missed at random. Little's test of missing completely at random was not suitable due to the small sample size (Harris et al., 2000). When examining the distribution of responses across items, we found that on the Importance subscale, few respondents selected the “Not at All” response, with most responses falling in the Moderate, Very, or Extremely categories. The Delivery subscale was also negatively skewed, but showed a more equal distribution across the response set. Finally, the Perception subscale had only one strongly disagree response and only a few disagree responses.

To examine the item- and scale-relatedness, we conducted polychoric correlations. The polychoric correlation matrices yielded results ranging from r = 0.325 to 0.922 for the Importance subscale, from r = 0.291 to 0.912 for the Delivery subscale, and from r = 0.445 to 0.679 for the Perceptions subscale, and showed clear divisions between the subscales. Full Polychoric correlation matrices for the Importance, Delivery, and Perceptions Scales are presented in Tables 3–5, respectively. Items were examined for redundancy (polychoric correlation |≥0.90|) or lack of conceptual relatedness (polychoric correlation |≤0.30|). For the Importance subscale, Item 9 (make sure my clients understand each decision that is made) and Item 7 (incorporate my clients’ perspective into their service plan) were highly correlated (r = 0.922). Also, Item 10 (make sure my clients know when they are making good progress) and Item 17 (connect my clients with culturally relevant services) were highly correlated (r = 0.922). For the Delivery subscale, Items 7 (incorporate my clients’ perspectives into their service plan) and Item 3 (ask my clients what is important to them) were highly correlated (r = 0.91), whereas Item 4 (be open with my clients about my concerns) and Item 2 (take the time to get to know my clients) showed a weak correlation (r = 0.291), just below the cutoff. For each of these pairings, the items were retained because they reflect different aspects of practice and were only slightly above/below the cutoff values. Had an item shown redundancy or lack of conceptual relatedness across both the Importance and Delivery subscales, that item would have likely been excluded from the ECWP scale. Yet, this did not occur with this sample.

Table 3.

Polychoric Correlations for ECWP Excellence—Importance Subscale.

	How important is it to…	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
1	be upfront with my clients.	-
2	take the time to get to know my clients.	.611	-
3	ask my clients what is important to them.	.517	.731	-
4	be open with my clients about my concerns.	.589	.570	.713	-
5	help my clients understand the options available to them.	.470	.502	.496	.504	-
6	meet with my clients to help them create a service plan.	.447	.609	.748	.569	.750	-
7	incorporate my clients’ perspectives into their service plan.	.464	.685	.890	.652	.537	.731	-
8	involve my clients in each stage of planning.	.548	.576	.825	.633	.596	.711	.734	-
9	make sure my clients understand each decision that is made.	.495	.815	.849	.748	.587	.718	.922	.797	-
10	make sure my clients know when they are making good progress.	.446	.628	.820	.646	.673	.769	.810	.827	.845	-
11	remind my clients of their strengths.	.593	.663	.551	.530	.513	.552	.620	.548	.617	.692	-
12	give my clients a copy of their service plan.	.639	.352	.666	.595	.510	.718	.701	.763	.721	.812	.659	-
13	encourage my clients to involve their friends and extended families in the change process.	.599	.570	.537	.443	.598	.689	.615	.670	.665	.700	.632	.825	-
14	see clients regularly.	.515	.632	.878	.545	.533	.681	.869	.808	.823	.834	.664	.708	.660	-
15	ask my client questions about their cultural values and views.	.520	.605	.828	.494	.404	.614	.622	.827	.721	.752	.576	.789	.720	.621	-
16	help my clients incorporate aspects of their culture into their service plan.	.325	.570	.708	.484	.412	.628	.747	.677	.767	.713	.656	.651	.618	.646	.873	-
17	connect my clients with culturally relevant services.	.466	.566	.821	.608	.607	.720	.739	.749	.790	.922	.599	.739	.706	.715	.856	.822	-

Note. Coefficients less than |0.30| or greater than |0.90| were examined. All items were significant. Bold reflects items with very high or very low levels of correaltion.

Table 4.

Polychoric Correlations for ECWP Excellence—Delivery Subscale.

	How often are you able to…	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
1	be upfront with my clients.	-
2	take the time to get to know my clients.	.631	-
3	ask my clients what is important to them.	.551	.651	-
4	be open with my clients about my concerns.	.486	.291	.588	-
5	help my clients understand the options available to them.	.635	.569	.674	.596	-
6	meet with my clients to help them create a service plan.	.513	.635	.753	.560	.617	-
7	incorporate my clients’ perspectives into their service plan.	.605	.696	.912	.653	.635	.711	-
8	involve my clients in each stage of planning.	.519	.584	.762	.744	.677	.730	.811	-
9	make sure my clients understand each decision that is made.	.529	.635	.743	.644	.632	.692	.873	.876	-
10	make sure my clients know when they are making good progress.	.477	.578	.738	.689	.736	.755	.752	.885	.865	-
11	remind my clients of their strengths.	.623	.673	.734	.661	.672	.663	.827	.788	.861	.840	-
12	give my clients a copy of their service plan.	.483	.344	.565	.607	.539	.650	.543	.620	.620	.571	.632	-
13	encourage my clients to involve their friends and extended families in the change process.	.584	.548	.675	.615	.650	.702	.674	.635	.647	.700	.772	.712	-
14	see clients regularly.	.537	.696	.681	.487	.619	.578	.697	.622	.576	.677	.625	.448	.594	-
15	ask my client questions about their cultural values and views.	.593	.477	.653	.631	.449	.739	.641	.715	.653	.686	.742	.671	.723	.433	-
16	help my clients incorporate aspects of their culture into their service plan.	.674	.473	.649	.686	.585	.676	.691	.709	.631	.665	.755	.717	.734	.627	.851	-
17	connect my clients with culturally-relevant services.	.496	.391	.445	.662	.670	.573	.528	.646	.579	.758	.695	.589	.748	.663	.660	.796	-

Note. Coefficients less than |0.30| or greater than |0.90| were examined. All items were significant. Bold reflects items with very high or very low levels of correaltion.

Table 5.

Polychoric Correlations for ECWP Excellence—Perceptions Subscale

	I often feel that…	1	2	3	4
18	I am well-connected in the network of community service providers.	-
19	My relationship with my clients is a partnership.	.462	-
20	I am knowledgeable about the most effective way to help my clients.	.557	.679	-
21	I am able to be clear about my role with clients	.535	.445	.665	-

Note. Coefficients less than |0.30| or greater than |0.90| were examined.

Polychoric correlations were then used to calculate ordinal alphas for the ECWP scale and its subscales. Table 6 presents the ordinal and Cronbach's alphas, as well as the ECWP subscale and total mean scale scores. Using polychoric correlations and ordinal alpha, the total scale and subscales showed excellent internal consistency (ordinal α = 0.824–0.975), which were comparable and slightly higher than the Cronbach's alphas (Cronbach's α = 0.773–0.965). We also calculated and examined the mean scale scores for each subscale and found that the Importance subscale was higher than the Delivery subscale when examining the individual items for discrepancies between responses to Importance and Delivery items. Notably, Item 14 (sees clients regularly) had 43.4% reporting this was extremely important, with 20.8% reporting they are always able to do so. For Item 3 (Take time to get to know my clients), 58.5% reported this as extremely important, with 22.6% reporting they seldom or only sometimes did this and only 28.3% reporting they do this always.

Table 6.

ECWP Scale and Subscale Ordinal and Cronbach's Alphas and Means.

Scales	Number of items	Ordinal Alpha	Cronbach's Alpha	M (SD)
Importance	17	0.972	0.953	4.37 (0.66)Range: 2.1–5.0
Delivery	17	0.971	0.958	3.72 (0.93)Range: 1.0–5.0
Perceptions	4	0.824	0.773	4.02 (0.62)Range: 2.0–5.0
Total scale	38	0.975	0.965	4.04 (0.62)Range: 2.2–5.0

Note. The table indicates the number of items in each scale as well as the ordinal alphas for each. The Importance and Delivery Items are the same, with a different lead in question (How important is… and How often do you…, respectively) and response sets (Not at all important–Extremely important and Seldom–Always, respectively).

Finally, to examine inter-scale relationships and test convergent validity, we examined the correlations between the ECWP Importance, Delivery, and Perception subscale mean scores with the CSS (Dettlaff et al., 2015) score. There was a strong positive correlation between the Importance and Delivery subscales (r = 0.610, p < .001), suggesting these were related but not redundant scales. The Perception scale also had a strong positive correlation (r = 0.621, p < .001) with the Delivery scale and a moderate-weak positive correlation (r = 0.392, p = .004) with the Importance scale. This suggests that the worker's perception of the practice was more strongly related to the practice delivery than to what they believed to be important for practice. Finally, we saw moderate to strong positive correlations between the CSS (Dettlaff et al., 2015) and each subscale (r = 0.448–0.631). See Table 7 for correlations between subscales and CSS (Dettlaff et al., 2015).

Table 7.

Pearson Correlation Matrix for the ECWP Mean Subscales Scores and CSS Mean Scale Score

Scale	Importance	Delivery	Perception	Case Skills Scale
Importance	-
Delivery	0.610**	-
Perceptions	0.392*	0.621**	-
Case Skills Scale	0.448*	0.426*	0.631**	-

Note. Pearson correlations matrix examining the relationships between the ECWP subscales and the CSS (Dettlaff et al., 2015).

*p < .01, **p < .00.

Discussion

The purpose of this study was to outline the development and testing of a novel measure of child welfare practice excellence. Building from an extensive literature review, we developed a measure of child welfare practice that covered items related to relational skills, CPS practice, and cultural competence, with dual scales addressing the importance of each element and how often each element was delivered as part of their practice. The scale also included four items related to the perception workers had about their practice. Overall, we found high ordinal alphas and inter-scale correlations for the ECWP, as well as convergent validity between the ECWP subscales and the CSS (Dettlaff et al., 2015), suggesting this is a promising measure of child welfare practice. We also considered the ability of the scale to identify the difference between what workers reported was important as compared to what they were doing with clients. We found that the Importance subscale tended to be negatively skewed, but did not have significant concerns with ceiling effects. This supported our initial decision to provide more response options at the positive end of the scale. However, future iterations of the scale may want to consider responses that validate the importance of each element, while also measuring the degree of importance (e.g., Least important to Most important). We also saw a trend toward negative skew in the Delivery scale, again with no concerns related to ceiling effects. Thus, it appeared the response set for the Delivery scale was well-suited with this sample. Finally, respondents mostly agreed or strongly agreed with the items in the Perception, which may indicate that workers within these agencies were confident about their abilities. However, given that this sample was small and focused on Indigenous child welfare agencies, further testing is required to support these findings, and should include focus groups with child welfare workers to ensure the meaningfulness of these scales, items, and response sets.

Limitations. This study was part of a larger mixed-methods project examining organizational environment and approach to practice and was not specifically designed to be a measurement validation study. Further, given the size of the sample, we were limited in our analytic approach for the validation of this measure. Future studies should include larger samples to enable the use of advanced psychometric statistical approaches (e.g., item response theory, factor analysis) to test the scale dimensions, reliability, and validity. Additionally, it would also be beneficial to include focus groups and consultation with current child welfare workers and agency leadership regarding the items and scales included, as well as the response sets used. Finally, while this scale was developed using a comprehensive literature review, it was tested among three Indigenous child welfare agencies from a similar geographic location in Western Canada. The scale may function differently among workers from non-Indigenous agencies outside of Western Canada, and should be tested in additional Indigenous and non-Indigenous child welfare agencies across Canada and in other countries (e.g., United States, Australia).

Discussion and Applications to Practice

In sum, the ECWP scale is a novel instrument that appears to be internally consistent and can be used to assess the ECWP excellence, including what workers considered to be important to their practice, what elements they regularly engaged in with their clients, and their perceptions of their practice. The conceptual model identified multiple constructs and items that were relevant, with items for the ECWP selected by a team with various child welfare practice and research expertise. Future studies should consider conducting focus groups with current child welfare workers and team leaders to validate or improve item selection. This measure should be tested with a larger sample of child welfare workers. Further, the ECWP scale should be tested with other measures related to child welfare practice, including the SECWT (Ellett, 2009) and the PSI (O’Hare et al., 1998). Additionally, future research may include utilizing a concept mapping approach, which allows for the brainstorming of ideas and the mapping of complex concepts related to underlying constructs (Hanson et al., 2013; Lebel et al., 2011; Trochim, 1989) or a Delphi method (Busschers et al., 2016; Hsu & Sandford, 2007) to reach consensus about elements essential to practice excellence.

A notable strength of the ECWP is the inclusion of scales that ask about the importance of practice elements, the delivery of practice elements, and worker perception of their practice. These three elements have the potential to inform how agencies assess practice and determine areas for future practice approach development. Understanding what workers believe to be important about practice can help tailor worker training programs aimed at improving practice, whereas having an assessment of what workers are doing with clients can help inform any gaps in practice. Further, the inclusion of these three subscales can enable agency leadership and workers to identify elements of practice where the reported level of importance does not align with delivery. For instance, a worker may report that they believe it is important to include the client's culture in practice, but if the service guidelines or tools do not include a space for cultural inclusions, workers may report lower levels of delivery in this area. This would provide evidence that there should be an organizational change to address the issue.

Additionally, a mismatch in importance and delivery may be related to individual worker resources, such as time limitations, access to resources, job stress, or burnout. Future research should examine differences between the importance and delivery scales, and then examine if there are any individual, team, or organizational level factors associated with any discrepancies. For example, the ECWP could be used in conjunction with a measure of organizational environment, which would include scales related to burnout, agency resources, training, caseload, peer support, etc. Used in conjunction, this would enable us to examine how external organizational and individual-level factors shape what worker see as important in their practice, the elements they regularly deliver, and how they perceive their practice. Thus, providing valuable information to ensure child welfare practice excellence. This initial measure shows promise in identifying what workers see as important ECWP, and how those elements are delivered.

Footnotes

Acknowledgments

The study design,data collection,analysis,interpretation,and manuscript writing were completed independently of the study funders. We would like to acknowledge that this work took place on the traditional and ancestral lands of many Indigenous communities,and to thank the three partnering agencies that invited and hosted us in their land as part of this project. We thank the workers who took time from their busy schedules to participate. Finally,we thank all the research assistants from the CSSCF for their work on this project,including Gurpreet Kaur,Navreen Bilkhu,Tracy Janzen,and Audra Davis. This study received approval from the University of British Columbia—Okanagan ethics board and from the three partnering agency Boards,and participants completed consent forms.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research,authorship,and/or publication of this article: This work was supported by the Social Sciences and Humanities Research Council (SSHRC) Insight Grant Program (F16-04723) and conducted by the Canadian Foundation for Innovation (nfrastructure Operating Grant - 27160)-funded Centre for the Study of Services to Children and Families.

ORCID iD

Sarah Dow-Fleisner

References

Aarons

G. A.

Hurlburt

Horwitz

S. M.

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health, 38(1), 4‐23. https://doi.org/10.1007/s10488-010-0327-7

An Act respecting First Nations, Inuit, and Métis children, youth, and families, S.C., C-92 (2019). https://laws-lois.justice.gc.ca/eng/annualstatutes/2019_24/ .

Barth

R. P.

Lee

B. R.

Lindsey

M. A.

Collins

K. S.

Strieder

Chorpita

B. F.

Sparks

J. A.

(2012). Evidence-based practice at a crossroads: The timely emergence of common elements and common factors. Research on Social Work Practice, 22(1), 108‐119. https://doi.org/10.1177%2F1049731511408440

Becker

S. J.

Midoun

M. M.

Zeithaml

V. A.

Clark

M. A.

Spirito

(2016). Dimensions of treatment quality most valued by adolescent substance users and their caregivers. Professional Psychology: Research and Practice, 47(2), 120‐129. https://doi.org/10.1037/pro0000066

Blackstock

Trocmé

(2005). Community-based child welfare for aboriginal children: Supporting resilience through structural change. Social Policy Journal of New Zealand, 24(12), 12‐33.

Boateng

G. O.

Neilands

T. B.

Frongillo

E. A.

Melgar-Quiñonez

H. R.

Young

S. L.

(2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 1‐18. https://doi.org/https://doi.og/10.3389/fpubh.2018.00149

Busschers

Boendermaker

Dinkgreve

(2016). Validation and operationalization of intensive family case management: A delphi study. Child and Adolescent Social Work Journal, 33(1), 69‐78. https://doi.org/10.1007/s10560-015-0403-7

Carnochan

McBeath

Chuang

Austin

M. J.

(2019). Perspectives of public and nonprofit managers on communications in human services contracting. Public Performance & Management Review, 42(3), 657‐684. https://doi.org/10.1080/15309576.2018.1495085

Chapman

M. V.

Gibbons

C. B.

Barth

R. P.

McRae

J. S.

& NSCAW Research Group. (2003). Parental views of in-home services: What predicts satisfaction with child welfare workers? Child Welfare, 82(5), 571‐596. https://www.jstor.org/stable/45390142

10.

Child, Family, and Community Service Act, R.S.B.C., C-46 (1996). https://www.bclaws.gov.bc.ca/civix/document/id/complete/statreg/00_96046_01 .

11.

Cohen

(1988). Set correlation and contingency tables. Applied Psychological Measurement, 12(4), 425‐434. https://doi.org/10.1177/014662168801200410

12.

Damashek

Bard

Hecht

(2012). Provider cultural competency, client satisfaction, and engagement in home-based programs to treat child abuse and neglect. Child Maltreatment, 17(1), 56‐66. https://doi.org/10.1177/1077559511423570

13.

Damiani-Taraba

Dumbrill

Gladstone

Koster

Leslie

Charles

(2017). The evolving relationship between casework skills, engagement, and positive case outcomes in child protection: A structural equation model. Children and Youth Services Review, 79, 456‐462. https://doi.org/10.1016/j.childyouth.2017.05.033

14.

D’Aunno

L. E.

Boel-Studt

Landsman

M. J.

(2014). Evidence-based elements of child welfare in-home services. Journal of Family Strengths, 14(1), 1‐44. http://digitalcommons.library.tmc.edu/jfs/vol14/iss1/3

15.

Davis

T. S.

Reno

Guada

Swenson

Peck

Saunders-Adams

Haas-Gehres

(2019). Social worker integrated care competencies scale (SWICCS): Assessing social worker clinical competencies for health care settings. Social Work in Health Care, 58(1), 75‐92. https://doi.org/10.1080/00981389.2018.1547346

16.

DeSilva

Aggarwal

N. K.

Lewis-Fernández

(2018). The DSM-5 cultural formulation interview: Bridging barriers toward a clinically integrated cultural assessment in psychiatry. Psychiatric Annals, 48(3), 154‐159. https://doi.org/10.3928/00485713-20180214-01

17.

Dettlaff

A. J.

Graham

J. C.

Holzman

Baumann

D. J.

Fluke

J. D.

(2015). Development of an instrument to understand the child protective services decision-making process, with a focus on placement decisions. Child Abuse & Neglect, 49, 24‐34. https://doi.org/10.1016/j.chiabu.2015.04.007

18.

Drake

(1996). Consumer and worker perceptions of key child welfare competencies. Children and Youth Services Review, 18(3), 261‐279. https://doi.org/10.1016/0190-7409(96)00004-7

19.

Ellett

A. J.

(2009). Intentions to remain employed in child welfare: The role of human caring, self-efficacy beliefs, and professional organizational culture. Children and Youth Services Review, 31(1), 78‐88. https://doi.org/10.1016/j.childyouth.2008.07.002

20.

Forrester

Westlake

Killian

Antonopolou

McCann

Thurnham

Hutchison

(2019). What is the relationship between worker skills and outcomes for families in child and family social work? The British Journal of Social Work, 49(8), 2148‐2167. https://doi.org/10.1093/bjsw/bcy126

21.

Fuller

T. L.

Paceley

M. S.

Schreiber

J. C.

(2015). Differential response family assessments: Listening to what parents say about service helpfulness. Child Abuse & Neglect, 39, 7‐17. https://doi.org/10.1016/j.chiabu.2014.05.010

22.

Gadermann

A. M.

Guhn

Zumbo

B. D.

(2012). Estimating ordinal reliability for likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research, and Evaluation, 17(1), 1‐13. https://doi.org/10.7275/n560-j767

23.

Gladstone

Dumbrill

Leslie

Koster

Young

Ismaila

(2012). Looking at engagement and outcome from the perspectives of child protection workers and parents. Children and Youth Services Review, 34(1), 112‐118. https://doi.org/10.1016/j.childyouth.2011.09.003

24.

Gladstone

Dumbrill

Leslie

Koster

Young

Ismaila

(2014). Understanding worker-parent engagement in child protection casework. Children and Youth Services Review, 44, 56‐64. https://doi.org/10.1016/j.childyouth.2014.06.002

25.

Haight

Waubanascum

Glesener

Marsalis

(2018). A scoping study of indigenous child welfare: The long emergency and preparations for the next seven generations. Children and Youth Services Review, 93, 397‐410. https://doi.org/10.1016/j.childyouth.2018.08.016

26.

Hanson

H. M.

Schiller

Winters

Sims-Gould

Clarke

Curran

Ashe

M. C.

(2013). Concept mapping applied to the intersection between older Adults’ outdoor walking and the built and social environments. Preventive Medicine, 57(6), 785‐791. https://doi.org/10.1016/j.ypmed.2013.08.023

27.

Harris

Poertner

Joe

(2000). The parents with children in foster care satisfaction scale. Administration in Social Work, 24(2), 15‐27. https://doi.org/10.1300/J147v24n02_02

28.

Horwitz

Marshall

(2015). Family engagement in child protection social work. Journal of Family Social Work, 18(4), 288‐301. https://doi.org/10.1080/10522158.2015.1079584

29.

Hsu

C. C.

Sandford

B. A.

(2007). The delphi technique: Making sense of consensus. Practical Assessment, Research, and Evaluation, 12(10), 1‐8. https://doi.org/10.7275/pdz9-th90

30.

Indian Act, R.S.C., c. I-5 (1951; 1985). https://laws-lois.justice.gc.ca/eng/acts/i-5/ .

31.

Jensen

I. B.

Studsrød

Ellingsen

I. T.

(2020). Child protection social workers’ constructions of children and childhood: An integrative review. Child & Family Social Work, 25(S1), 83‐96. https://doi.org/10.1111/cfs.12720

32.

Kapp

S. A.

Vela

R. H.

(2004). The parent satisfaction with foster care services scale. Child Welfare, 83(3), 263‐287. http://www.jstor.org/stable/45400604

33.

Keys

(2009). Determining the skills for child protection practice: Emerging from the quagmire!. Child Abuse Review, 18(5), 316‐332. https://doi.org/10.1002/car.1089

34.

Krentzman

A. R.

Townsend

A. L.

(2008). Review of multidisciplinary measures of cultural competence for use in social work education. Journal of Social Work Education, 44(2), 7‐32. https://doi.org/10.5175/JSWE.2008.200600003

35.

Lebel

Cantinotti

Pampalon

Thériault

Smith

L. A.

Hamelin

A. M.

(2011). Concept mapping of diet and physical activity: Uncovering local stakeholders perception in the Quebec city region. Social Science & Medicine, 72(3), 439‐445. https://doi.org/10.1016/j.socscimed.2010.09.013

36.

Leung

Cheung

(2013). Factor analyzing the “ASK” cultural competency self-assessment scale for child protective services. Children and Youth Services Review, 35(12), 1993‐2002. https://doi.org/10.1016/j.childyouth.2013.09.014

37.

(2013). Little's test of missing completely at random. The Stata Journal, 13(4), 795‐809. https://doi.org/10.1177/1536867X1301300407

38.

Lundahl

McDonald

Vanderloo

(2020). Service users’ perspectives of child welfare services: A systematic review using the practice model as a guide. Journal of Public Child Welfare, 14(2), 174‐191. https://doi.org/10.1080/15548732.2018.1548406

39.

Macgowan

M. J.

Dillon

F. R.

Spadola

C. E.

(2018). Factor structure of a standards-based inventory of competencies in social work with groups. Journal of Evidence-Informed Social Work, 15(4), 403‐419. https://doi.org/10.1080/23761407.2018.1464996

40.

Marsh

J. C.

Angell

Andrews

C. M.

Curry

(2012). Client-provider relationship and treatment outcome: A systematic review of substance abuse, child welfare, and mental health services research. Journal of the Society for Social Work and Research, 3(4), 233‐267. https://doi.org/10.5243/jsswr.2012.15

41.

Mason

(2012). Social work the ‘art of relationship’: Parents’ perspectives on an intensive family support project. Child & Family Social Work, 17(3), 368‐377. https://doi.org/10.1111/j.1365-2206.2011.00791.x

42.

Ministry of Children and Family Development [MCFD] (2021). Delegated Aboriginal Agencies in BC. https://www2.gov.bc.ca/gov/content/family-social-supports/data-monitoring-quality-assurance/reporting-monitoring/accountability/delegated-aboriginal-agencies.

43.

Mundy

C. L.

Neufeld

A. N.

Wells

S. J.

(2016). A culturally relevant measure of client satisfaction in child welfare services. Children and Youth Services Review, 70, 177‐189. https://doi.org/10.1016/j.childyouth.2016.09.011

44.

O’Hare

Collins

Walsh

(1998). Validation of the practice skills inventory with experienced clinical social workers. Research on Social Work Practice, 8(5), 552‐563. https://doi.org/10.1177%2F104973159800800504

45.

Olin

S. S.

Kutash

Pollock

Burns

B. J.

Kuppinger

Hoagwood

K. E.

(2014). Developing quality indicators for family support in community team-based mental health care. Administration and Policy in Mental Health, 41(1), 7‐20. https://doi.org/10.1007/s10488-013-0501-9

46.

Palmer

Maiter

Manji

(2006). Effective intervention in child protective services: Learning from parents. Children and Youth Services Review, 28(7), 812‐824. https://doi.org/10.1016/j.childyouth.2005.08.010

47.

Potter

C. C.

Leake

Longworth-Reed

Altschul

Rienks

(2016). Measuring organizational health in child welfare agencies. Children and Youth Services Review, 61(C), 31‐39. https://doi.org/10.1016/j.childyouth.2015.11.002

48.

Poulin

Young

(1997). Development of a helping relationship inventory for social work practice. Research on Social Work Practice, 7(4), 463‐489. https://doi.org/10.1177%2F104973159700700403

49.

Public Health Association of Canada [PHAC] (2019). Provincial and Territorial Child Protection Legislation Policy 2018. Ministry of Health. https://www.canada.ca/content/dam/phac-aspc/documents/services/publications/health-risks-safety/provincial-territorial-child-protection-legislation-policy-2018/64-03-18-2245_Child-Protection_EN-FINAL02.pdf .

50.

Rowe

J. M.

Kim

Chung

Hessenauer

(2020). Development and validation of a field evaluation instrument to assess student competency. Journal of Social Work Education, 56(1), 142‐154. https://doi.org/10.1080/10437797.2018.1548988

51.

RStudio Team (2020). RStudio: Integrated development for R. RStudio, PBC. http://www.rstudio.com/.

52.

Schreiber

J. C.

Fuller

Paceley

M. S.

(2013). Engagement in child protective services: Parent perceptions of worker skills. Children and Youth Services Review, 35(4), 707‐715. https://doi.org/10.1016/j.childyouth.2013.01.018

53.

Sinai-Glazer

(2020). The essentials of the helping relationship between social workers and clients. Social Work, 65(3), 245‐256. https://doi.org/10.1093/sw/swaa028

54.

StataCorp (2017). Stata statistical software: Release 15. StataCorp LLC.

55.

Stene

K. L.

Dow-Fleisner

S. J.

Ermacora

Agathen

Falconnier

Stager

Wells

S. J.

(2020). Measuring the quality of care in kinship foster care placements. Children and Youth Services Review, 116, 105‐136. https://doi.org/10.1016/j.childyouth.2020.105136

56.

Tilbury

Ramsay

(2018). A systematic scoping review of parental satisfaction with child protection services. Evaluation and Program Planning, 66, 141‐146. https://doi.org/10.1016/j.evalprogplan.2017.10.010

57.

Toros

DiNitto

D. M.

Tiko

(2018). Family engagement in the child welfare system: A scoping review. Children and Youth Services Review, 88, 598‐607. https://doi.org/10.1016/j.childyouth.2018.03.011

58.

Trochim

W. M. K.

(1989). An Introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12(1), 1‐16. https://doi.org/10.1016/0149-7189(89)90016-5

59.

Trotter

(2002). Worker skill and client outcome in child protection. Child Abuse Review, 11(1), 38‐50. https://doi.org/10.1002/car.719

60.

Trotter

(2004). Helping abused children and their families. Allen & Unwin.

61.

Wampold

B. E.

(2015). How important are the common factors in psychotherapy? An update. World Psychiatry, 14(3), 270‐277. https://doi.org/10.1002/wps.20238

62.

Watkins

K. E.

Marsick

V. J.

(1997). Dimensions of the learning organization questionnaire [Survey]. Partners for the Learning Organization.

63.

Wells

S. J.

Girling

Evanoff

Neufeld

McQuaid

Sholmi

Cross

Healey

Vergara

(2015, July). Innovations in service delivery in the British Columbia Ministry of Children and Family Development (MCFD) Final Report. Centre for the Study of Services to Children and Families, University of British Columbia.

64.

Zumbo

B. D.

Gadermann

A. M.

Zeisser

(2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6(1), 21‐29. https://doi.org/10.22237/jmasm/1177992180