Sage Journals: Discover world-class research

Abstract

Purpose

We sought to determine the correlation between the Numeric Rating Scale (NRS) and Critical-Care Pain Observation Tool (CPOT) to determine whether clinical factors modified the relationship between NRS and CPOT assessments.

Materials and Methods

We included nonventilated adults admitted to the MICU or SICU who could self-report pain and had at least 3 paired NRS and CPOT assessments. We performed Spearman correlation to assess overall correlation and performed proportional odds logistic regression to evaluate whether the relationship between NRS and CPOT assessments was modified by clinical factors.

Results

Nursing staff performed NRS and CPOT assessments every 4 h in 1302 patients, leading to 61,142 matched assessments. We found that the NRS and CPOT have a Spearman correlation coefficient of 0.56 and an intraclass correlation coefficient of 0.32 in intensive care unit patients. Factors that modified the relationship between the NRS and CPOT included the presence of delirium (P < .001) and lower mean daily Richmond Agitation Sedation Scale (<0.001).

Conclusions

The correlation coefficient between the NRS and the CPOT was found to be 0.56. The presence of delirium, decreased level of arousal, modified the relationship between the NRS and CPOT. Self-reported and behavioral pain assessments cannot be used interchangeably in critically ill adults.

Keywords

sedation analgesia CPOT NRS pain

Introduction

Pain is defined as an unpleasant sensory and emotional experience associated with, or resembling that associated with, actual or potential tissue damage by the International Association for the Study of Pain.¹ Pain is personal, multidimensional, and, therefore, most commonly described by patient self-report. Impaired communication, altered mental status, mechanical ventilation, and other characteristics of critically ill patients make a uniform means of pain assessment difficult.²

A comprehensive review of the psychometric properties of behavioral pain assessment tools was conducted. The development, reliability, validity, feasibility, and clinical utility of 9 behavioral pain assessment tools for noncommunicative critically ill patient and 4 tools for other noncommunicative patients were evaluated. The Behavioral Pain Scale (BPS), BPS-Nonintubated (BPS-NI), and the Critical-Care Pain Observation Tool (CPOT) were found to have the strongest psychometric properties having been validated in various countries and in multiple languages.³ The Clinical Practice Guidelines for the Prevention and Management of Pain, Agitation/Sedation, Delirium, Immobility, and Sleep Disruption in Adult Patients in Adult Patients in the intensive care unit (ICU) (PADIS) suggest using a Numeric Rating Scale (NRS) either verbally or visually in critically ill patients able to self-report pain.² The PADIS Guidelines recommend the use of the CPOT or the BPS in intubated and BPS-NI critically ill adults unable to self-report pain.²

While a patient-reported NRS score of 1 to 3 indicates mild pain, 4 to 6 moderate pain, and 7 to 10 severe pain, similar categories have not been established with the nurse-performed objective CPOT assessment, which ranges from 0 to 8.⁴ A CPOT score of 3 or greater indicates pain is likely present, and higher scores indicate increased intensity of pain behaviors. Despite these differences, many clinicians equate NRS and CPOT scores. These values have not been shown to be interchangeable, and the correlation between NRS and CPOT values have been assessed in several studies, commonly during procedures, and reported to be low even during rest.^5,6 Furthermore, discrepant scores may affect treatment decisions by clinicians.

The purpose of this study was to determine the correlation coefficient between the NRS and the CPOT in a pragmatic setting by clinical staff prospectively assessing a large cohort of nonventilated critically ill patients who could self-report pain. We additionally aimed to evaluate whether the relationship between NRS and CPOT assessments was modified by relevant baseline and clinical course factors.

Materials and Methods

Study Setting

This prospective cohort study was conducted in medical and surgical ICU patients at Vanderbilt University Medical Center between June 2016 and June 2017. The study (#170120) was approved by the Institutional Review Board at Vanderbilt University with a waiver of informed consent. Procedures were followed in accordance with the ethical standards of the responsible committee on human experimentation (institutional or regional) and with the Helsinki Declaration of 1975.

Pain, agitation, and delirium are managed in these ICUs according to the Society of Critical Care Medicine's Pain, Agitation, Delirium, Immobility, and Sleep Guidelines (PADIS).² Sedation and analgesia are titrated using the Richmond Agitation Sedation Scale (RASS) and the NRS or CPOT, respectively. Delirium is evaluated using the Confusion Method for the ICU (CAM-ICU).⁷ We collected baseline patient demographic information and daily hospital course data, including the nursing pain assessments, mechanical ventilation, and medication exposure, from the electronic health record.

Study Sites and Patient Population

We included adults (≥18-year-old) admitted to the study medical or surgical ICU that were nonventilated and could communicate, self-report pain, and had at least 3 paired NRS and CPOT assessments. To be qualified as paired, the CPOT and NRS had to be conducted within 60 min of one another. Exclusion criteria were as follows: RASS −4 or −5 for the entire ICU stay, limb or facial injuries, quadriplegia, neuromuscular blocker recipients, traumatic brain injury, stroke affecting limb mobility, requirement of physical restraints, or established cognitive impairment.

Study Assessments

Nursing staff performed NRS and CPOT every 4 h for research purposes. The NRS is pain score where the patient is asked to rate their pain on a scale from 0 (no pain) to 10 (the worse pain imaginable).⁸ A patient is considered to be in moderate pain if the NRS is greater than or equal to 4. The CPOT includes 4 different behavioral categories: (1) facial expression, (2) body movements, (3) muscle tension, and (4) compliance with the ventilator or vocalization in nonintubated patients.⁶ A score between 0 and 2 was given for each of the different components for a total of 0 to 8. Scores of 3 or greater are indicative of pain. Nurses were performing additional assessments that are not part of standard of care. The nurses would not ordinarily collect CPOT scores on patients that are able to self-report at the study center. Clinical care and treatment of pain were under the discretion of the treatment team.

Study Outcomes

The primary outcome was the correlation coefficient between the NRS and the CPOT score. We additionally examined whether the relationship between NRS and CPOT assessments was modified by clinical factors.

Covariates

Baseline covariates included age, body mass index (BMI), sex, and pain medications prior to admission (oxycodone, hydrocodone/acetaminophen, gabapentin, acetaminophen, ibuprofen, pregabalin, tramadol, and oxycodone/acetaminophen). Hospital-acquired covariates included mean daily RASS, days of delirium, days of mechanical ventilation, UHC expected mortality, admission unit (MICU or SICU), and intravenous or oral analgesia/sedation received while in the ICU (propofol, midazolam, lorazepam, morphine, hydromorphone, fentanyl, oxycodone, oxycodone/acetaminophen, hydrocodone/acetaminophen). Methadone, fentanyl patches, and buprenorphine were not included as they are rarely used in the ICUs in this study.

Statistical Analysis

Continuous variables were summarized as median with interquartile range (IQR). Categorical variables were summarized as a number with a percentage (n, %). To test the correlation between the paired NRS and CPOT assessments, we performed Spearman correlation. A perfect correlation is defined as 1. A strong correlation is defined as 0.7 to 0.9. A moderate correlation is defined as 0.4 to 0.6. A weak correlation is defined as 0.1 to 0.3.⁹ The Spearman correlation was weighted by the number of pairs per patient. We performed proportional odds logistic regression assessing the association of NRS, baseline and time-varying clinical factors, and NRS interaction with clinical factors with CPOT to evaluate whether the relationship between NRS and CPOT assessments was modified by clinical factors. All 2-way interactions were included. The regression model was fit on the CPOT/NRS level. Because each patient had multiple assessments included in the data, we used cluster sandwich covariance estimator with the patient ID as a cluster in order to adjust the variance in our model to account for the repeated measurements. We allowed age, BMI, days on the ventilator, and time from admission to CPOT to have a quadratic relationship with CPOT. Multivariable analysis is reported with odds ratios and 95% confidence intervals. A P value of <.05 was considered significant. All analysis was performed by using statistical software R version 3.5.2 (R Development Core Team; https://www.r.project.org).

Results

Between June 2016 and June 2017, a total of 354,684 NRS records and 152,750 CPOT records were collected (Figure 1). We included 1302 patients with 61,142 paired assessments. The median NRS score was 0 (IQR, 0-4), while the median CPOT score was 0 (IQR, 0-1). Of the paired assessments, 78% were taken at the same time resulting in a median time within paired assessments of 0 h (IQR, 0-0 h). The mean time within paired assessments was 10.4 min (standard deviation = 21.3 min). Patient characteristics at baseline are presented in Table 1. The majority of patients were older males admitted to the MICU and nonventilated.

Figure 1.

Patient flow diagram.

Table 1.

Patient Characteristics at Baseline.^a

	All patients
Characteristic	(n = 1320)
Age (years)	60 [46-69]
Male	730 (56%)
BMI	27 [23-33]
Unit
MICU	849 (65%)
SICU	453 (35%)
UHC expected mortality	0.0547 (0.0125-0.2056)
ICU length of stay (days)	4 [2.1-7.6]
Duration of mechanical ventilation (days)	1 [0-3]
Analgesics prior to admission
Oxycodone	62 (5%)
Hydrocodone and acetaminophen	24 (2%)
Gabapentin	54 (4%)
Acetaminophen	147 (11%)
Ibuprofen	25 (2%)
Pregabalin	7 (1%)
Tramadol	8 (1%)
Oxycodone and acetaminophen	6 (0%)
Ketorlac	0 (0%)
Oxymorphone	0 (0%)
Delirium	623 (48%)
Analgesics and sedation in the ICU
Propofol	401 (31%)
Midazolam	141 (11%)
Lorazepam	213 (16%)
Dexmedetomidine	8 (1%)
Morphine	126 (10%)
Hydromorphone	509 (39%)
Fentanyl	345 (26%)
Oxycodone	519 (40%)
Oxycodone and acetaminophen	47 (4%)
Hydrocodone and acetaminophen	66 (5%)

Abbreviations: BMI, body mass index; ICU, intensive care unit.

Data are given as number (percentage) or median [interquartile range].

Score Correlation

We found that the NRS and CPOT have a Spearman correlation coefficient of 0.56 and an intraclass correlation coefficient of 0.32 in ICU patients (Figure 2). Among the total 61,142 pairs of CPOT and NRS, 47,840 pairs were measured at same time (Spearman correlation coefficient of 0.68), and 13,302 pairs were not measured at same time (Spearman correlation coefficient of 0.20). The frequency distribution of CPOT by NRS is shown in Supplemental Table 1. Additionally, we display the correlation between NRS and CPOT assessments by ICU type, delirium status, age, mechanical ventilation, and other groups (Supplemental Figure 1 and Supplemental Table 2). The correlation coefficient was higher in the MICU (0.55) as compared to the SICU (0.51) although this difference is most likely not clinically relevant. The correlation coefficient was also higher if delirium was present (0.71) as compared to absent (0.5). Lower RASS scores, a higher number of ventilator days, and older age improved the correlation.

Figure 2.

The size of the circle indicates relative number of observations, except point (0,0), which has 5.

Clinical Factors That Affected the Relationship Between the NRS and CPOT

Outcomes of the logistic regression assessing for effects of clinical factors on the relationship between the NRS and CPOT are displayed in Table 2. Factors that modified the relationship between the NRS and CPOT included the presence of delirium (P < .001) and lower mean daily RASS (<0.001; Figure 3). Medications that modified the relationship between the NRS and CPOT included home use of pregabalin (P = .017) or oxycodone/acetaminophen (P = .004) and in-hospital use of propofol (P = .015), IV hydromorphone (P < .001), IV fentanyl (P < .001), oxycodone (P < .001), oxycodone/acetaminophen (P = .001), and hydrocodone/acetaminophen (P < .001; Figure 4). Factors that did not significantly modify the relationship between NRS and CPOT included age, duration of mechanical ventilation, medical versus surgical ICU, sex, BMI, UHC expected mortality, and severity of illness.

Figure 3.

Nonmedication risk factors for modification of the relationship between the CPOT and Numeric Rating Scale (NRS).

Figure 4.

Medication risk factors for modification of the relationship between the CPOT and Numeric Rating Scale (NRS).

Table 2.

Association of Patient Factors and Modification of the Relationship Between the Numerical Rating Scale and Critical Care Pain Observation Tool.^a

Independent variable	OR	95% CI	P value
Age	0.97	0.89-1.05	.285
BMI	0.98	0.90-1.06	.369
Duration of mechanical ventilation	0.75	0.65-0.87	.196
Unit (MICU:SICU)	0.93	0.81-1.03	.322
UHC expected mortality	0.80	0.68-0.94	.464
Sex (Male:Female)	0.99	0.87-1.12	.378
Delirium daily (Yes:No)	0.69	0.57-0.71	<.001
Mean RASS daily	0.89	0.85-0.93	<.001
Medications prior to admission (Yes:No)
Oxycodone	1.09	0.85-1.41	<.001
Hydrocodone and acetaminophen	0.85	0.67-1.10	<.001
Gabapentin	1.28	0.99-1.64	.379
Acetaminophen	0.93	0.77-1.14	.139
Ibuprofen	0.94	0.49-1.79	.675
Pregabalin	0.70	0.30-1.64	.017
Tramadol	1.14	0.58-2.27	.977
Oxycodone and acetaminophen	0.69	0.33-1.45	.004
Analgesics and sedation in the ICU (Yes: No)
Propofol	1.27	1.06-1.49	.015
Midazolam	1.4	1.05-1.88	.166
Lorazepam	0.75	0.59-0.96	.88
Morphine	0.57	0.43-0.75	.065
Hydromorphone	0.53	0.47-0.62	<.001
Fentanyl	0.50	0.42-0.58	<.001
Oxycodone	0.68	0.59-0.78	<.001
Oxycodone and acetaminophen	0.63	0.39-1	.001
Hydrocodone and acetaminophen	0.98	0.68-1.39	<.001

Abbreviations: CPOT, Critical-Care Pain Observation Tool; RASS, Richmond Agitation Sedation Scale; BMI, body mass index; ICU, intensive care unit; OR, odds ratio; CI, confidence interval.

We fit proportional odds logistic regression model to assess the association between Numeric Rating Scale and CPOT with adjustment of baseline demographics and time-varying clinical covariates. All 2-way interactions were included. Because each patient had multiple assessments included in the data, we used cluster sandwich covariance estimator with the patient ID as a cluster in order to adjust the variance in our model to account for these repeated measurements. We allowed the age, BMI, days on ventilator, and time from until admission to CPOT to have a quadratic relationship with CPOT.

Discussion

We found that the NRS and CPOT have a correlation coefficient of 0.56 and an intraclass correlation coefficient of 0.32 in ICU patients. Given that pain fluctuates over time, assessments measured at the same time had much higher correlation than those matched within 60 min. We found a higher correlation coefficient in the MICU, patients with delirium, lower RASS scores, higher number of ventilator days, and in older patients. We found that the presence of delirium and decreased arousal (ie, lower mean daily RASS scores) modified the relationship between the NRS and CPOT. Additionally, propofol, IV hydromorphone, IV fentanyl, pregabalin, oxycodone, oxycodone with acetaminophen, and hydrocodone with acetaminophen were medications that modified the relationship between the NRS and CPOT. Other factors such as age, sex, BMI, duration of mechanical ventilation, ICU type, and severity of illness, however, did not modify the relationship.

Our study adds to a limited body of literature regarding the correlation between CPOT assessments and self-reported measures. A study examining a total of 55 critically ill adults had both self-reporting via the Faces Pain Thermometer (FPT) and CPOT performed during a nociceptive procedure (turning) and a non-nociceptive procedure (taking noninvasive blood pressure). A high Pearson correlation coefficient of 0.71 (P ≤ .05) was found between the FPT and CPOT during turning.⁵ Higher correlation was found in other studies.¹⁰ The CPOT was initially validated in 105 cardiac surgery patients. Following extubation, the patients’ self-reported pain intensity scores poorly correlated with the CPOT scores, with Spearman correlations of 0.49, 0.59, and 0.40.¹¹ A prospective, observational study of 115 patients, including 67 nondelirious patients and 48 delirious patients, was conducted to assess the correlation of the NRS or Wong-Baker Faces pain scale (self-reported) to the CPOT or BPS. In nondelirious patients, self-reported and BPSs were poorly correlated (Spearman correlation coefficient 0.28). Additionally, there was a poor correlation between self-reported and BPSs in delirious patients (Spearman correlation coefficient 0.23).¹² Our study was by far the largest evaluation of the correlation between patient-reported and objective assessments for pain in hospitalized patients. Overall, we found a medium correlation between the NRS and CPOT in critically ill patients, indicating these 2 assessment techniques are not interchangeable and should be treated differently when determining clinical management. Notably, the authors also found higher correlation in the MICU, patients with delirium, lower RASS scores, higher number of ventilator days, and in older patients.

An observational study of 101 patients for 3 days following ICU admission was conducted to compare the CPOT and BPS with the visual analog scale (VAS) and to identify the best combination of scales to evaluate pain in patients unable to self-report. Both the CPOT and BPS were significantly correlated with VAS (P < .0001). The sensitivity of the CPOT and BPS was 76.5% and 62.7%, respectively. The combination of the CPOT and BPS increased the sensitivity to 80.4%.¹³

In addition to assessing for the correlation coefficient between NRS and CPOT, we sought to examine factors that modified the relationship between the NRS and CPOT. Given the nature of pain and the use of these measures to guide therapy, including medications, it is imperative to identify potential patients with discordance between perceived pain scores and observational assessments. We found that the presence of delirium, decreased levels of arousal, and a number of commonly administered analgesics and sedatives modified the relationship between the NRS and CPOT. These factors can all indicate impaired sensorium or affect the sensorium, and the uniquity of these factors across ICU patient populations may contribute to the overall correlation coefficient between NRS and CPOT. Patients with reduced sensorium (ie, lower RASS, hypoactive delirium), in general, may not have been experiencing significant pain and would thus have less potential difference in perceived and reported pain (NRS) versus observed pain behaviors (CPOT) at the lower end of the scales. Additionally, it is important to note that self-reporting of pain in the presence of delirium is not recommended as it may lead to unreliable information. Some important clinical factors such as age, sex, BMI, duration of mechanical ventilation, medical versus surgical, and severity of illness, however, did not modify the relationship.

Our study has several strengths. This is the largest evaluation to determine the correlation coefficient between the NRS and CPOT, with over 1300 patients and 61,000 paired assessments. We included a large number of medical and surgical patients, and assessments were performed by bedside nurses, both of which increase generalizability. Medication and care regimens were reflective of clinical practice. We were able to evaluate important demographic and hospital course aspects to determine risk factors that modified the relationship between the CPOT and NRS. Both Spearman correlation and intraclass correlation measure the association between NRS and CPOT. Spearman correlation focuses on the association between the change of NRS and the change of CPOT. Intraclass correlation, meanwhile, focuses on the degree of concordance between NRS and CPOT. Since the NRS and CPOT were on different scales, it is reasonable to see the lower intraclass correlation. The intraclass correlation was presented to show that even when the CPOT is moderately correlated with NRS, they do not have good agreement because of the different measurement scale.¹⁴

Weaknesses of our study included patients from a single center and retrospective data collection, with the exception of pain assessments, from the electronic health record which is limited by missing or incorrect data entry. Further, we were unable to adjust for admission diagnosis to the ICU, dosing and compliance with home analgesics, and dosing with in-hospital administered analgesia in evaluating correlations. We treated assessments as paired if they occurred within 60 min of each other, and pain could have changed or been managed within that period of time. Our median time within assessments, however, was 0 h. Additionally, timing of pain assessments relative to activity was not prescribed by the study or documented, and as such could have been performed at rest or during activity which may impact scores. Future studies need to be conducted in additional patient populations and examine doses of analgesics and sedatives on correlation coefficient between the NRS and CPOT.

Conclusions

In critically ill patients, correlation coefficient between the NRS and the CPOT was found to be of 0.56 and an intraclass correlation coefficient of 0.32 in a pragmatic setting by clinical staff assessing a large cohort of medical and surgical ICU patients. We found a higher correlation coefficient in the MICU, patients with delirium, lower RASS scores, higher number of ventilator days, and in older patients. The presence of delirium and decreased arousal modified the relationship between the NRS and CPOT. Intravenous propofol, hydromorphone, fentanyl, pregabalin, oxycodone, oxycodone with acetaminophen, and hydrocodone with acetaminophen administration during hospitalization also modified the relationship between the CPOT and NRS. The NRS and CPOT cannot be used interchangeably in critically ill adults. A reliable self-report (in the absence of confusion or delirium) should be obtained whenever possible. Behavioral scales such as the CPOT should be used as an alternative measure of pain.²

Supplemental Material

sj-docx-1-jic-10.1177_08850666231187336 - Supplemental material for Correlation of the Critical Care Pain Observation Tool and Numeric Rating Scale in Intensive Care Unit Patients

Supplemental material, sj-docx-1-jic-10.1177_08850666231187336 for Correlation of the Critical Care Pain Observation Tool and Numeric Rating Scale in Intensive Care Unit Patients by Joanna L Stollings, Kelli A Rumbaugh, Li Wang, Christina J Hayhurst, E Wesley Ely and Christopher G Hughes in Journal of Intensive Care Medicine

Footnotes

Author Contributions

Study concept and design: Joanna L Stollings,Kelli A Rumbaugh,Christina J Hayhurst,and Christopher G Hughes. Acquisition of data: Joanna L Stollings and Kelli A Rumbaugh. Analysis and interpretation of data: Joanna L Stollings,Kelli A Rumbaugh,Li Wang,Christina J Hayhurst,E Wesley Ely and Christopher G Hughes. Drafting of the manuscript: Joanna L Stollings and Kelli A Rumbaugh. Critical revision of the manuscript for important intellectual content: Joanna L Stollings,Kelli A Rumbaugh,Li Wang,Christina J Hayhurst,E Wesley Ely,and Christopher G Hughes. Statistical analysis: Li Wang,Study supervision: Joanna L Stollings,Kelli A Rumbaugh,Li Wang,Christina J Hayhurst,E Wesley Ely,and Christopher G Hughes. All had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Li Wang conducted and is responsible for the data analysis.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research,authorship,and/or publication of this article: Dr Ely received honoraria for nonpromotional speaking from Hospira Inc. Dr Ely and Hughes have received research grant from Dr Franz Kohler Chemie GMBH. Dr Hughes is a consultant for Sedana Medical. Keywords: Pain,Critical Care Pain Observation Tool,Numeric Rating Scale. Dr Stollings,Dr Rumbaugh,Ms. Wang,and Dr Hayhurst have nothing to disclose.

Funding

The author(s) disclosed receipt of the following financial support for the research,authorship,and/or publication of this article: Financial support for the study was provided by the Vanderbilt Institute for Clinical and Translational Research (UL1 TR000445 and UL1TR002243 from NCATS/NIH). E.W.E. is supported by the Veterans Affairs Tennessee Valley Geriatric Research,Education and Clinical Center (Nashville,Tennessee,USA). E.W.E. is supported by the VA Clinical Science Research and Development Service (Washington,District of Columbia,USA) and the National Institutes of Health AG027472,AG035117,HL111111 (Bethesda,Maryland,USA). C.G.H. is supported by National Institutes of Health HL111111 and AG061161 (Bethesda,Maryland,USA).

ORCID iD

Joanna L Stollings

Supplemental Material

Supplemental material for this article is available online.

References

Raja

Carr

Cohen

, et al. The revised international association for the study of pain definition of pain: concepts, challenges, and compromises. Pain. 2020;161(9):1976-1982.

Devlin

Skrobik

Gelinas

, et al. Clinical practice guidelines for the prevention and management of pain, agitation/sedation, delirium, immobility, and sleep disruption in adult patients in the ICU. Crit Care Med. 2018;46(9):e825-e873.

Gelinas

Joffe

Szumita

, et al. A psychometric analysis update of behavioral pain assessment tools for noncommunicative, critically ill adults. AACN Adv Crit Care. 2019;30(4):365-387.

Serlin

Mendoza

Nakamura

Edwards

Cleeland

. When is cancer pain mild, moderate or severe–grading pain severity by its interference with function. Pain. 1995;61(2):277-284.

Gelinas

Johnston

. Pain assessment in the critically ill ventilated adult: validation of the critical-care pain observation tool and physiologic indicators. Clin J Pain. 2007;23(6):497-505.

Gelinas

Fillion

Puntillo

Viens

Fortier

. Validation of the critical-care pain observation tool in adult patients. Am J Crit Care. 2006;15(4):420-427.

Ely

Margolin

Francis

, et al. Evaluation of delirium in critically ill patients: validation of the confusion assessment method for the intensive care unit (CAM-ICU). Crit Care Med. 2001;29(7):1370-1379.

Jensen

Miller

Fisher

. Assessment of pain during medical procedures: a comparison of three scales. Clin J Pain. 1998;14(4):343-349.

Akoglu

. User's guide to correlation coefficients. Turk J Emerg Med. 2018;18(3):91-93.

10.

Chen

Yang

, et al. Validation of a Chinese version critical-care pain observation tool in nonintubated and intubated critically ill patients: two cross-sectional studies. J Clin Nurs. 2019;28(15–16):2824-2832.

11.

Gelinas

Harel

Fillion

Puntillo

Johnston

. Sensitivity and specificity of the critical-care pain observation tool for the detection of pain in intubated adults after cardiac surgery. J Pain Symptom Manage. 2009;37(1):58-67.

12.

Bouajram

Sebat

Love

Louie

Wilson

Duby

. Comparison of self-reported and behavioral pain assessment tools in critically ill patients. J Intensive Care Med. 2020;35(5):453-460.

13.

Severgnini

Pelosi

Contino

Serafinelli

Novario

Chiaranda

. Accuracy of critical care pain observation tool and behavioral pain scale to assess pain in critically ill conscious and unconscious patients: prospective, observational study. J Intensive Care. 2016;4:68.

14.

Kottner

Gajewski

Streiner

. Guidelines for reporting reliability and agreement studies (GRRAS). Int J Nurs Stud. 2011;48(6):659-660.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

1.80 MB

0.00 MB