Respond to the letter on ‘Interobserver reliability of the ankle brachial index,toe–brachial index and distal pulse palpation in patients with diabetes: a methodological issue’

Abstract

We respond to the letter of Nadery and Shahsavari regarding our paper entitled ‘Interobserver reliability of the ankle brachial index, toe -brachial index and distal pulse palpation in patients with diabetes. In this letter, we concluded that despite some limitations, the kappa coefficient is an informative measure of agreement in most circumstances that we can use in this type of clinical research.

Dear Editor,

We would like to respond to the letter of Nadery and Shahsavari regarding our paper entitled ‘Interobserver reliability of the ankle brachial index, toe–brachial index and distal pulse palpation in patients with diabetes’. ¹

The authors claim that no value of kappa can be universally accepted as a sign of a good agreement; however, kappa coefficient is an informative measure of agreement in most circumstances that is widely used in clinical research.^2–7

The authors argue that kappa value to assess the agreement of a qualitative variable has two weaknesses. First, it depends on the prevalence in each category. This is true but especially when raters report a very high prevalence of the condition of interest, because some of the overlap in their diagnoses may reflect their common knowledge about the disease in the population being rated. This should be considered ‘true’ agreement, but it is attributed to chance agreement (i.e. kappa = 0). ² This situation did not happen in our study; for instance, regarding to the prevalence of palpation of distal pulses, 13 (31%), 24 (57%) and 17 (40 %) posterior tibial pulses were absent for the very experienced, medium-experienced and inexperienced clinicians, respectively. Notwithstanding, we included in our study several limitations. First, the characteristics of our study population, in a specialised diabetic foot unit, are probably different than in a basic health-care unit where the prevalence of peripheral artery disease (PAD) is lower.

However, the authors said that kappa value is also dependent on the number of categories but we used only two categories: dichotomous variables (Yes/No), it is not an ordinal qualitative variable or ranked variables such us low, moderate or severe. For this situation, it is not necessary to apply weighted kappa. Finally, authors suggest applying Fleiss kappa because we have more than two observers. Nevertheless, when we performed the kappa index between two clinicians, we observed similar results than with three clinicians. For instance, Table 1 (experienced–medium experienced clinician), Table 2 (medium experienced–inexperienced clinician) and Table 3 (experienced–inexperienced) depict the contingency tables and the kappa index of the palpation of tibial posterior pulses among two clinicians with different levels of experience. We observed a moderate agreement among clinicians with similar experience (K₁ = 0.503, p < 0.001, and K₂ = 0.490, p = 0.001) and a low agreement among experienced and inexperienced clinicians (K₃ = 0.384, p = 0.011). When we performed the kappa index among these three clinicians, we observed a moderate agreement (K = 0.45, p < 0.001). We think that this result is similar than the kappa index with two clinicians, and for this situation, we concluded that the palpation of distal pulses are not highly reproducible and reliable between clinicians with different levels of experience under routine conditions. In some research, such as Meyer et al., the kappa index was used to evaluate the agreement between four pathologists (more than two observers) for identification of specific histologic findings. ⁶ However, our paper was a brief report and we prefer to use the kappa index with three clinicians to summarise the results. We agree that by performing the kappa index among two clinicians, we could conclude that the variability in the palpation of tibial posterior pulses increases as the degree of experience diminishes (K₁ = 0.503, K₂ = 0.490 and K₃ = 0.384). These results can be comparable to ankle–brachial index and toe–brachial index.

Table 1.

Contingency tables of the palpation of tibial posterior artery among experienced and medium experienced clinician. ⁸

Posterior tibial artery palpation, K1 = 0.503, p < 0.001 (moderate a )		Clinician 2 (medium experienced)		Total
		No	Yes
Clinician 1 (Experienced)	No	13	0	13
	Yes	11	18	29
	Total	24	18	42

K₁: kappa coefficient between very experienced and medium-experienced clinicians.

a

Strength of agreement (Landis and Koch criteria).

Table 2.

Contingency tables of the palpation of tibial posterior artery among medium-experienced and inexperienced clinicians. ⁸

Posterior tibial artery palpation K2 = 0.490, p = 0.001 (moderate a )		Clinician 3 (inexperienced)		Total
		No	Yes
Clinician 2 (medium experienced)	No	15	9	24
	Yes	2	16	18
	Total	17	25	42

K₂: kappa coefficient between medium-experienced and inexperienced clinicians.

a

Strength of agreement (Landis and Koch criteria).

Table 3.

Contingency tables of the palpation of tibial posterior artery among medium-experienced and inexperienced clinician. ⁸

Posterior tibial artery palpation, K3 = 0.384, p = 0.011 (low a )		Clinician 3 (inexperienced)		Total
		No	Yes
Clinician 1 (experienced)	No	9	4	13
	Yes	8	21	29
	Total	17	25	42

K₂: kappa coefficient between medium-experienced and inexperienced clinicians.

K₃: kappa coefficient between very experienced and inexperienced clinicians.

a

Strength of agreement (Landis and Koch criteria).

In this letter, we concluded that despite some limitations, the kappa coefficient is an informative measure of agreement in most circumstances that we can use in this type of clinical research.