This study examined performer, rater, occasion, and sequence as sources of variability in music performance assessment. Generalizability theory served as the study's basis. Performers were 8 high school wind instrumentalists who had recently performed a solo. The author audio-recorded performers playing excerpts from their solo three times, establishing an occasion variable. To establish a rater variable, 10 certified adjudicators were asked to rate the performances from 0 (poor) to 100 (excellent). Raters were randomly assigned to one of five performance sequences, thus nesting raters within a sequence variable. Two G (generalizability) studies established that occasion and sequence produced virtually no measurement error. Raters were a strong source of error. D (decision) studies established the one-rater, one-occasion scenario as unreliable. In scenarios using the generalizability coefficient as a criterion, 5 hypothetical raters were necessary to reach the .80 benchmark. Using the dependability index, 17 hypothetical raters were necessary to reach .80.
Bergee, M.J. (1995). Primary and higher-order factors in a scale assessing concert band performance. Bulletin of the Council for Research in Music Education, 126, 1-14.
2.
Bergee, M.J. (2003). Faculty interjudge reliability of music performance evaluation. Journal of Research in Music Education, 51, 137-148.
3.
Bergee, M.J. (2006). Validation of a model of extramusical influences on solo and small-ensemble festival ratings. Journal of Research in Music Education, 54, 244-256.
4.
Bergee, M.J., & McWhirter, J.L. (2005). Selected influences on solo and small-ensemble festival ratings: Replication and extension. Journal of Research in Music Education, 53, 177-190.
5.
Bergee, M.J., & Platt, M.C. (2003). Influence of selected variables on solo and small-ensemble festival ratings. Journal of Research in Music Education, 51, 342-353.
6.
Bergee, M.J., & Westfall, C.R. (2005). Stability of a model explaining selected extramusical influences on solo and small-ensemble festival ratings . Journal of Research in Music Education, 53, 253-271.
7.
Brennan, R.L. (2001). Generalizability theory. New York/Berlin: Springer-Verlag .
8.
Bruine de Bruin, W. (2005). Save the last dance for me: Unwanted serial position effects in jury evaluations. Acta Psychologica, 118, 245-260.
9.
Burnsed, V., Hinkle, D., & King, S. (1985). Performance evaluation reliability at selected concert festivals. Journal of Band Research, 21(1), 22-29.
10.
Carmines, E.G., & Zeller, R.A. (1979). Reliability and validity assessment (Sage Quantitative Applications in the Social Sciences Series No. 17). Beverly Hills, CA: Sage.
11.
Duerksen, G. (1972). Some effects of expectation on evaluation of recorded musical performances. Journal of Research in Music Education , 20, 268-272.
12.
Fiske, H.E., Jr. (1975). Judge-group differences in the rating of secondary school trumpet performers. Journal of Research in Music Education , 23, 186-196.
13.
Fiske, H.E., Jr. (1977). Relationship of selected factors in trumpet performance adjudication reliability. Journal of Research in Music Education , 25, 256-263.
14.
Fiske, H.E., Jr. (1983). Judging musical performance: Method or madness?Update: Applications of Research in Music Education, 1(3), 7-10.
15.
Flôres, R.G., & Ginsburgh, V.A. (1996). The Queen Elisabeth musical competition: How fair is the final ranking? The Statistician , 45, 97-104.
16.
Hare, R.Y. (1960). Problems and considerations in adjudicating music contests. The Instrumentalist, 14, 106-107.
17.
Hoyt, C. (1941). Test reliability estimated by analysis of variance . Psychometrika, 6, 153-160.
18.
James, L.R., Demaree, R.G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology , 69, 85-98.
19.
James, L.R., Demaree, R.G., & Wolf, G. (1993). rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.
20.
Kieffer, K.M. (1999). Why generalizability theory is essential and classical test theory is often inadequate. In B. Thompson (Ed.), Advances in social science methodology (Vol. 5, pp. 149-170). Greenwich, CT: JAI.
21.
Mish, F. C. et al. (Ed.). (2003). Merriam-Webster's collegiate dictionary (11th ed.). Springfield, MA: Author .
22.
Mills, J. (1987). Assessment of solo musical performance: A preliminary study. Bulletin of the Council for Research in Music Education , 91, 119-125.
Radocy, R.E., & Boyle, J.D. (1987). Measurement and evaluation of musical experiences . New York: Schirmer.
25.
Sagin, D.P. (1983). The development and validation of a university band performance rating scale. Journal of Band Research, 18(2), 1-11.
26.
Schmidt, F.L., & Hunter, J.E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology, 74, 368-370.
27.
Smith, B. (2004). Five judges' evaluation of audiotaped string performance . Bulletin of the Council of Research in Music Education , 160, 61-69.
28.
Spearman, C.E. (1904). The proof of measurement of association between two things. American Journal of Psychology, 15, 72-101.
29.
Stanley, M., Brooker, R., & Gilbert, R. (2002). Examiner perceptions of using criteria in music performance assessment. Research Studies in Music Education , 18, 43-52.
30.
Thompson, S., & Williamon, A. (2003). Evaluating evaluation: Musical performance assessments as a research tool. Music Perception, 21, 21-41.
31.
Thompson, W.F., Diamond, C.T., & Balkwill, L.L. (1998). The adjudication of six performances of a Chopin etude: A study of expert knowledge. Psychology of Music, 26, 154-174.
32.
Vasil, T. (1973). The effects of systematically varying selected factors on music performance adjudication. Unpublished doctoral dissertation , University of Connecticut.
33.
Wapnick, J., & Ekholm, E. (1997). Expert consensus in solo voice performance evaluation . Journal of Voice, 11, 429-436.
34.
Wapnick, J., Flowers, P., Alegant, M., & Jasinskas, L. (1993). Consistency in piano performance evaluation. Journal of Research in Music Education, 41, 282-292.
35.
Wapnick, J., Ryan, C., Lacaille, N., & Darrow, A.A. (2004). Effects of selected variables on musicians' ratings of high-level piano performances. International Journal of Music Education, 22, 7-20.
36.
Wilson, V.E. (1977). Objectivity and effect of order of appearance in judging of synchronized swimming meets. Perceptual and Motor Skills, 44, 295-298.
