Automated scoring based on Latent Semantic Analysis (LSA) has been successfully used to score essays and constrained short answer responses. Scoring tests that capture open-ended, short answer responses poses some challenges for machine learning approaches. We used LSA techniques to score short answer responses to the Consequences Test, a measure of creativity and divergent thinking that encourages a wide range of potential responses. Analyses demonstrated that the LSA scores were highly correlated with conventional Consequence Test scores, reaching a correlation of .94 with human raters and were moderately correlated with performance criteria. This approach to scoring short answer constructed responses solves many practical problems including the time for humans to rate open-ended responses and the difficulty in achieving reliable scoring.
AttaliY. (2013). Validity and reliability of automated essay scoring. In ShermisM. D.BursteinJ. (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 181-199). New York, NY: Routledge.
3.
BergamaschiS.PoL. (2015, December). Comparing LDA and LSA topic models for content-based movie recommendation systems. Paper presented at the International Conference on Web Information Systems and Technologies, Lisbon, Portugal.
4.
BlumD.HollingH. (2017). Spearman’s law of diminishing returns. A meta-analysis. Intelligence, 65, 60-66.
5.
ChristensenP. R.MerrifieldP. R.GuilfordJ. P. (1953). Consequences Form A-1. Beverly Hills, CA: Sheridan Supply.
6.
CvitanicT.LeeB.SongH. I.FuK.RosenD. (2016, October-November). LDA v. LSA: A comparison of two computational text analysis tools for the functional categorization of patents. Paper presented at the International Conference on Case Based Reasoning, Atlanta, GA.
7.
GuilfordJ. P.GuilfordJ. S. (1980). Consequences: Manual of instructions and operations. Orange, CA: Sheridan Psychological Services.
KerstingN. B.SherinB. L.StiglerJ. W. (2014). Automated scoring of teachers’ open-ended responses to video prompts: Bringing the Classroom-Video-Analysis assessment to scale. Educational and Psychological Measurement, 74, 905-926.
10.
LandauerT. K.DumaisS. (1997). A solution to Plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104, 211-240.
11.
LandauerT. K.FoltzP. W.LahamD. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259-284.
12.
LandauerT. K.LahamD.FoltzP. W. (2000). The intelligent essay assessor. IEEE Intelligent Systems, 15(5), 27-31.
13.
LandauerT. K.LahamR. D.FoltzP. W. (2003). Automated scoring and annotation of essays with the intelligent essay assessor. Assessment in Education, 10, 295-308.
14.
LandauerT. K.McNamaraD. S.DennisS.KintschW. (2007). Handbook of latent semantic analysis. Mahwah, NJ: Lawrence Erlbaum.
15.
LaVoieN.CiancioloA.MartinJ. (2015, April). Automated assessment of diagnostic skill. Poster presented at the CGEA CGSA COSR conference, Columbus, OH.
16.
LaVoieN.StreeterL.LochbaumK.WroblewskiD.BoyceL.A.KrupnickC.PsotkaJ. (2010). Automating expertise in collaborative learning environments. Journal of Asynchronous Learning Networks, 14(4), 97-119.
17.
LiuO. L.BrewC.BlackmoreJ.GerardL.MadhokJ.LinnM. C. (2014). Automated scoring of constructed-response science items: Prospects and obstacles. Educational Measurement: Issues and Practice, 33(2), 19-28.
18.
MartinD. I.BerryM. W. (2010). Latent Semantic Indexing. In BatesM. J.MaackM.N. (Eds.), Encyclopedia of Library and Information Sciences (pp. 3195-3204). New York, NY: Taylor & Francis.
19.
McArdleJ. J.Ferrer-CajaE.HamagamiF.WoodcockR. W. (2002). Comparative longitudinal structural analyses of the growth and decline of multiple intellectual abilities over the life span. Developmental Psychology, 38, 115-142.
20.
MumfordM. D.MarksM. A.ConnellyM. S.ZaccaroS. J.JohnsonJ. F. (1998). Domain-based scoring of divergent-thinking tests: Validation evidence in an occupational sample. Creativity Research Journal, 11, 151-163.
21.
PutkaD. J. (Ed.). (2009). Initial development and validation of assessments for predicting disenrollment of four-year scholarship recipients from the Reserve Officer Training Corps (Study Report 2009-06). Arlington, VA: U.S. Army Research Institute for the Behavioral Sciences.
22.
ShermisM. D. (2014). State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration. Assessing Writing, 20, 53-76.
23.
ShermisM. D.BursteinJ.HigginsD.ZechnerK. (2010). Automated essay scoring: Writing assessment and instruction. In BakerE.McGawB.PetersenN. S. (Eds.), International encyclopedia of education (3rd ed., pp. 75-80). Oxford, England: Elsevier.
24.
StemlerS. E.SternbergR. J.GrigorenkoE. L.JarvinL.SharpesK. (2009). Using the theory of successful intelligence as a framework for developing assessments in AP physics. Contemporary Educational Psychology, 34, 195-209.
25.
SternbergR. J. (2015). Successful intelligence: A model for testing intelligence beyond IQ tests. European Journal of Education and Psychology, 8, 76-84.
