Hide
Раскрыть

Address:
Armyanskii ln., 4-2, Moscow, 101000, Russia

Е-mail:
hsejournal.psychology@gmail.com

Ekaterina Valueva1,2, Anastasia Panfilova1, Antonina Rafikova1,3
  • 1 Institute of Psychology, Russian Academy of Sciences, 13 build. 1 Yaroslavskaya Str., Moscow, 129366, Russian Federation
  • 2 Moscow State University of Psychology & Education, 29 Sretenka Str., Moscow, 127051, Russian Federation
  • 3 State Academic University for the Humanities, 26 Maronovskiy Pereulok, Moscow, 119049, Russian Federation

Automatic Scoring of Verbal Divergent Thinking Tests: From Lexical Databases to Large Language Models

2024. Vol. 21. No. 1. P. 202–225 [issue contents]

The article explores the evolution of methods for automatically assessing verbal divergent thinking tests. Researchers have increasingly focused on the ability to evaluate the originality of respondents' answers by calculating their semantic distance from the stimulus task. From 2009 to 2019, latent semantic analysis became the primary method for assessing semantic distances. Overall, in terms of internal consistency and correlation with expert ratings, its application yielded satisfactory results, maintaining an acceptable balance of quality and effort expended. However, issues emerged (dependence on the corpus used, result instability, systematic distortions related to the length of analyzed responses), prompting researchers to transition to more advanced models of distributional semantics (GloVe, Word2Vec etc.), large language models, and supervised learning. Large language models, especially those fine-tuned on creativity test materials, demonstrated higher effectiveness compared to models assessing semantic distances and approached expert evaluations. In addition to evaluating originality, the article considers works proposing methods for automatic assessment of elaboration, flexibility, associative flow, and divergent semantic integration. References to online platforms that allow for automatic assessments of originality in responses to divergent tests are provided. The issue of interpreting results obtained through large language models is discussed. A drawback of using these models is the lack of understanding of the basis on which judgments of the originality of creative products are made. The perspectives of applying explainable artificial intelligence for evaluating results of verbal and non-verbal tests of creative thinking are being discussed.

Citation: Valueva E., Panfilova A., Rafikova A. (2024) Avtomaticheskaya otsenka testov na verbal'nuyu kreativnost': ot leksicheskikh baz dannykh k bol'shim yazykovym modelyam [Automatic Scoring of Verbal Divergent Thinking Tests: From Lexical Databases to Large Language Models]. Psychology. Journal of Higher School of Economics, vol. 21, no 1, pp. 202-225 (in Russian)
BiBTeX
RIS
 
 
Rambler's Top100 rss