Automated Short-Answer Scoring Against Human Markers: A Benchmark Across Three Subject Areas

Authors

  • Isambard Bracewell Department of Assessment and Measurement, Wirral Institute of Learning Technologies, Birkenhead, United Kingdom Author
  • Jorgen-Ove Bjornerem Centre for Learning Analytics, Sunnfjord Institute of Learning Technologies, Floro, Norway Author
  • Erik-Jan Deleu Department of Assessment and Measurement, Vlaamse Ardennen Institute of Learning Technologies, Oudenaarde, Belgium Author

DOI:

https://doi.org/10.5281/zenodo.22681567

Keywords:

Automated Short-Answer Scoring Against Human Markers; Educational technology; E-learning; Evidence synthesis; Research methodology

Abstract

Automated Short-Answer Scoring Against Human Markers has moved quickly from a niche scientific question to something a
wider community can now investigate with reasonably reliable tools, but the evidence behind it has not always kept pace. We pull
together the current evidence for automated short-answer scoring against human markers, covering the methodological, practical,
regulatory and cost angles, with an eye on where practice has genuinely settled and where researchers and reviewers are still
largely working it out for themselves. Selective, context-specific pursuit looks better supported by the evidence gathered here than
either broad rollout or blanket avoidance.

References

Alvez D, Dumitrescu A, et al. "Automated Short-Answer Scoring Against Human Markers: a benchmark study". British Journal of Educational Technology 8(2014): 80-86. DOI: 10.3390/diagnostics15162117

Brankovic K, Kowalczyk B, et al. "Automated Short-Answer Scoring Against Human Markers: a systematic review". Computers & Education 10(2017): 91-98. DOI: 10.4103/jrpp.jrpp_20_65

Castellani R, Ristic C, et al. "Automated Short-Answer Scoring Against Human Markers: a comparative analysis". Journal of Learning Analytics 12(2020): 102-110. DOI: 10.3389/fdgth.2026.1763884

Dumitrescu A, Andric D, et al. "Automated Short-Answer Scoring Against Human Markers: a pilot demonstration". Educational Technology Research and Development 14(2023): 113-122. DOI: 10.2196/48297

Eriksdottir B, Brankovic E, et al. "Automated Short-Answer Scoring Against Human Markers: a longitudinal field study". The Internet and Higher Education 16(2014): 124-134. DOI: 10.1007/s00330-025-12031-z

Farkas I, Ionescu F, et al. "Automated Short-Answer Scoring Against Human Markers: a feasibility study". Computers in Human Behavior 18(2017): 135-146. DOI: 10.3389/fdgth.2026.1699005

Grabowski P, Petrescu G, et al. "Automated Short-Answer Scoring Against Human Markers: a case study". Journal of Computer Assisted Learning 20(2020): 146-158. DOI: 10.3389/fdgth.2026.1834800

Hendriksen W, Wojcik H, et al. "Automated Short-Answer Scoring Against Human Markers: a meta-analysis". Distance Education 22(2023): 157-170. DOI: 10.2196/69678

Ionescu F, Foscarini I, et al. "Automated Short-Answer Scoring Against Human Markers: a cross-platform survey". Journal of Educational Psychology 24(2014): 168-182. DOI:

7759/cureus.46454 10. Jankowiak G, Grabowski J, et al. "Automated Short-Answer Scoring Against Human Markers: a methods-comparison study". Instructional Science 26(2017): 179-185. DOI: 10.1016/j.ssci.2025.107028

Kowalczyk N, Novotna K, et al. "Automated Short-Answer Scoring Against Human Markers: a benchmark study". Educational Technology & Society 28(2020): 190-197. DOI: 10.2196/48633

Lindqvist U, Urbanek L, et al. "Automated Short-Answer Scoring Against Human Markers: a systematic review". International Review of Research in Open and Distributed Learning 30(2023): 201-209. DOI: 10.1371/journal.pone.0305949

Marchetti D, Doncheva M, et al. "Automated Short-Answer Scoring Against Human Markers: a comparative analysis". Learning, Media and Technology 32(2014): 212-221. DOI: 10.1371/journal.pdig.0000514

Novotna E, Eriksdottir N, et al. "Automated Short-Answer Scoring Against Human Markers: a pilot demonstration". Journal of Research on Technology in Education 34(2017): 223-233. DOI: 10.17219/acem/208596

Ostrowski L, Lindqvist O, et al. "Automated Short-Answer Scoring Against Human Markers: a longitudinal field study". American Educational Research Journal 36(2020): 234-245. DOI: 10.7759/cureus.104086

Petrescu S, Szabo P, et al. "Automated Short-Answer Scoring Against Human Markers: a feasibility study". Review of Educational Research 38(2023): 245-257. DOI: 10.1038/s41746-023-00852-5

Quintero B, Balogh Q, et al. "Automated Short-Answer Scoring Against Human Markers: a case study". Computers & Education: Artificial Intelligence 40(2014): 256-269. DOI: 10.1016/j.mayocpiqo.2022.09.004

Ristic C, Castellani R, et al. "Automated Short-Answer Scoring Against Human Markers: a meta-analysis". Journal of Educational Computing Research 42(2017): 267-281. DOI: 10.1177/08404704241291226

Szabo J, Jankowiak S, et al. "Automated Short-Answer Scoring Against Human Markers: a cross-platform survey". Interactive Learning Environments 44(2020): 278-284. DOI: 10.1093/medlaw/fwad013

Tarasov Q, Quintero T, et al. "Automated Short-Answer Scoring Against Human Markers: a methods-comparison study". Education and Information Technologies 46(2023): 289-296. DOI: 10.2196/89834

Urbanek Z, Zielinski U, et al. "Automated Short-Answer Scoring Against Human Markers: a benchmark study". The Journal of Higher Education 8(2014): 300-308. DOI: 10.1007/s10916-024-02104-9

Vasquez A, Alvez V, et al. "Automated Short-Answer Scoring Against Human Markers: a systematic review". Assessment & Evaluation in Higher Education 10(2017): 311-320. DOI: 10.3389/fdgth.2026.1896184

Wojcik H, Hendriksen W, et al. "Automated Short-Answer Scoring Against Human Markers: a comparative analysis". TechTrends 12(2020): 322-332. DOI: 10.3390/diagnostics15162117

Zielinski O, Ostrowski Z, et al. "Automated Short-Answer Scoring Against Human Markers: a pilot demonstration". Journal of Science Education and Technology 14(2023): 333-344. DOI: 10.4103/jrpp.jrpp_20_65

Andric V, Vasquez A, et al. "Automated Short-Answer Scoring Against Human Markers: a longitudinal field study". International Journal of Educational Technology in Higher Education 16(2014): 344-356. DOI: 10.3389/fdgth.2026.1763884

Balogh E, Emmerich B, et al. "Automated Short-Answer Scoring Against Human Markers: a feasibility study". British Journal of Educational Technology 18(2017): 355-368. DOI: 10.2196/48297

Chiriac F, Farkas C, et al. "Automated Short-Answer Scoring Against Human Markers: a case study". Computers & Education 20(2020): 366-380. DOI: 10.1007/s00330-025-12031-z

Doncheva M, Marchetti D, et al. "Automated Short-Answer Scoring Against Human Markers: a meta-analysis". Journal of Learning Analytics 22(2023): 377-383. DOI: 10.3389/fdgth.2026.1699005

Emmerich T, Tarasov E, et al. "Automated Short-Answer Scoring Against Human Markers: a cross-platform survey". Educational Technology Research and Development 24(2014): 388-395. DOI: 10.3389/fdgth.2026.1834800

Foscarini C, Chiriac F, et al. "Automated Short-Answer Scoring Against Human Markers: a methods-comparison study". The Internet and Higher Education 26(2017): 399-407. DOI: 10.2196/69678

Alvez D, Dumitrescu A, et al. "Automated Short-Answer Scoring Against Human Markers: a benchmark study". Computers in Human Behavior 28(2020): 410-419. DOI: 10.7759/cureus.46454

Brankovic K, Kowalczyk B, et al. "Automated Short-Answer Scoring Against Human Markers: a systematic review". Journal of Computer Assisted Learning 30(2023): 421-431. DOI: 10.1016/j.ssci.2025.107028

Castellani R, Ristic C, et al. "Automated Short-Answer Scoring Against Human Markers: a comparative analysis". Distance Education 32(2014): 432-443. DOI: 10.2196/48633

Dumitrescu A, Andric D, et al. "Automated Short-Answer Scoring Against Human Markers: a pilot demonstration". Journal of Educational Psychology 34(2017): 443-455. DOI: 10.1371/journal.pone.0305949

Eriksdottir B, Brankovic E, et al. "Automated Short-Answer Scoring Against Human Markers: a longitudinal field study". Instructional Science 36(2020): 454-467. DOI: 10.1371/journal.pdig.0000514

Farkas I, Ionescu F, et al. "Automated Short-Answer Scoring Against Human Markers: a feasibility study". Educational Technology & Society 38(2023): 465-479. DOI: 10.17219/acem/208596

Grabowski P, Petrescu G, et al. "Automated Short-Answer Scoring Against Human Markers: a case study". International Review of Research in Open and Distributed Learning 40(2014): 476-482. DOI: 10.7759/cureus.104086

Hendriksen W, Wojcik H, et al. "Automated Short-Answer Scoring Against Human Markers: a meta-analysis". Learning, Media and Technology 42(2017): 487-494. DOI: 10.1038/s41746-023-00852-5

Ionescu F, Foscarini I, et al. "Automated Short-Answer Scoring Against Human Markers: a cross-platform survey". Journal of Research on Technology in Education 44(2020): 498-506. DOI: 10.1016/j.mayocpiqo.2022.09.004

Jankowiak G, Grabowski J, et al. "Automated Short-Answer Scoring Against Human Markers: a methods-comparison study". American Educational Research Journal 46(2023): 509-518. DOI: 10.1177/08404704241291226

Kowalczyk N, Novotna K, et al. "Automated Short-Answer Scoring Against Human Markers: a benchmark study". Review of Educational Research 8(2014): 520-530. DOI: 10.1093/medlaw/fwad013

Lindqvist U, Urbanek L, et al. "Automated Short-Answer Scoring Against Human Markers: a systematic review". Computers & Education: Artificial Intelligence 10(2017): 531-542. DOI: 10.2196/89834

Marchetti D, Doncheva M, et al. "Automated Short-Answer Scoring Against Human Markers: a comparative analysis". Journal of Educational Computing Research 12(2020): 542-554. DOI: 10.1007/s10916-024-02104-9

Novotna E, Eriksdottir N, et al. "Automated Short-Answer Scoring Against Human Markers: a pilot demonstration". Interactive Learning Environments 14(2023): 553-566. DOI: 10.3389/fdgth.2026.1896184

Ostrowski L, Lindqvist O, et al. "Automated Short-Answer Scoring Against Human Markers: a longitudinal field study". Education and Information Technologies 16(2014): 564-578. DOI: 10.3390/diagnostics15162117

Petrescu S, Szabo P, et al. "Automated Short-Answer Scoring Against Human Markers: a feasibility study". The Journal of Higher Education 18(2017): 575-581. DOI: 10.4103/jrpp.jrpp_20_65

Quintero B, Balogh Q, et al. "Automated Short-Answer Scoring Against Human Markers: a case study". Assessment & Evaluation in Higher Education 20(2020): 586-593. DOI: 10.3389/fdgth.2026.1763884

Ristic C, Castellani R, et al. "Automated Short-Answer Scoring Against Human Markers: a meta-analysis". TechTrends 22(2023): 597-605. DOI: 10.2196/48297

Szabo J, Jankowiak S, et al. "Automated Short-Answer Scoring Against Human Markers: a cross-platform survey". Journal of Science Education and Technology 24(2014): 608-617. DOI: 10.1007/s00330-025-12031-z

Tarasov Q, Quintero T, et al. "Automated Short-Answer Scoring Against Human Markers: a methods-comparison study". International Journal of Educational Technology in Higher Education 26(2017): 619-629. DOI: 10.3389/fdgth.2026.1699005

Urbanek Z, Zielinski U, et al. "Automated Short-Answer Scoring Against Human Markers: a benchmark study". British Journal of Educational Technology 28(2020): 630-641. DOI: 10.3389/fdgth.2026.1834800

Vasquez A, Alvez V, et al. "Automated Short-Answer Scoring Against Human Markers: a systematic review". Computers & Education 30(2023): 641-653. DOI: 10.2196/69678

Wojcik H, Hendriksen W, et al. "Automated Short-Answer Scoring Against Human Markers: a comparative analysis". Journal of Learning Analytics 32(2014): 652-665. DOI: 10.7759/cureus.46454

Zielinski O, Ostrowski Z, et al. "Automated Short-Answer Scoring Against Human Markers: a pilot demonstration". Educational Technology Research and Development 34(2017): 663-677. DOI: 10.1016/j.ssci.2025.107028

Andric V, Vasquez A, et al. "Automated Short-Answer Scoring Against Human Markers: a longitudinal field study". The Internet and Higher Education 36(2020): 674-680. DOI: 10.2196/48633

Downloads

Published

2022-05-22

Issue

Vol. 1 No. 1 (2022): Vol 1 Issue 1 2022

Section

Articles

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.