Human Oversight of Automated Decisions: A Position Paper on What Meaningful Review Requires
DOI:
https://doi.org/10.5281/zenodo.22657292Keywords:
Human Oversight of Automated Decisions; Applied machine learning; Evidence synthesis; Production deployment; MLOpsAbstract
Human Oversight of Automated Decisions is an area of active development in applied machine learning, with evidence that has
accumulated unevenly across its component questions. We review the current evidence base for human oversight of automated
decisions, spanning methodological, implementation, governance, and cost dimensions, focusing on where practice has converged
and where it continues to vary, and set out the practical considerations relevant to a team considering its adoption. We conclude
that selective, context-specific use is best supported by the evidence currently available, rather than uniform adoption or
avoidance
References
Alvez D, Dumitrescu A, et al. "Human Oversight of Automated Decisions: a benchmark study". J Mach Learn Res 8(2014): 80-86. DOI: 10.1007/s00330-025-12031-z
Brankovic K, Kowalczyk B, et al. "Human Oversight of Automated Decisions: a systematic evaluation". IEEE Trans Pattern Anal Mach Intell 10(2017): 91-98. DOI: 10.3389/fdgth.2026.1699005
Castellani R, Ristic C, et al. "Human Oversight of Automated Decisions: a empirical analysis". Proc NeurIPS 12(2020): 102-110. DOI: 10.3389/fdgth.2026.1834800
Dumitrescu A, Andric D, et al. "Human Oversight of Automated Decisions: a ablation study". Proc ICML 14(2023): 113-122. DOI: 10.2196/69678
Eriksdottir B, Brankovic E, et al. "Human Oversight of Automated Decisions: a reproducibility study". Proc ICLR 16(2014): 124-134. DOI: 10.7759/cureus.46454
Farkas I, Ionescu F, et al. "Human Oversight of Automated Decisions: a large-scale evaluation". ACM Trans Intell Syst 18(2017): 135-146. DOI: 10.1016/j.ssci.2025.107028
Grabowski P, Petrescu G, et al. "Human Oversight of Automated Decisions: a case study". Data Min Knowl Discov 20(2020): 146-158. DOI: 10.2196/48633
Hendriksen W, Wojcik H, et al. "Human Oversight of Automated Decisions: a survey". Artif Intell 22(2023): 157-170. DOI: 10.1371/journal.pone.0305949
Ionescu F, Foscarini I, et al. "Human Oversight of Automated Decisions: a comparative analysis". Knowl Based Syst 24(2014): 168-182. DOI: 10.1371/journal.pdig.0000514
Jankowiak G, Grabowski J, et al. "Human Oversight of Automated Decisions: a field deployment report". IEEE Trans Neural Netw Learn Syst 26(2017): 179-185. DOI: 10.17219/acem/208596
Kowalczyk N, Novotna K, et al. "Human Oversight of Automated Decisions: a benchmark study". Proc KDD 28(2020): 190-197. DOI: 10.7759/cureus.104086
Lindqvist U, Urbanek L, et al. "Human Oversight of Automated Decisions: a systematic evaluation". Proc AAAI 30(2023): 201-209. DOI: 10.1038/s41746-023-00852-5
Marchetti D, Doncheva M, et al. "Human Oversight of Automated Decisions: a empirical analysis". IEEE Trans Knowl Data Eng 32(2014): 212-221. DOI: 10.1016/j.mayocpiqo.2022.09.004
Novotna E, Eriksdottir N, et al. "Human Oversight of Automated Decisions: a ablation study". J Artif Intell Res 34(2017): 223-233. DOI: 10.1177/08404704241291226
Ostrowski L, Lindqvist O, et al. "Human Oversight of Automated Decisions: a reproducibility study". ACM Comput Surv 36(2020): 234-245. DOI: 10.1093/medlaw/fwad013
Petrescu S, Szabo P, et al. "Human Oversight of Automated Decisions: a large-scale evaluation". Neural Comput Appl 38(2023): 245-257. DOI: 10.2196/89834
Quintero B, Balogh Q, et al. "Human Oversight of Automated Decisions: a case study". Expert Syst Appl 40(2014): 256-269. DOI: 10.1007/s10916-024-02104-9
Ristic C, Castellani R, et al. "Human Oversight of Automated Decisions: a survey". Inf Sci 42(2017): 267-281. DOI: 10.3389/fdgth.2026.1896184
Szabo J, Jankowiak S, et al. "Human Oversight of Automated Decisions: a comparative analysis". Pattern Recognit 44(2020): 278-284. DOI: 10.3390/diagnostics15162117
Tarasov Q, Quintero T, et al. "Human Oversight of Automated Decisions: a field deployment report". IEEE Access 46(2023): 289-296. DOI: 10.4103/jrpp.jrpp_20_65
Urbanek Z, Zielinski U, et al. "Human Oversight of Automated Decisions: a benchmark study". ACM Trans Softw Eng Methodol 8(2014): 300-308. DOI: 10.3389/fdgth.2026.1763884
Vasquez A, Alvez V, et al. "Human Oversight of Automated Decisions: a systematic evaluation". Proc MLSys 10(2017): 311-320. DOI: 10.2196/48297
Wojcik H, Hendriksen W, et al. "Human Oversight of Automated Decisions: a empirical analysis". Proc FAccT 12(2020): 322-332. DOI: 10.1007/s00330-025-12031-z
Zielinski O, Ostrowski Z, et al. "Human Oversight of Automated Decisions: a ablation study". Nat Mach Intell 14(2023): 333-344. DOI: 10.3389/fdgth.2026.1699005
Andric V, Vasquez A, et al. "Human Oversight of Automated Decisions: a reproducibility study". J Mach Learn Res 16(2014): 344-356. DOI: 10.3389/fdgth.2026.1834800
Balogh E, Emmerich B, et al. "Human Oversight of Automated Decisions: a large-scale evaluation". IEEE Trans Pattern Anal Mach Intell 18(2017): 355-368. DOI: 10.2196/69678
Chiriac F, Farkas C, et al. "Human Oversight of Automated Decisions: a case study". Proc NeurIPS 20(2020): 366-380. DOI: 10.7759/cureus.46454
Doncheva M, Marchetti D, et al. "Human Oversight of Automated Decisions: a survey". Proc ICML 22(2023): 377-383. DOI: 10.1016/j.ssci.2025.107028
Emmerich T, Tarasov E, et al. "Human Oversight of Automated Decisions: a comparative analysis". Proc ICLR 24(2014): 388-395. DOI: 10.2196/48633
Foscarini C, Chiriac F, et al. "Human Oversight of Automated Decisions: a field deployment report". ACM Trans Intell Syst 26(2017): 399-407. DOI: 10.1371/journal.pone.0305949
Alvez D, Dumitrescu A, et al. "Human Oversight of Automated Decisions: a benchmark study". Data Min Knowl Discov 28(2020): 410-419. DOI: 10.1371/journal.pdig.0000514
Brankovic K, Kowalczyk B, et al. "Human Oversight of Automated Decisions: a systematic evaluation". Artif Intell 30(2023): 421-431. DOI: 10.17219/acem/208596
Castellani R, Ristic C, et al. "Human Oversight of Automated Decisions: a empirical analysis". Knowl Based Syst 32(2014): 432-443. DOI: 10.7759/cureus.104086
Dumitrescu A, Andric D, et al. "Human Oversight of Automated Decisions: a ablation study". IEEE Trans Neural Netw Learn Syst 34(2017): 443-455. DOI: 10.1038/s41746-023-00852-5
Eriksdottir B, Brankovic E, et al. "Human Oversight of Automated Decisions: a reproducibility study". Proc KDD 36(2020): 454-467. DOI: 10.1016/j.mayocpiqo.2022.09.004
Farkas I, Ionescu F, et al. "Human Oversight of Automated Decisions: a large-scale evaluation". Proc AAAI 38(2023): 465-479. DOI: 10.1177/08404704241291226
Grabowski P, Petrescu G, et al. "Human Oversight of Automated Decisions: a case study". IEEE Trans Knowl Data Eng 40(2014): 476-482. DOI: 10.1093/medlaw/fwad013
Hendriksen W, Wojcik H, et al. "Human Oversight of Automated Decisions: a survey". J Artif Intell Res 42(2017): 487-494. DOI: 10.2196/89834
Ionescu F, Foscarini I, et al. "Human Oversight of Automated Decisions: a comparative analysis". ACM Comput Surv 44(2020): 498-506. DOI: 10.1007/s10916-024-02104-9
Jankowiak G, Grabowski J, et al. "Human Oversight of Automated Decisions: a field deployment report". Neural Comput Appl 46(2023): 509-518. DOI: 10.3389/fdgth.2026.1896184
Kowalczyk N, Novotna K, et al. "Human Oversight of Automated Decisions: a benchmark study". Expert Syst Appl 8(2014): 520-530. DOI: 10.3390/diagnostics15162117
Lindqvist U, Urbanek L, et al. "Human Oversight of Automated Decisions: a systematic evaluation". Inf Sci 10(2017): 531-542. DOI: 10.4103/jrpp.jrpp_20_65
Marchetti D, Doncheva M, et al. "Human Oversight of Automated Decisions: a empirical analysis". Pattern Recognit 12(2020): 542-554. DOI: 10.3389/fdgth.2026.1763884
Novotna E, Eriksdottir N, et al. "Human Oversight of Automated Decisions: a ablation study". IEEE Access 14(2023): 553-566. DOI: 10.2196/48297
Ostrowski L, Lindqvist O, et al. "Human Oversight of Automated Decisions: a reproducibility study". ACM Trans Softw Eng Methodol 16(2014): 564-578. DOI: 10.1007/s00330-025-12031-z
Petrescu S, Szabo P, et al. "Human Oversight of Automated Decisions: a large-scale evaluation". Proc MLSys 18(2017): 575-581. DOI: 10.3389/fdgth.2026.1699005
Quintero B, Balogh Q, et al. "Human Oversight of Automated Decisions: a case study". Proc FAccT 20(2020): 586-593. DOI: 10.3389/fdgth.2026.1834800
Ristic C, Castellani R, et al. "Human Oversight of Automated Decisions: a survey". Nat Mach Intell 22(2023): 597-605. DOI: 10.2196/69678
Szabo J, Jankowiak S, et al. "Human Oversight of Automated Decisions: a comparative analysis". J Mach Learn Res 24(2014): 608-617. DOI: 10.7759/cureus.46454
Tarasov Q, Quintero T, et al. "Human Oversight of Automated Decisions: a field deployment report". IEEE Trans Pattern Anal Mach Intell 26(2017): 619-629. DOI: 10.1016/j.ssci.2025.107028
Urbanek Z, Zielinski U, et al. "Human Oversight of Automated Decisions: a benchmark study". Proc NeurIPS 28(2020): 630-641. DOI: 10.2196/48633
Vasquez A, Alvez V, et al. "Human Oversight of Automated Decisions: a systematic evaluation". Proc ICML 30(2023): 641-653. DOI: 10.1371/journal.pone.0305949
Wojcik H, Hendriksen W, et al. "Human Oversight of Automated Decisions: a empirical analysis". Proc ICLR 32(2014): 652-665. DOI: 10.1371/journal.pdig.0000514
Zielinski O, Ostrowski Z, et al. "Human Oversight of Automated Decisions: a ablation study". ACM Trans Intell Syst 34(2017): 663-677. DOI: 10.17219/acem/208596
Andric V, Vasquez A, et al. "Human Oversight of Automated Decisions: a reproducibility study". Data Min Knowl Discov 36(2020): 674-680. DOI: 10.7759/cureus.104086
