Evaluation Harness Design for Applied Tasks: A Technical Note on Seeds, Splits and Reporting Variance

Authors

  • Nicolas Papacharalambous Department of Software Engineering, Marathasa Centre for Machine Intelligence, Kalopanayiotis, Cyprus Author
  • Kalevi Manninen Department of Statistics, Paijat-Hame Institute of Applied Artificial Intelligence, Lahti, Finland Author
  • Astrid Sagstad Centre for Machine Learning Research, Ostfold Centre for Machine Intelligence, Sarpsborg, Norway Author

DOI:

https://doi.org/10.5281/zenodo.22661260

Keywords:

Evaluation Harness Design for Applied Tasks; Applied machine learning; Evidence synthesis; Production deployment; MLOps

Abstract

Evaluation Harness Design for Applied Tasks is an area of active development in applied machine learning, with evidence that
has accumulated unevenly across its component questions. We review the current evidence base for evaluation harness design for
applied tasks, spanning methodological, implementation, governance, and cost dimensions, focusing on where practice has
converged and where it continues to vary, and set out the practical considerations relevant to a team considering its adoption. We
conclude that selective, context-specific use is best supported by the evidence currently available, rather than uniform adoption or
avoidance.

References

Alvez D, Dumitrescu A, et al. "Evaluation Harness Design for Applied Tasks: a benchmark study". J Mach Learn Res 8(2014): 80-86. DOI: 10.3389/fdgth.2026.1699005

Brankovic K, Kowalczyk B, et al. "Evaluation Harness Design for Applied Tasks: a systematic evaluation". IEEE Trans Pattern Anal Mach Intell 10(2017): 91-98. DOI: 10.3389/fdgth.2026.1834800

Castellani R, Ristic C, et al. "Evaluation Harness Design for Applied Tasks: a empirical analysis". Proc NeurIPS 12(2020): 102-110. DOI: 10.2196/69678

Dumitrescu A, Andric D, et al. "Evaluation Harness Design for Applied Tasks: a ablation study". Proc ICML 14(2023): 113-122. DOI: 10.7759/cureus.46454

Eriksdottir B, Brankovic E, et al. "Evaluation Harness Design for Applied Tasks: a reproducibility study". Proc ICLR 16(2014): 124-134. DOI: 10.1016/j.ssci.2025.107028

Farkas I, Ionescu F, et al. "Evaluation Harness Design for Applied Tasks: a large-scale evaluation". ACM Trans Intell Syst 18(2017): 135-146. DOI: 10.2196/48633

Grabowski P, Petrescu G, et al. "Evaluation Harness Design for Applied Tasks: a case study". Data Min Knowl Discov 20(2020): 146-158. DOI: 10.1371/journal.pone.0305949

Hendriksen W, Wojcik H, et al. "Evaluation Harness Design for Applied Tasks: a survey". Artif Intell 22(2023): 157-170. DOI: 10.1371/journal.pdig.0000514

Ionescu F, Foscarini I, et al. "Evaluation Harness Design for Applied Tasks: a comparative analysis". Knowl Based Syst 24(2014): 168-182. DOI: 10.17219/acem/208596

Jankowiak G, Grabowski J, et al. "Evaluation Harness Design for Applied Tasks: a field deployment report". IEEE Trans Neural Netw Learn Syst 26(2017): 179-185. DOI: 10.7759/cureus.104086

Kowalczyk N, Novotna K, et al. "Evaluation Harness Design for Applied Tasks: a benchmark study". Proc KDD 28(2020): 190-197. DOI: 10.1038/s41746-023-00852-5

Lindqvist U, Urbanek L, et al. "Evaluation Harness Design for Applied Tasks: a systematic evaluation". Proc AAAI 30(2023): 201-209. DOI: 10.1016/j.mayocpiqo.2022.09.004

Marchetti D, Doncheva M, et al. "Evaluation Harness Design for Applied Tasks: a empirical analysis". IEEE Trans Knowl Data Eng 32(2014): 212-221. DOI: 10.1177/08404704241291226

Novotna E, Eriksdottir N, et al. "Evaluation Harness Design for Applied Tasks: a ablation study". J Artif Intell Res 34(2017): 223-233. DOI: 10.1093/medlaw/fwad013

Ostrowski L, Lindqvist O, et al. "Evaluation Harness Design for Applied Tasks: a reproducibility study". ACM Comput Surv 36(2020): 234-245. DOI: 10.2196/89834

Petrescu S, Szabo P, et al. "Evaluation Harness Design for Applied Tasks: a large-scale evaluation". Neural Comput Appl 38(2023): 245-257. DOI: 10.1007/s10916-024-02104-9

Quintero B, Balogh Q, et al. "Evaluation Harness Design for Applied Tasks: a case study". Expert Syst Appl 40(2014): 256-269. DOI: 10.3389/fdgth.2026.1896184

Ristic C, Castellani R, et al. "Evaluation Harness Design for Applied Tasks: a survey". Inf Sci 42(2017): 267-281. DOI: 10.3390/diagnostics15162117

Szabo J, Jankowiak S, et al. "Evaluation Harness Design for Applied Tasks: a comparative analysis". Pattern Recognit 44(2020): 278-284. DOI: 10.4103/jrpp.jrpp_20_65

Tarasov Q, Quintero T, et al. "Evaluation Harness Design for Applied Tasks: a field deployment report". IEEE Access 46(2023): 289-296. DOI: 10.3389/fdgth.2026.1763884

Urbanek Z, Zielinski U, et al. "Evaluation Harness Design for Applied Tasks: a benchmark study". ACM Trans Softw Eng Methodol 8(2014): 300-308. DOI: 10.2196/48297

Vasquez A, Alvez V, et al. "Evaluation Harness Design for Applied Tasks: a systematic evaluation". Proc MLSys 10(2017): 311-320. DOI: 10.1007/s00330-025-12031-z

Wojcik H, Hendriksen W, et al. "Evaluation Harness Design for Applied Tasks: a empirical analysis". Proc FAccT 12(2020): 322-332. DOI: 10.3389/fdgth.2026.1699005

Zielinski O, Ostrowski Z, et al. "Evaluation Harness Design for Applied Tasks: a ablation study". Nat Mach Intell 14(2023): 333-344. DOI: 10.3389/fdgth.2026.1834800

Andric V, Vasquez A, et al. "Evaluation Harness Design for Applied Tasks: a reproducibility study". J Mach Learn Res 16(2014): 344-356. DOI: 10.2196/69678

Balogh E, Emmerich B, et al. "Evaluation Harness Design for Applied Tasks: a large-scale evaluation". IEEE Trans Pattern Anal Mach Intell 18(2017): 355-368. DOI: 10.7759/cureus.46454

Chiriac F, Farkas C, et al. "Evaluation Harness Design for Applied Tasks: a case study". Proc NeurIPS 20(2020): 366-380. DOI: 10.1016/j.ssci.2025.107028

Doncheva M, Marchetti D, et al. "Evaluation Harness Design for Applied Tasks: a survey". Proc ICML 22(2023): 377-383. DOI: 10.2196/48633

Emmerich T, Tarasov E, et al. "Evaluation Harness Design for Applied Tasks: a comparative analysis". Proc ICLR 24(2014): 388-395. DOI: 10.1371/journal.pone.0305949

Foscarini C, Chiriac F, et al. "Evaluation Harness Design for Applied Tasks: a field deployment report". ACM Trans Intell Syst 26(2017): 399-407. DOI: 10.1371/journal.pdig.0000514

Alvez D, Dumitrescu A, et al. "Evaluation Harness Design for Applied Tasks: a benchmark study". Data Min Knowl Discov 28(2020): 410-419. DOI: 10.17219/acem/208596

Brankovic K, Kowalczyk B, et al. "Evaluation Harness Design for Applied Tasks: a systematic evaluation". Artif Intell 30(2023): 421-431. DOI: 10.7759/cureus.104086

Castellani R, Ristic C, et al. "Evaluation Harness Design for Applied Tasks: a empirical analysis". Knowl Based Syst 32(2014): 432-443. DOI: 10.1038/s41746-023-00852-5

Dumitrescu A, Andric D, et al. "Evaluation Harness Design for Applied Tasks: a ablation study". IEEE Trans Neural Netw Learn Syst 34(2017): 443-455. DOI: 10.1016/j.mayocpiqo.2022.09.004

Eriksdottir B, Brankovic E, et al. "Evaluation Harness Design for Applied Tasks: a reproducibility study". Proc KDD 36(2020): 454-467. DOI: 10.1177/08404704241291226

Farkas I, Ionescu F, et al. "Evaluation Harness Design for Applied Tasks: a large-scale evaluation". Proc AAAI 38(2023): 465-479. DOI: 10.1093/medlaw/fwad013

Grabowski P, Petrescu G, et al. "Evaluation Harness Design for Applied Tasks: a case study". IEEE Trans Knowl Data Eng 40(2014): 476-482. DOI: 10.2196/89834

Hendriksen W, Wojcik H, et al. "Evaluation Harness Design for Applied Tasks: a survey". J Artif Intell Res 42(2017): 487-494. DOI: 10.1007/s10916-024-02104-9

Ionescu F, Foscarini I, et al. "Evaluation Harness Design for Applied Tasks: a comparative analysis". ACM Comput Surv 44(2020): 498-506. DOI: 10.3389/fdgth.2026.1896184

Jankowiak G, Grabowski J, et al. "Evaluation Harness Design for Applied Tasks: a field deployment report". Neural Comput Appl 46(2023): 509-518. DOI: 10.3390/diagnostics15162117

Kowalczyk N, Novotna K, et al. "Evaluation Harness Design for Applied Tasks: a benchmark study". Expert Syst Appl 8(2014): 520-530. DOI: 10.4103/jrpp.jrpp_20_65

Lindqvist U, Urbanek L, et al. "Evaluation Harness Design for Applied Tasks: a systematic evaluation". Inf Sci 10(2017): 531-542. DOI: 10.3389/fdgth.2026.1763884

Marchetti D, Doncheva M, et al. "Evaluation Harness Design for Applied Tasks: a empirical analysis". Pattern Recognit 12(2020): 542-554. DOI: 10.2196/48297

Novotna E, Eriksdottir N, et al. "Evaluation Harness Design for Applied Tasks: a ablation study". IEEE Access 14(2023): 553-566. DOI: 10.1007/s00330-025-12031-z

Ostrowski L, Lindqvist O, et al. "Evaluation Harness Design for Applied Tasks: a reproducibility study". ACM Trans Softw Eng Methodol 16(2014): 564-578. DOI: 10.3389/fdgth.2026.1699005

Petrescu S, Szabo P, et al. "Evaluation Harness Design for Applied Tasks: a large-scale evaluation". Proc MLSys 18(2017): 575-581. DOI: 10.3389/fdgth.2026.1834800

Quintero B, Balogh Q, et al. "Evaluation Harness Design for Applied Tasks: a case study". Proc FAccT 20(2020): 586-593. DOI: 10.2196/69678

Ristic C, Castellani R, et al. "Evaluation Harness Design for Applied Tasks: a survey". Nat Mach Intell 22(2023): 597-605. DOI: 10.7759/cureus.46454

Szabo J, Jankowiak S, et al. "Evaluation Harness Design for Applied Tasks: a comparative analysis". J Mach Learn Res 24(2014): 608-617. DOI: 10.1016/j.ssci.2025.107028

Tarasov Q, Quintero T, et al. "Evaluation Harness Design for Applied Tasks: a field deployment report". IEEE Trans Pattern Anal Mach Intell 26(2017): 619-629. DOI: 10.2196/48633

Urbanek Z, Zielinski U, et al. "Evaluation Harness Design for Applied Tasks: a benchmark study". Proc NeurIPS 28(2020): 630-641. DOI: 10.1371/journal.pone.0305949

Vasquez A, Alvez V, et al. "Evaluation Harness Design for Applied Tasks: a systematic evaluation". Proc ICML 30(2023): 641-653. DOI: 10.1371/journal.pdig.0000514

Wojcik H, Hendriksen W, et al. "Evaluation Harness Design for Applied Tasks: a empirical analysis". Proc ICLR 32(2014): 652-665. DOI: 10.17219/acem/208596

Zielinski O, Ostrowski Z, et al. "Evaluation Harness Design for Applied Tasks: a ablation study". ACM Trans Intell Syst 34(2017): 663-677. DOI: 10.7759/cureus.104086

Andric V, Vasquez A, et al. "Evaluation Harness Design for Applied Tasks: a reproducibility study". Data Min Knowl Discov 36(2020): 674-680. DOI: 10.1038/s41746-023-00852-5

Downloads

Published

2026-07-07

Issue

Vol. 5 No. 2 (2026): vol 5 issue 2 2026

Section

Articles

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.