Benchmark Contamination in Public Leaderboards: Measuring Overlap Between Test Sets and Pretraining Corpora
DOI:
https://doi.org/10.5281/zenodo.22660489Keywords:
Benchmark Contamination in Public Leaderboards; Applied machine learning; Evidence synthesis; Production deployment; MLOpsAbstract
Benchmark Contamination in Public Leaderboards is an area of active development in applied machine learning, with evidence
that has accumulated unevenly across its component questions. We review the current evidence base for benchmark contamination
in public leaderboards, spanning methodological, implementation, governance, and cost dimensions, focusing on where practice
has converged and where it continues to vary, and set out the practical considerations relevant to a team considering its adoption.
We conclude that selective, context-specific use is best supported by the evidence currently available, rather than uniform
adoption or avoidance.
References
Alvez D, Dumitrescu A, et al. "Benchmark Contamination in Public Leaderboards: a benchmark study". J Mach Learn Res 8(2014): 80-86. DOI: 10.3389/fdgth.2026.1834800
Brankovic K, Kowalczyk B, et al. "Benchmark Contamination in Public Leaderboards: a systematic evaluation". IEEE Trans Pattern Anal Mach Intell 10(2017): 91-98. DOI: 10.2196/69678
Castellani R, Ristic C, et al. "Benchmark Contamination in Public Leaderboards: a empirical analysis". Proc NeurIPS 12(2020): 102-110. DOI: 10.7759/cureus.46454
Dumitrescu A, Andric D, et al. "Benchmark Contamination in Public Leaderboards: a ablation study". Proc ICML 14(2023): 113-122. DOI: 10.1016/j.ssci.2025.107028
Eriksdottir B, Brankovic E, et al. "Benchmark Contamination in Public Leaderboards: a reproducibility study". Proc ICLR 16(2014): 124-134. DOI: 10.2196/48633
Farkas I, Ionescu F, et al. "Benchmark Contamination in Public Leaderboards: a large-scale evaluation". ACM Trans Intell Syst 18(2017): 135-146. DOI: 10.1371/journal.pone.0305949
Grabowski P, Petrescu G, et al. "Benchmark Contamination in Public Leaderboards: a case study". Data Min Knowl Discov 20(2020): 146-158. DOI: 10.1371/journal.pdig.0000514
Hendriksen W, Wojcik H, et al. "Benchmark Contamination in Public Leaderboards: a survey". Artif Intell 22(2023): 157-170. DOI: 10.17219/acem/208596
Ionescu F, Foscarini I, et al. "Benchmark Contamination in Public Leaderboards: a comparative analysis". Knowl Based Syst 24(2014): 168-182. DOI: 10.7759/cureus.104086
Jankowiak G, Grabowski J, et al. "Benchmark Contamination in Public Leaderboards: a field deployment report". IEEE Trans Neural Netw Learn Syst 26(2017): 179-185. DOI: 10.1038/s41746-023-00852-5
Kowalczyk N, Novotna K, et al. "Benchmark Contamination in Public Leaderboards: a benchmark study". Proc KDD 28(2020): 190-197. DOI: 10.1016/j.mayocpiqo.2022.09.004
Lindqvist U, Urbanek L, et al. "Benchmark Contamination in Public Leaderboards: a systematic evaluation". Proc AAAI 30(2023): 201-209. DOI: 10.1177/08404704241291226
Marchetti D, Doncheva M, et al. "Benchmark Contamination in Public Leaderboards: a empirical analysis". IEEE Trans Knowl Data Eng 32(2014): 212-221. DOI: 10.1093/medlaw/fwad013
Novotna E, Eriksdottir N, et al. "Benchmark Contamination in Public Leaderboards: a ablation study". J Artif Intell Res 34(2017): 223-233. DOI: 10.2196/89834
Ostrowski L, Lindqvist O, et al. "Benchmark Contamination in Public Leaderboards: a reproducibility study". ACM Comput Surv 36(2020): 234-245. DOI: 10.1007/s10916-024-02104-9
Petrescu S, Szabo P, et al. "Benchmark Contamination in Public Leaderboards: a large-scale evaluation". Neural Comput Appl 38(2023): 245-257. DOI: 10.3389/fdgth.2026.1896184
Quintero B, Balogh Q, et al. "Benchmark Contamination in Public Leaderboards: a case study". Expert Syst Appl 40(2014): 256-269. DOI: 10.3390/diagnostics15162117
Ristic C, Castellani R, et al. "Benchmark Contamination in Public Leaderboards: a survey". Inf Sci 42(2017): 267-281. DOI: 10.4103/jrpp.jrpp_20_65
Szabo J, Jankowiak S, et al. "Benchmark Contamination in Public Leaderboards: a comparative analysis". Pattern Recognit 44(2020): 278-284. DOI: 10.3389/fdgth.2026.1763884
Tarasov Q, Quintero T, et al. "Benchmark Contamination in Public Leaderboards: a field deployment report". IEEE Access 46(2023): 289-296. DOI: 10.2196/48297
Urbanek Z, Zielinski U, et al. "Benchmark Contamination in Public Leaderboards: a benchmark study". ACM Trans Softw Eng Methodol 8(2014): 300-308. DOI: 10.1007/s00330-025-12031-z
Vasquez A, Alvez V, et al. "Benchmark Contamination in Public Leaderboards: a systematic evaluation". Proc MLSys 10(2017): 311-320. DOI: 10.3389/fdgth.2026.1699005
Wojcik H, Hendriksen W, et al. "Benchmark Contamination in Public Leaderboards: a empirical analysis". Proc FAccT 12(2020): 322-332. DOI: 10.3389/fdgth.2026.1834800
Zielinski O, Ostrowski Z, et al. "Benchmark Contamination in Public Leaderboards: a ablation study". Nat Mach Intell 14(2023): 333-344. DOI: 10.2196/69678
Andric V, Vasquez A, et al. "Benchmark Contamination in Public Leaderboards: a reproducibility study". J Mach Learn Res 16(2014): 344-356. DOI: 10.7759/cureus.46454
Balogh E, Emmerich B, et al. "Benchmark Contamination in Public Leaderboards: a large-scale evaluation". IEEE Trans Pattern Anal Mach Intell 18(2017): 355-368. DOI: 10.1016/j.ssci.2025.107028
Chiriac F, Farkas C, et al. "Benchmark Contamination in Public Leaderboards: a case study". Proc NeurIPS 20(2020): 366-380. DOI: 10.2196/48633
Doncheva M, Marchetti D, et al. "Benchmark Contamination in Public Leaderboards: a survey". Proc ICML 22(2023): 377-383. DOI: 10.1371/journal.pone.0305949
Emmerich T, Tarasov E, et al. "Benchmark Contamination in Public Leaderboards: a comparative analysis". Proc ICLR 24(2014): 388-395. DOI: 10.1371/journal.pdig.0000514
Foscarini C, Chiriac F, et al. "Benchmark Contamination in Public Leaderboards: a field deployment report". ACM Trans Intell Syst 26(2017): 399-407. DOI: 10.17219/acem/208596
Alvez D, Dumitrescu A, et al. "Benchmark Contamination in Public Leaderboards: a benchmark study". Data Min Knowl Discov 28(2020): 410-419. DOI: 10.7759/cureus.104086
Brankovic K, Kowalczyk B, et al. "Benchmark Contamination in Public Leaderboards: a systematic evaluation". Artif Intell 30(2023): 421-431. DOI: 10.1038/s41746-023-00852-5
Castellani R, Ristic C, et al. "Benchmark Contamination in Public Leaderboards: a empirical analysis". Knowl Based Syst 32(2014): 432-443. DOI: 10.1016/j.mayocpiqo.2022.09.004
Dumitrescu A, Andric D, et al. "Benchmark Contamination in Public Leaderboards: a ablation study". IEEE Trans Neural Netw Learn Syst 34(2017): 443-455. DOI: 10.1177/08404704241291226
Eriksdottir B, Brankovic E, et al. "Benchmark Contamination in Public Leaderboards: a reproducibility study". Proc KDD 36(2020): 454-467. DOI: 10.1093/medlaw/fwad013
Farkas I, Ionescu F, et al. "Benchmark Contamination in Public Leaderboards: a large-scale evaluation". Proc AAAI 38(2023): 465-479. DOI: 10.2196/89834
Grabowski P, Petrescu G, et al. "Benchmark Contamination in Public Leaderboards: a case study". IEEE Trans Knowl Data Eng 40(2014): 476-482. DOI: 10.1007/s10916-024-02104-9
Hendriksen W, Wojcik H, et al. "Benchmark Contamination in Public Leaderboards: a survey". J Artif Intell Res 42(2017): 487-494. DOI: 10.3389/fdgth.2026.1896184
Ionescu F, Foscarini I, et al. "Benchmark Contamination in Public Leaderboards: a comparative analysis". ACM Comput Surv 44(2020): 498-506. DOI: 10.3390/diagnostics15162117
Jankowiak G, Grabowski J, et al. "Benchmark Contamination in Public Leaderboards: a field deployment report". Neural Comput Appl 46(2023): 509-518. DOI: 10.4103/jrpp.jrpp_20_65
Kowalczyk N, Novotna K, et al. "Benchmark Contamination in Public Leaderboards: a benchmark study". Expert Syst Appl 8(2014): 520-530. DOI: 10.3389/fdgth.2026.1763884
Lindqvist U, Urbanek L, et al. "Benchmark Contamination in Public Leaderboards: a systematic evaluation". Inf Sci 10(2017): 531-542. DOI: 10.2196/48297
Marchetti D, Doncheva M, et al. "Benchmark Contamination in Public Leaderboards: a empirical analysis". Pattern Recognit 12(2020): 542-554. DOI: 10.1007/s00330-025-12031-z
Novotna E, Eriksdottir N, et al. "Benchmark Contamination in Public Leaderboards: a ablation study". IEEE Access 14(2023): 553-566. DOI: 10.3389/fdgth.2026.1699005
Ostrowski L, Lindqvist O, et al. "Benchmark Contamination in Public Leaderboards: a reproducibility study". ACM Trans Softw Eng Methodol 16(2014): 564-578. DOI: 10.3389/fdgth.2026.1834800
Petrescu S, Szabo P, et al. "Benchmark Contamination in Public Leaderboards: a large-scale evaluation". Proc MLSys 18(2017): 575-581. DOI: 10.2196/69678
Quintero B, Balogh Q, et al. "Benchmark Contamination in Public Leaderboards: a case study". Proc FAccT 20(2020): 586-593. DOI: 10.7759/cureus.46454
Ristic C, Castellani R, et al. "Benchmark Contamination in Public Leaderboards: a survey". Nat Mach Intell 22(2023): 597-605. DOI: 10.1016/j.ssci.2025.107028
Szabo J, Jankowiak S, et al. "Benchmark Contamination in Public Leaderboards: a comparative analysis". J Mach Learn Res 24(2014): 608-617. DOI: 10.2196/48633
Tarasov Q, Quintero T, et al. "Benchmark Contamination in Public Leaderboards: a field deployment report". IEEE Trans Pattern Anal Mach Intell 26(2017): 619-629. DOI: 10.1371/journal.pone.0305949
Urbanek Z, Zielinski U, et al. "Benchmark Contamination in Public Leaderboards: a benchmark study". Proc NeurIPS 28(2020): 630-641. DOI: 10.1371/journal.pdig.0000514
Vasquez A, Alvez V, et al. "Benchmark Contamination in Public Leaderboards: a systematic evaluation". Proc ICML 30(2023): 641-653. DOI: 10.17219/acem/208596
Wojcik H, Hendriksen W, et al. "Benchmark Contamination in Public Leaderboards: a empirical analysis". Proc ICLR 32(2014): 652-665. DOI: 10.7759/cureus.104086
Zielinski O, Ostrowski Z, et al. "Benchmark Contamination in Public Leaderboards: a ablation study". ACM Trans Intell Syst 34(2017): 663-677. DOI: 10.1038/s41746-023-00852-5
Andric V, Vasquez A, et al. "Benchmark Contamination in Public Leaderboards: a reproducibility study". Data Min Knowl Discov 36(2020): 674-680. DOI: 10.1016/j.mayocpiqo.2022.09.004
