Retrieval Evaluation Beyond Recall at K: Aligning Offline Metrics With Downstream Answer Quality

Authors

  • Botond Ferenczy Centre for Information Retrieval, Jaszsag Institute of Applied Artificial Intelligence, Jaszbereny, Hungary Author
  • Klemens Rieplhuber Centre for Language Technology, Bucklige Welt Institute of Applied Artificial Intelligence, Wiener Neustadt, Austria Author
  • Torben Skaarup Department of Statistics, Vendsyssel Institute of Applied Artificial Intelligence, Hjorring, Denmark Author

DOI:

https://doi.org/10.5281/zenodo.22660788

Keywords:

Retrieval Evaluation Beyond Recall at K; Applied machine learning; Evidence synthesis; Production deployment; MLOps

Abstract

Retrieval Evaluation Beyond Recall at K is an area of active development in applied machine learning, with evidence that has
accumulated unevenly across its component questions. We review the current evidence base for retrieval evaluation beyond recall
at k, spanning methodological, implementation, governance, and cost dimensions, focusing on where practice has converged and
where it continues to vary, and set out the practical considerations relevant to a team considering its adoption. We conclude that
selective, context-specific use is best supported by the evidence currently available, rather than uniform adoption or avoidance.

References

Alvez D, Dumitrescu A, et al. "Retrieval Evaluation Beyond Recall at K: a benchmark study". J Mach Learn Res 8(2014): 80-86. DOI: 10.1177/08404704241291226

Brankovic K, Kowalczyk B, et al. "Retrieval Evaluation Beyond Recall at K: a systematic evaluation". IEEE Trans Pattern Anal Mach Intell 10(2017): 91-98. DOI: 10.1093/medlaw/fwad013

Castellani R, Ristic C, et al. "Retrieval Evaluation Beyond Recall at K: a empirical analysis". Proc NeurIPS 12(2020): 102-110. DOI: 10.2196/89834

Dumitrescu A, Andric D, et al. "Retrieval Evaluation Beyond Recall at K: a ablation study". Proc ICML 14(2023): 113-122. DOI: 10.1007/s10916-024-02104-9

Eriksdottir B, Brankovic E, et al. "Retrieval Evaluation Beyond Recall at K: a reproducibility study". Proc ICLR 16(2014): 124-134. DOI: 10.3389/fdgth.2026.1896184

Farkas I, Ionescu F, et al. "Retrieval Evaluation Beyond Recall at K: a large-scale evaluation". ACM Trans Intell Syst 18(2017): 135-146. DOI: 10.3390/diagnostics15162117

Grabowski P, Petrescu G, et al. "Retrieval Evaluation Beyond Recall at K: a case study". Data Min Knowl Discov 20(2020): 146-158. DOI: 10.4103/jrpp.jrpp_20_65

Hendriksen W, Wojcik H, et al. "Retrieval Evaluation Beyond Recall at K: a survey". Artif Intell 22(2023): 157-170. DOI: 10.3389/fdgth.2026.1763884

Ionescu F, Foscarini I, et al. "Retrieval Evaluation Beyond Recall at K: a comparative analysis". Knowl Based Syst 24(2014): 168-182. DOI: 10.2196/48297

Jankowiak G, Grabowski J, et al. "Retrieval Evaluation Beyond Recall at K: a field deployment report". IEEE Trans Neural Netw Learn Syst 26(2017): 179-185. DOI: 10.1007/s00330-025-12031-z

Kowalczyk N, Novotna K, et al. "Retrieval Evaluation Beyond Recall at K: a benchmark study". Proc KDD 28(2020): 190-197. DOI: 10.3389/fdgth.2026.1699005

Lindqvist U, Urbanek L, et al. "Retrieval Evaluation Beyond Recall at K: a systematic evaluation". Proc AAAI 30(2023): 201-209. DOI: 10.3389/fdgth.2026.1834800

Marchetti D, Doncheva M, et al. "Retrieval Evaluation Beyond Recall at K: a empirical analysis". IEEE Trans Knowl Data Eng 32(2014): 212-221. DOI: 10.2196/69678

Novotna E, Eriksdottir N, et al. "Retrieval Evaluation Beyond Recall at K: a ablation study". J Artif Intell Res 34(2017): 223-233. DOI: 10.7759/cureus.46454

Ostrowski L, Lindqvist O, et al. "Retrieval Evaluation Beyond Recall at K: a reproducibility study". ACM Comput Surv 36(2020): 234-245. DOI: 10.1016/j.ssci.2025.107028

Petrescu S, Szabo P, et al. "Retrieval Evaluation Beyond Recall at K: a large-scale evaluation". Neural Comput Appl 38(2023): 245-257. DOI: 10.2196/48633

Quintero B, Balogh Q, et al. "Retrieval Evaluation Beyond Recall at K: a case study". Expert Syst Appl 40(2014): 256-269. DOI: 10.1371/journal.pone.0305949

Ristic C, Castellani R, et al. "Retrieval Evaluation Beyond Recall at K: a survey". Inf Sci 42(2017): 267-281. DOI: 10.1371/journal.pdig.0000514

Szabo J, Jankowiak S, et al. "Retrieval Evaluation Beyond Recall at K: a comparative analysis". Pattern Recognit 44(2020): 278-284. DOI: 10.17219/acem/208596

Tarasov Q, Quintero T, et al. "Retrieval Evaluation Beyond Recall at K: a field deployment report". IEEE Access 46(2023): 289-296. DOI: 10.7759/cureus.104086

Urbanek Z, Zielinski U, et al. "Retrieval Evaluation Beyond Recall at K: a benchmark study". ACM Trans Softw Eng Methodol 8(2014): 300-308. DOI: 10.1038/s41746-023-00852-5

Vasquez A, Alvez V, et al. "Retrieval Evaluation Beyond Recall at K: a systematic evaluation". Proc MLSys 10(2017): 311-320. DOI: 10.1016/j.mayocpiqo.2022.09.004

Wojcik H, Hendriksen W, et al. "Retrieval Evaluation Beyond Recall at K: a empirical analysis". Proc FAccT 12(2020): 322-332. DOI: 10.1177/08404704241291226

Zielinski O, Ostrowski Z, et al. "Retrieval Evaluation Beyond Recall at K: a ablation study". Nat Mach Intell 14(2023): 333-344. DOI: 10.1093/medlaw/fwad013

Andric V, Vasquez A, et al. "Retrieval Evaluation Beyond Recall at K: a reproducibility study". J Mach Learn Res 16(2014): 344-356. DOI: 10.2196/89834

Balogh E, Emmerich B, et al. "Retrieval Evaluation Beyond Recall at K: a large-scale evaluation". IEEE Trans Pattern Anal Mach Intell 18(2017): 355-368. DOI: 10.1007/s10916-024-02104-9

Chiriac F, Farkas C, et al. "Retrieval Evaluation Beyond Recall at K: a case study". Proc NeurIPS 20(2020): 366-380. DOI: 10.3389/fdgth.2026.1896184

Doncheva M, Marchetti D, et al. "Retrieval Evaluation Beyond Recall at K: a survey". Proc ICML 22(2023): 377-383. DOI: 10.3390/diagnostics15162117

Emmerich T, Tarasov E, et al. "Retrieval Evaluation Beyond Recall at K: a comparative analysis". Proc ICLR 24(2014): 388-395. DOI: 10.4103/jrpp.jrpp_20_65

Foscarini C, Chiriac F, et al. "Retrieval Evaluation Beyond Recall at K: a field deployment report". ACM Trans Intell Syst 26(2017): 399-407. DOI: 10.3389/fdgth.2026.1763884

Alvez D, Dumitrescu A, et al. "Retrieval Evaluation Beyond Recall at K: a benchmark study". Data Min Knowl Discov 28(2020): 410-419. DOI: 10.2196/48297

Brankovic K, Kowalczyk B, et al. "Retrieval Evaluation Beyond Recall at K: a systematic evaluation". Artif Intell 30(2023): 421-431. DOI: 10.1007/s00330-025-12031-z

Castellani R, Ristic C, et al. "Retrieval Evaluation Beyond Recall at K: a empirical analysis". Knowl Based Syst 32(2014): 432-443. DOI: 10.3389/fdgth.2026.1699005

Dumitrescu A, Andric D, et al. "Retrieval Evaluation Beyond Recall at K: a ablation study". IEEE Trans Neural Netw Learn Syst 34(2017): 443-455. DOI: 10.3389/fdgth.2026.1834800

Eriksdottir B, Brankovic E, et al. "Retrieval Evaluation Beyond Recall at K: a reproducibility study". Proc KDD 36(2020): 454-467. DOI: 10.2196/69678

Farkas I, Ionescu F, et al. "Retrieval Evaluation Beyond Recall at K: a large-scale evaluation". Proc AAAI 38(2023): 465-479. DOI: 10.7759/cureus.46454

Grabowski P, Petrescu G, et al. "Retrieval Evaluation Beyond Recall at K: a case study". IEEE Trans Knowl Data Eng 40(2014): 476-482. DOI: 10.1016/j.ssci.2025.107028

Hendriksen W, Wojcik H, et al. "Retrieval Evaluation Beyond Recall at K: a survey". J Artif Intell Res 42(2017): 487-494. DOI: 10.2196/48633

Ionescu F, Foscarini I, et al. "Retrieval Evaluation Beyond Recall at K: a comparative analysis". ACM Comput Surv 44(2020): 498-506. DOI: 10.1371/journal.pone.0305949

Jankowiak G, Grabowski J, et al. "Retrieval Evaluation Beyond Recall at K: a field deployment report". Neural Comput Appl 46(2023): 509-518. DOI: 10.1371/journal.pdig.0000514

Kowalczyk N, Novotna K, et al. "Retrieval Evaluation Beyond Recall at K: a benchmark study". Expert Syst Appl 8(2014): 520-530. DOI: 10.17219/acem/208596

Lindqvist U, Urbanek L, et al. "Retrieval Evaluation Beyond Recall at K: a systematic evaluation". Inf Sci 10(2017): 531-542. DOI: 10.7759/cureus.104086

Marchetti D, Doncheva M, et al. "Retrieval Evaluation Beyond Recall at K: a empirical analysis". Pattern Recognit 12(2020): 542-554. DOI: 10.1038/s41746-023-00852-5

Novotna E, Eriksdottir N, et al. "Retrieval Evaluation Beyond Recall at K: a ablation study". IEEE Access 14(2023): 553-566. DOI: 10.1016/j.mayocpiqo.2022.09.004

Ostrowski L, Lindqvist O, et al. "Retrieval Evaluation Beyond Recall at K: a reproducibility study". ACM Trans Softw Eng Methodol 16(2014): 564-578. DOI: 10.1177/08404704241291226

Petrescu S, Szabo P, et al. "Retrieval Evaluation Beyond Recall at K: a large-scale evaluation". Proc MLSys 18(2017): 575-581. DOI: 10.1093/medlaw/fwad013

Quintero B, Balogh Q, et al. "Retrieval Evaluation Beyond Recall at K: a case study". Proc FAccT 20(2020): 586-593. DOI: 10.2196/89834

Ristic C, Castellani R, et al. "Retrieval Evaluation Beyond Recall at K: a survey". Nat Mach Intell 22(2023): 597-605. DOI: 10.1007/s10916-024-02104-9

Szabo J, Jankowiak S, et al. "Retrieval Evaluation Beyond Recall at K: a comparative analysis". J Mach Learn Res 24(2014): 608-617. DOI: 10.3389/fdgth.2026.1896184

Tarasov Q, Quintero T, et al. "Retrieval Evaluation Beyond Recall at K: a field deployment report". IEEE Trans Pattern Anal Mach Intell 26(2017): 619-629. DOI: 10.3390/diagnostics15162117

Urbanek Z, Zielinski U, et al. "Retrieval Evaluation Beyond Recall at K: a benchmark study". Proc NeurIPS 28(2020): 630-641. DOI: 10.4103/jrpp.jrpp_20_65

Vasquez A, Alvez V, et al. "Retrieval Evaluation Beyond Recall at K: a systematic evaluation". Proc ICML 30(2023): 641-653. DOI: 10.3389/fdgth.2026.1763884

Wojcik H, Hendriksen W, et al. "Retrieval Evaluation Beyond Recall at K: a empirical analysis". Proc ICLR 32(2014): 652-665. DOI: 10.2196/48297

Zielinski O, Ostrowski Z, et al. "Retrieval Evaluation Beyond Recall at K: a ablation study". ACM Trans Intell Syst 34(2017): 663-677. DOI: 10.1007/s00330-025-12031-z

Andric V, Vasquez A, et al. "Retrieval Evaluation Beyond Recall at K: a reproducibility study". Data Min Knowl Discov 36(2020): 674-680. DOI: 10.3389/fdgth.2026.1699005

Downloads

Published

2025-01-01

Issue

Vol. 4 No. 1 (2025): vol 4 issue 1 2025

Section

Articles

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.