Continuous Evaluation of Deployed Models: A Technical Note on Shadow Traffic and Guardrail Metrics
DOI:
https://doi.org/10.5281/zenodo.22660833Keywords:
Continuous Evaluation of Deployed Models; Applied machine learning; Evidence synthesis; Production deployment; MLOpsAbstract
Continuous Evaluation of Deployed Models is an area of active development in applied machine learning, with evidence that has
accumulated unevenly across its component questions. We review the current evidence base for continuous evaluation of deployed
models, spanning methodological, implementation, governance, and cost dimensions, focusing on where practice has converged
and where it continues to vary, and set out the practical considerations relevant to a team considering its adoption. We conclude
that selective, context-specific use is best supported by the evidence currently available, rather than uniform adoption or
avoidance.
References
Alvez D, Dumitrescu A, et al. "Continuous Evaluation of Deployed Models: a benchmark study". J Mach Learn Res 8(2014): 80-86. DOI: 10.3390/diagnostics15162117
Brankovic K, Kowalczyk B, et al. "Continuous Evaluation of Deployed Models: a systematic evaluation". IEEE Trans Pattern Anal Mach Intell 10(2017): 91-98. DOI: 10.4103/jrpp.jrpp_20_65
Castellani R, Ristic C, et al. "Continuous Evaluation of Deployed Models: a empirical analysis". Proc NeurIPS 12(2020): 102-110. DOI: 10.3389/fdgth.2026.1763884
Dumitrescu A, Andric D, et al. "Continuous Evaluation of Deployed Models: a ablation study". Proc ICML 14(2023): 113-122. DOI: 10.2196/48297
Eriksdottir B, Brankovic E, et al. "Continuous Evaluation of Deployed Models: a reproducibility study". Proc ICLR 16(2014): 124-134. DOI: 10.1007/s00330-025-12031-z
Farkas I, Ionescu F, et al. "Continuous Evaluation of Deployed Models: a large-scale evaluation". ACM Trans Intell Syst 18(2017): 135-146. DOI: 10.3389/fdgth.2026.1699005
Grabowski P, Petrescu G, et al. "Continuous Evaluation of Deployed Models: a case study". Data Min Knowl Discov 20(2020): 146-158. DOI: 10.3389/fdgth.2026.1834800
Hendriksen W, Wojcik H, et al. "Continuous Evaluation of Deployed Models: a survey". Artif Intell 22(2023): 157-170. DOI: 10.2196/69678
Ionescu F, Foscarini I, et al. "Continuous Evaluation of Deployed Models: a comparative analysis". Knowl Based Syst 24(2014): 168-182. DOI: 10.7759/cureus.46454
Jankowiak G, Grabowski J, et al. "Continuous Evaluation of Deployed Models: a field deployment report". IEEE Trans Neural Netw Learn Syst 26(2017): 179-185. DOI: 10.1016/j.ssci.2025.107028
Kowalczyk N, Novotna K, et al. "Continuous Evaluation of Deployed Models: a benchmark study". Proc KDD 28(2020): 190-197. DOI: 10.2196/48633
Lindqvist U, Urbanek L, et al. "Continuous Evaluation of Deployed Models: a systematic evaluation". Proc AAAI 30(2023): 201-209. DOI: 10.1371/journal.pone.0305949
Marchetti D, Doncheva M, et al. "Continuous Evaluation of Deployed Models: a empirical analysis". IEEE Trans Knowl Data Eng 32(2014): 212-221. DOI: 10.1371/journal.pdig.0000514
Novotna E, Eriksdottir N, et al. "Continuous Evaluation of Deployed Models: a ablation study". J Artif Intell Res 34(2017): 223-233. DOI: 10.17219/acem/208596
Ostrowski L, Lindqvist O, et al. "Continuous Evaluation of Deployed Models: a reproducibility study". ACM Comput Surv 36(2020): 234-245. DOI: 10.7759/cureus.104086
Petrescu S, Szabo P, et al. "Continuous Evaluation of Deployed Models: a large-scale evaluation". Neural Comput Appl 38(2023): 245-257. DOI: 10.1038/s41746-023-00852-5
Quintero B, Balogh Q, et al. "Continuous Evaluation of Deployed Models: a case study". Expert Syst Appl 40(2014): 256-269. DOI: 10.1016/j.mayocpiqo.2022.09.004
Ristic C, Castellani R, et al. "Continuous Evaluation of Deployed Models: a survey". Inf Sci 42(2017): 267-281. DOI: 10.1177/08404704241291226
Szabo J, Jankowiak S, et al. "Continuous Evaluation of Deployed Models: a comparative analysis". Pattern Recognit 44(2020): 278-284. DOI: 10.1093/medlaw/fwad013
Tarasov Q, Quintero T, et al. "Continuous Evaluation of Deployed Models: a field deployment report". IEEE Access 46(2023): 289-296. DOI: 10.2196/89834
Urbanek Z, Zielinski U, et al. "Continuous Evaluation of Deployed Models: a benchmark study". ACM Trans Softw Eng Methodol 8(2014): 300-308. DOI: 10.1007/s10916-024-02104-9
Vasquez A, Alvez V, et al. "Continuous Evaluation of Deployed Models: a systematic evaluation". Proc MLSys 10(2017): 311-320. DOI: 10.3389/fdgth.2026.1896184
Wojcik H, Hendriksen W, et al. "Continuous Evaluation of Deployed Models: a empirical analysis". Proc FAccT 12(2020): 322-332. DOI: 10.3390/diagnostics15162117
Zielinski O, Ostrowski Z, et al. "Continuous Evaluation of Deployed Models: a ablation study". Nat Mach Intell 14(2023): 333-344. DOI: 10.4103/jrpp.jrpp_20_65
Andric V, Vasquez A, et al. "Continuous Evaluation of Deployed Models: a reproducibility study". J Mach Learn Res 16(2014): 344-356. DOI: 10.3389/fdgth.2026.1763884
Balogh E, Emmerich B, et al. "Continuous Evaluation of Deployed Models: a large-scale evaluation". IEEE Trans Pattern Anal Mach Intell 18(2017): 355-368. DOI: 10.2196/48297
Chiriac F, Farkas C, et al. "Continuous Evaluation of Deployed Models: a case study". Proc NeurIPS 20(2020): 366-380. DOI: 10.1007/s00330-025-12031-z
Doncheva M, Marchetti D, et al. "Continuous Evaluation of Deployed Models: a survey". Proc ICML 22(2023): 377-383. DOI: 10.3389/fdgth.2026.1699005
Emmerich T, Tarasov E, et al. "Continuous Evaluation of Deployed Models: a comparative analysis". Proc ICLR 24(2014): 388-395. DOI: 10.3389/fdgth.2026.1834800
Foscarini C, Chiriac F, et al. "Continuous Evaluation of Deployed Models: a field deployment report". ACM Trans Intell Syst 26(2017): 399-407. DOI: 10.2196/69678
Alvez D, Dumitrescu A, et al. "Continuous Evaluation of Deployed Models: a benchmark study". Data Min Knowl Discov 28(2020): 410-419. DOI: 10.7759/cureus.46454
Brankovic K, Kowalczyk B, et al. "Continuous Evaluation of Deployed Models: a systematic evaluation". Artif Intell 30(2023): 421-431. DOI: 10.1016/j.ssci.2025.107028
Castellani R, Ristic C, et al. "Continuous Evaluation of Deployed Models: a empirical analysis". Knowl Based Syst 32(2014): 432-443. DOI: 10.2196/48633
Dumitrescu A, Andric D, et al. "Continuous Evaluation of Deployed Models: a ablation study". IEEE Trans Neural Netw Learn Syst 34(2017): 443-455. DOI: 10.1371/journal.pone.0305949
Eriksdottir B, Brankovic E, et al. "Continuous Evaluation of Deployed Models: a reproducibility study". Proc KDD 36(2020): 454-467. DOI: 10.1371/journal.pdig.0000514
Farkas I, Ionescu F, et al. "Continuous Evaluation of Deployed Models: a large-scale evaluation". Proc AAAI 38(2023): 465-479. DOI: 10.17219/acem/208596
Grabowski P, Petrescu G, et al. "Continuous Evaluation of Deployed Models: a case study". IEEE Trans Knowl Data Eng 40(2014): 476-482. DOI: 10.7759/cureus.104086
Hendriksen W, Wojcik H, et al. "Continuous Evaluation of Deployed Models: a survey". J Artif Intell Res 42(2017): 487-494. DOI: 10.1038/s41746-023-00852-5
Ionescu F, Foscarini I, et al. "Continuous Evaluation of Deployed Models: a comparative analysis". ACM Comput Surv 44(2020): 498-506. DOI: 10.1016/j.mayocpiqo.2022.09.004
Jankowiak G, Grabowski J, et al. "Continuous Evaluation of Deployed Models: a field deployment report". Neural Comput Appl 46(2023): 509-518. DOI: 10.1177/08404704241291226
Kowalczyk N, Novotna K, et al. "Continuous Evaluation of Deployed Models: a benchmark study". Expert Syst Appl 8(2014): 520-530. DOI: 10.1093/medlaw/fwad013
Lindqvist U, Urbanek L, et al. "Continuous Evaluation of Deployed Models: a systematic evaluation". Inf Sci 10(2017): 531-542. DOI: 10.2196/89834
Marchetti D, Doncheva M, et al. "Continuous Evaluation of Deployed Models: a empirical analysis". Pattern Recognit 12(2020): 542-554. DOI: 10.1007/s10916-024-02104-9
Novotna E, Eriksdottir N, et al. "Continuous Evaluation of Deployed Models: a ablation study". IEEE Access 14(2023): 553-566. DOI: 10.3389/fdgth.2026.1896184
Ostrowski L, Lindqvist O, et al. "Continuous Evaluation of Deployed Models: a reproducibility study". ACM Trans Softw Eng Methodol 16(2014): 564-578. DOI: 10.3390/diagnostics15162117
Petrescu S, Szabo P, et al. "Continuous Evaluation of Deployed Models: a large-scale evaluation". Proc MLSys 18(2017): 575-581. DOI: 10.4103/jrpp.jrpp_20_65
Quintero B, Balogh Q, et al. "Continuous Evaluation of Deployed Models: a case study". Proc FAccT 20(2020): 586-593. DOI: 10.3389/fdgth.2026.1763884
Ristic C, Castellani R, et al. "Continuous Evaluation of Deployed Models: a survey". Nat Mach Intell 22(2023): 597-605. DOI: 10.2196/48297
Szabo J, Jankowiak S, et al. "Continuous Evaluation of Deployed Models: a comparative analysis". J Mach Learn Res 24(2014): 608-617. DOI: 10.1007/s00330-025-12031-z
Tarasov Q, Quintero T, et al. "Continuous Evaluation of Deployed Models: a field deployment report". IEEE Trans Pattern Anal Mach Intell 26(2017): 619-629. DOI: 10.3389/fdgth.2026.1699005
Urbanek Z, Zielinski U, et al. "Continuous Evaluation of Deployed Models: a benchmark study". Proc NeurIPS 28(2020): 630-641. DOI: 10.3389/fdgth.2026.1834800
Vasquez A, Alvez V, et al. "Continuous Evaluation of Deployed Models: a systematic evaluation". Proc ICML 30(2023): 641-653. DOI: 10.2196/69678
Wojcik H, Hendriksen W, et al. "Continuous Evaluation of Deployed Models: a empirical analysis". Proc ICLR 32(2014): 652-665. DOI: 10.7759/cureus.46454
Zielinski O, Ostrowski Z, et al. "Continuous Evaluation of Deployed Models: a ablation study". ACM Trans Intell Syst 34(2017): 663-677. DOI: 10.1016/j.ssci.2025.107028
Andric V, Vasquez A, et al. "Continuous Evaluation of Deployed Models: a reproducibility study". Data Min Knowl Discov 36(2020): 674-680. DOI: 10.2196/48633
