Serving Latency Under Bursty Traffic: A Technical Note on Batching, Caching and Tail Behaviour

Authors

  • Panikos Sofocleous Department of Platform Engineering, Marathasa Centre for Machine Intelligence, Kalopanayiotis, Cyprus Author
  • Amadeu Ourique Department of Machine Learning Operations, Alto Minho Institute of Applied Artificial Intelligence, Viana do Castelo, Portugal Author
  • Reto Aeschlimann Department of High Performance Computing, Surselva Institute of Applied Artificial Intelligence, Ilanz, Switzerland Author

DOI:

https://doi.org/10.5281/zenodo.22660693

Keywords:

Serving Latency Under Bursty Traffic; Applied machine learning; Evidence synthesis; Production deployment; MLOps

Abstract

Serving Latency Under Bursty Traffic is an area of active development in applied machine learning, with evidence that has
accumulated unevenly across its component questions. We review the current evidence base for serving latency under bursty
traffic, spanning methodological, implementation, governance, and cost dimensions, focusing on where practice has converged
and where it continues to vary, and set out the practical considerations relevant to a team considering its adoption. We conclude
that selective, context-specific use is best supported by the evidence currently available, rather than uniform adoption or
avoidance.

References

Alvez D, Dumitrescu A, et al. "Serving Latency Under Bursty Traffic: a benchmark study". J Mach Learn Res 8(2014): 80-86. DOI: 10.1038/s41746-023-00852-5

Brankovic K, Kowalczyk B, et al. "Serving Latency Under Bursty Traffic: a systematic evaluation". IEEE Trans Pattern Anal Mach Intell 10(2017): 91-98. DOI: 10.1016/j.mayocpiqo.2022.09.004

Castellani R, Ristic C, et al. "Serving Latency Under Bursty Traffic: a empirical analysis". Proc NeurIPS 12(2020): 102-110. DOI: 10.1177/08404704241291226

Dumitrescu A, Andric D, et al. "Serving Latency Under Bursty Traffic: a ablation study". Proc ICML 14(2023): 113-122. DOI: 10.1093/medlaw/fwad013

Eriksdottir B, Brankovic E, et al. "Serving Latency Under Bursty Traffic: a reproducibility study". Proc ICLR 16(2014): 124-134. DOI: 10.2196/89834

Farkas I, Ionescu F, et al. "Serving Latency Under Bursty Traffic: a large-scale evaluation". ACM Trans Intell Syst 18(2017): 135-146. DOI: 10.1007/s10916-024-02104-9

Grabowski P, Petrescu G, et al. "Serving Latency Under Bursty Traffic: a case study". Data Min Knowl Discov 20(2020): 146-158. DOI: 10.3389/fdgth.2026.1896184

Hendriksen W, Wojcik H, et al. "Serving Latency Under Bursty Traffic: a survey". Artif Intell 22(2023): 157-170. DOI: 10.3390/diagnostics15162117

Ionescu F, Foscarini I, et al. "Serving Latency Under Bursty Traffic: a comparative analysis". Knowl Based Syst 24(2014): 168-182. DOI:

4103/jrpp.jrpp_20_65 10. Jankowiak G, Grabowski J, et al. "Serving Latency Under Bursty Traffic: a field deployment report". IEEE Trans Neural Netw Learn Syst 26(2017): 179-185. DOI: 10.3389/fdgth.2026.1763884

Kowalczyk N, Novotna K, et al. "Serving Latency Under Bursty Traffic: a benchmark study". Proc KDD 28(2020): 190-197. DOI: 10.2196/48297

Lindqvist U, Urbanek L, et al. "Serving Latency Under Bursty Traffic: a systematic evaluation". Proc AAAI 30(2023): 201-209. DOI: 10.1007/s00330-025-12031-z

Marchetti D, Doncheva M, et al. "Serving Latency Under Bursty Traffic: a empirical analysis". IEEE Trans Knowl Data Eng 32(2014): 212-221. DOI: 10.3389/fdgth.2026.1699005

Novotna E, Eriksdottir N, et al. "Serving Latency Under Bursty Traffic: a ablation study". J Artif Intell Res 34(2017): 223-233. DOI: 10.3389/fdgth.2026.1834800

Ostrowski L, Lindqvist O, et al. "Serving Latency Under Bursty Traffic: a reproducibility study". ACM Comput Surv 36(2020): 234-245. DOI: 10.2196/69678

Petrescu S, Szabo P, et al. "Serving Latency Under Bursty Traffic: a large-scale evaluation". Neural Comput Appl 38(2023): 245-257. DOI: 10.7759/cureus.46454

Quintero B, Balogh Q, et al. "Serving Latency Under Bursty Traffic: a case study". Expert Syst Appl 40(2014): 256-269. DOI: 10.1016/j.ssci.2025.107028

Ristic C, Castellani R, et al. "Serving Latency Under Bursty Traffic: a survey". Inf Sci 42(2017): 267-281. DOI: 10.2196/48633

Szabo J, Jankowiak S, et al. "Serving Latency Under Bursty Traffic: a comparative analysis". Pattern Recognit 44(2020): 278-284. DOI: 10.1371/journal.pone.0305949

Tarasov Q, Quintero T, et al. "Serving Latency Under Bursty Traffic: a field deployment report". IEEE Access 46(2023): 289-296. DOI: 10.1371/journal.pdig.0000514

Urbanek Z, Zielinski U, et al. "Serving Latency Under Bursty Traffic: a benchmark study". ACM Trans Softw Eng Methodol 8(2014): 300-308. DOI: 10.17219/acem/208596

Vasquez A, Alvez V, et al. "Serving Latency Under Bursty Traffic: a systematic evaluation". Proc MLSys 10(2017): 311-320. DOI: 10.7759/cureus.104086

Wojcik H, Hendriksen W, et al. "Serving Latency Under Bursty Traffic: a empirical analysis". Proc FAccT 12(2020): 322-332. DOI: 10.1038/s41746-023-00852-5

Zielinski O, Ostrowski Z, et al. "Serving Latency Under Bursty Traffic: a ablation study". Nat Mach Intell 14(2023): 333-344. DOI: 10.1016/j.mayocpiqo.2022.09.004

Andric V, Vasquez A, et al. "Serving Latency Under Bursty Traffic: a reproducibility study". J Mach Learn Res 16(2014): 344-356. DOI: 10.1177/08404704241291226

Balogh E, Emmerich B, et al. "Serving Latency Under Bursty Traffic: a large-scale evaluation". IEEE Trans Pattern Anal Mach Intell 18(2017): 355-368. DOI: 10.1093/medlaw/fwad013

Chiriac F, Farkas C, et al. "Serving Latency Under Bursty Traffic: a case study". Proc NeurIPS 20(2020): 366-380. DOI: 10.2196/89834

Doncheva M, Marchetti D, et al. "Serving Latency Under Bursty Traffic: a survey". Proc ICML 22(2023): 377-383. DOI: 10.1007/s10916-024-02104-9

Emmerich T, Tarasov E, et al. "Serving Latency Under Bursty Traffic: a comparative analysis". Proc ICLR 24(2014): 388-395. DOI: 10.3389/fdgth.2026.1896184

Foscarini C, Chiriac F, et al. "Serving Latency Under Bursty Traffic: a field deployment report". ACM Trans Intell Syst 26(2017): 399-407. DOI: 10.3390/diagnostics15162117

Alvez D, Dumitrescu A, et al. "Serving Latency Under Bursty Traffic: a benchmark study". Data Min Knowl Discov 28(2020): 410-419. DOI: 10.4103/jrpp.jrpp_20_65

Brankovic K, Kowalczyk B, et al. "Serving Latency Under Bursty Traffic: a systematic evaluation". Artif Intell 30(2023): 421-431. DOI: 10.3389/fdgth.2026.1763884

Castellani R, Ristic C, et al. "Serving Latency Under Bursty Traffic: a empirical analysis". Knowl Based Syst 32(2014): 432-443. DOI: 10.2196/48297

Dumitrescu A, Andric D, et al. "Serving Latency Under Bursty Traffic: a ablation study". IEEE Trans Neural Netw Learn Syst 34(2017): 443-455. DOI: 10.1007/s00330-025-12031-z

Eriksdottir B, Brankovic E, et al. "Serving Latency Under Bursty Traffic: a reproducibility study". Proc KDD 36(2020): 454-467. DOI: 10.3389/fdgth.2026.1699005

Farkas I, Ionescu F, et al. "Serving Latency Under Bursty Traffic: a large-scale evaluation". Proc AAAI 38(2023): 465-479. DOI: 10.3389/fdgth.2026.1834800

Grabowski P, Petrescu G, et al. "Serving Latency Under Bursty Traffic: a case study". IEEE Trans Knowl Data Eng 40(2014): 476-482. DOI: 10.2196/69678

Hendriksen W, Wojcik H, et al. "Serving Latency Under Bursty Traffic: a survey". J Artif Intell Res 42(2017): 487-494. DOI: 10.7759/cureus.46454

Ionescu F, Foscarini I, et al. "Serving Latency Under Bursty Traffic: a comparative analysis". ACM Comput Surv 44(2020): 498-506. DOI: 10.1016/j.ssci.2025.107028

Jankowiak G, Grabowski J, et al. "Serving Latency Under Bursty Traffic: a field deployment report". Neural Comput Appl 46(2023): 509-518. DOI: 10.2196/48633

Kowalczyk N, Novotna K, et al. "Serving Latency Under Bursty Traffic: a benchmark study". Expert Syst Appl 8(2014): 520-530. DOI: 10.1371/journal.pone.0305949

Lindqvist U, Urbanek L, et al. "Serving Latency Under Bursty Traffic: a systematic evaluation". Inf Sci 10(2017): 531-542. DOI: 10.1371/journal.pdig.0000514

Marchetti D, Doncheva M, et al. "Serving Latency Under Bursty Traffic: a empirical analysis". Pattern Recognit 12(2020): 542-554. DOI: 10.17219/acem/208596

Novotna E, Eriksdottir N, et al. "Serving Latency Under Bursty Traffic: a ablation study". IEEE Access 14(2023): 553-566. DOI: 10.7759/cureus.104086

Ostrowski L, Lindqvist O, et al. "Serving Latency Under Bursty Traffic: a reproducibility study". ACM Trans Softw Eng Methodol 16(2014): 564-578. DOI: 10.1038/s41746-023-00852-5

Petrescu S, Szabo P, et al. "Serving Latency Under Bursty Traffic: a large-scale evaluation". Proc MLSys 18(2017): 575-581. DOI: 10.1016/j.mayocpiqo.2022.09.004

Quintero B, Balogh Q, et al. "Serving Latency Under Bursty Traffic: a case study". Proc FAccT 20(2020): 586-593. DOI: 10.1177/08404704241291226

Ristic C, Castellani R, et al. "Serving Latency Under Bursty Traffic: a survey". Nat Mach Intell 22(2023): 597-605. DOI: 10.1093/medlaw/fwad013

Szabo J, Jankowiak S, et al. "Serving Latency Under Bursty Traffic: a comparative analysis". J Mach Learn Res 24(2014): 608-617. DOI: 10.2196/89834

Tarasov Q, Quintero T, et al. "Serving Latency Under Bursty Traffic: a field deployment report". IEEE Trans Pattern Anal Mach Intell 26(2017): 619-629. DOI: 10.1007/s10916-024-02104-9

Urbanek Z, Zielinski U, et al. "Serving Latency Under Bursty Traffic: a benchmark study". Proc NeurIPS 28(2020): 630-641. DOI: 10.3389/fdgth.2026.1896184

Vasquez A, Alvez V, et al. "Serving Latency Under Bursty Traffic: a systematic evaluation". Proc ICML 30(2023): 641-653. DOI: 10.3390/diagnostics15162117

Wojcik H, Hendriksen W, et al. "Serving Latency Under Bursty Traffic: a empirical analysis". Proc ICLR 32(2014): 652-665. DOI: 10.4103/jrpp.jrpp_20_65

Zielinski O, Ostrowski Z, et al. "Serving Latency Under Bursty Traffic: a ablation study". ACM Trans Intell Syst 34(2017): 663-677. DOI: 10.3389/fdgth.2026.1763884

Andric V, Vasquez A, et al. "Serving Latency Under Bursty Traffic: a reproducibility study". Data Min Knowl Discov 36(2020): 674-680. DOI: 10.2196/48297

Downloads

Published

2024-06-12

Issue

Vol. 3 No. 1 (2024): vol 3 issue 1 2024

Section

Articles

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.