Automated Program Repair on Real Bug Reports: A Replication Study Across Two Benchmarks

Authors

  • Roberta Borg-Cardona Department of Software Engineering, Birgu Institute of Applied Artificial Intelligence, Birgu, Malta Author
  • Jarmila Duchonova Department of Software Engineering, Zahorie Institute of Applied Artificial Intelligence, Malacky, Slovakia Author
  • Ottone Guglielminetti Centre for Machine Learning Research, Polesine Institute of Data Science, Rovigo, Italy Author

DOI:

https://doi.org/10.5281/zenodo.22660677

Keywords:

Automated Program Repair on Real Bug Reports; Applied machine learning; Evidence synthesis; Production deployment; MLOps

Abstract

Automated Program Repair on Real Bug Reports is an area of active development in applied machine learning, with evidence that
has accumulated unevenly across its component questions. We review the current evidence base for automated program repair on
real bug reports, spanning methodological, implementation, governance, and cost dimensions, focusing on where practice has
converged and where it continues to vary, and set out the practical considerations relevant to a team considering its adoption. We
conclude that selective, context-specific use is best supported by the evidence currently available, rather than uniform adoption or
avoidance.

References

Alvez D, Dumitrescu A, et al. "Automated Program Repair on Real Bug Reports: a benchmark study". J Mach Learn Res 8(2014): 80-86. DOI: 10.7759/cureus.104086

Brankovic K, Kowalczyk B, et al. "Automated Program Repair on Real Bug Reports: a systematic evaluation". IEEE Trans Pattern Anal Mach Intell 10(2017): 91-98. DOI: 10.1038/s41746-023-00852-5

Castellani R, Ristic C, et al. "Automated Program Repair on Real Bug Reports: a empirical analysis". Proc NeurIPS 12(2020): 102-110. DOI: 10.1016/j.mayocpiqo.2022.09.004

Dumitrescu A, Andric D, et al. "Automated Program Repair on Real Bug Reports: a ablation study". Proc ICML 14(2023): 113-122. DOI: 10.1177/08404704241291226

Eriksdottir B, Brankovic E, et al. "Automated Program Repair on Real Bug Reports: a reproducibility study". Proc ICLR 16(2014): 124-134. DOI: 10.1093/medlaw/fwad013

Farkas I, Ionescu F, et al. "Automated Program Repair on Real Bug Reports: a large-scale evaluation". ACM Trans Intell Syst 18(2017): 135-146. DOI: 10.2196/89834

Grabowski P, Petrescu G, et al. "Automated Program Repair on Real Bug Reports: a case study". Data Min Knowl Discov 20(2020): 146-158. DOI: 10.1007/s10916-024-02104-9

Hendriksen W, Wojcik H, et al. "Automated Program Repair on Real Bug Reports: a survey". Artif Intell 22(2023): 157-170. DOI: 10.3389/fdgth.2026.1896184

Ionescu F, Foscarini I, et al. "Automated Program Repair on Real Bug Reports: a comparative analysis". Knowl Based Syst 24(2014): 168-182. DOI: 10.3390/diagnostics15162117

Jankowiak G, Grabowski J, et al. "Automated Program Repair on Real Bug Reports: a field deployment report". IEEE Trans Neural Netw Learn Syst 26(2017): 179-185. DOI: 10.4103/jrpp.jrpp_20_65

Kowalczyk N, Novotna K, et al. "Automated Program Repair on Real Bug Reports: a benchmark study". Proc KDD 28(2020): 190-197. DOI: 10.3389/fdgth.2026.1763884

Lindqvist U, Urbanek L, et al. "Automated Program Repair on Real Bug Reports: a systematic evaluation". Proc AAAI 30(2023): 201-209. DOI: 10.2196/48297

Marchetti D, Doncheva M, et al. "Automated Program Repair on Real Bug Reports: a empirical analysis". IEEE Trans Knowl Data Eng 32(2014): 212-221. DOI: 10.1007/s00330-025-12031-z

Novotna E, Eriksdottir N, et al. "Automated Program Repair on Real Bug Reports: a ablation study". J Artif Intell Res 34(2017): 223-233. DOI: 10.3389/fdgth.2026.1699005

Ostrowski L, Lindqvist O, et al. "Automated Program Repair on Real Bug Reports: a reproducibility study". ACM Comput Surv 36(2020): 234-245. DOI: 10.3389/fdgth.2026.1834800

Petrescu S, Szabo P, et al. "Automated Program Repair on Real Bug Reports: a large-scale evaluation". Neural Comput Appl 38(2023): 245-257. DOI: 10.2196/69678

Quintero B, Balogh Q, et al. "Automated Program Repair on Real Bug Reports: a case study". Expert Syst Appl 40(2014): 256-269. DOI: 10.7759/cureus.46454

Ristic C, Castellani R, et al. "Automated Program Repair on Real Bug Reports: a survey". Inf Sci 42(2017): 267-281. DOI: 10.1016/j.ssci.2025.107028

Szabo J, Jankowiak S, et al. "Automated Program Repair on Real Bug Reports: a comparative analysis". Pattern Recognit 44(2020): 278-284. DOI: 10.2196/48633

Tarasov Q, Quintero T, et al. "Automated Program Repair on Real Bug Reports: a field deployment report". IEEE Access 46(2023): 289-296. DOI: 10.1371/journal.pone.0305949

Urbanek Z, Zielinski U, et al. "Automated Program Repair on Real Bug Reports: a benchmark study". ACM Trans Softw Eng Methodol 8(2014): 300-308. DOI: 10.1371/journal.pdig.0000514

Vasquez A, Alvez V, et al. "Automated Program Repair on Real Bug Reports: a systematic evaluation". Proc MLSys 10(2017): 311-320. DOI: 10.17219/acem/208596

Wojcik H, Hendriksen W, et al. "Automated Program Repair on Real Bug Reports: a empirical analysis". Proc FAccT 12(2020): 322-332. DOI: 10.7759/cureus.104086

Zielinski O, Ostrowski Z, et al. "Automated Program Repair on Real Bug Reports: a ablation study". Nat Mach Intell 14(2023): 333-344. DOI: 10.1038/s41746-023-00852-5

Andric V, Vasquez A, et al. "Automated Program Repair on Real Bug Reports: a reproducibility study". J Mach Learn Res 16(2014): 344-356. DOI: 10.1016/j.mayocpiqo.2022.09.004

Balogh E, Emmerich B, et al. "Automated Program Repair on Real Bug Reports: a large-scale evaluation". IEEE Trans Pattern Anal Mach Intell 18(2017): 355-368. DOI: 10.1177/08404704241291226

Chiriac F, Farkas C, et al. "Automated Program Repair on Real Bug Reports: a case study". Proc NeurIPS 20(2020): 366-380. DOI: 10.1093/medlaw/fwad013

Doncheva M, Marchetti D, et al. "Automated Program Repair on Real Bug Reports: a survey". Proc ICML 22(2023): 377-383. DOI: 10.2196/89834

Emmerich T, Tarasov E, et al. "Automated Program Repair on Real Bug Reports: a comparative analysis". Proc ICLR 24(2014): 388-395. DOI: 10.1007/s10916-024-02104-9

Foscarini C, Chiriac F, et al. "Automated Program Repair on Real Bug Reports: a field deployment report". ACM Trans Intell Syst 26(2017): 399-407. DOI: 10.3389/fdgth.2026.1896184

Alvez D, Dumitrescu A, et al. "Automated Program Repair on Real Bug Reports: a benchmark study". Data Min Knowl Discov 28(2020): 410-419. DOI: 10.3390/diagnostics15162117

Brankovic K, Kowalczyk B, et al. "Automated Program Repair on Real Bug Reports: a systematic evaluation". Artif Intell 30(2023): 421-431. DOI: 10.4103/jrpp.jrpp_20_65

Castellani R, Ristic C, et al. "Automated Program Repair on Real Bug Reports: a empirical analysis". Knowl Based Syst 32(2014): 432-443. DOI: 10.3389/fdgth.2026.1763884

Dumitrescu A, Andric D, et al. "Automated Program Repair on Real Bug Reports: a ablation study". IEEE Trans Neural Netw Learn Syst 34(2017): 443-455. DOI: 10.2196/48297

Eriksdottir B, Brankovic E, et al. "Automated Program Repair on Real Bug Reports: a reproducibility study". Proc KDD 36(2020): 454-467. DOI: 10.1007/s00330-025-12031-z

Farkas I, Ionescu F, et al. "Automated Program Repair on Real Bug Reports: a large-scale evaluation". Proc AAAI 38(2023): 465-479. DOI: 10.3389/fdgth.2026.1699005

Grabowski P, Petrescu G, et al. "Automated Program Repair on Real Bug Reports: a case study". IEEE Trans Knowl Data Eng 40(2014): 476-482. DOI: 10.3389/fdgth.2026.1834800

Hendriksen W, Wojcik H, et al. "Automated Program Repair on Real Bug Reports: a survey". J Artif Intell Res 42(2017): 487-494. DOI: 10.2196/69678

Ionescu F, Foscarini I, et al. "Automated Program Repair on Real Bug Reports: a comparative analysis". ACM Comput Surv 44(2020): 498-506. DOI: 10.7759/cureus.46454

Jankowiak G, Grabowski J, et al. "Automated Program Repair on Real Bug Reports: a field deployment report". Neural Comput Appl 46(2023): 509-518. DOI: 10.1016/j.ssci.2025.107028

Kowalczyk N, Novotna K, et al. "Automated Program Repair on Real Bug Reports: a benchmark study". Expert Syst Appl 8(2014): 520-530. DOI: 10.2196/48633

Lindqvist U, Urbanek L, et al. "Automated Program Repair on Real Bug Reports: a systematic evaluation". Inf Sci 10(2017): 531-542. DOI: 10.1371/journal.pone.0305949

Marchetti D, Doncheva M, et al. "Automated Program Repair on Real Bug Reports: a empirical analysis". Pattern Recognit 12(2020): 542-554. DOI: 10.1371/journal.pdig.0000514

Novotna E, Eriksdottir N, et al. "Automated Program Repair on Real Bug Reports: a ablation study". IEEE Access 14(2023): 553-566. DOI: 10.17219/acem/208596

Ostrowski L, Lindqvist O, et al. "Automated Program Repair on Real Bug Reports: a reproducibility study". ACM Trans Softw Eng Methodol 16(2014): 564-578. DOI: 10.7759/cureus.104086

Petrescu S, Szabo P, et al. "Automated Program Repair on Real Bug Reports: a large-scale evaluation". Proc MLSys 18(2017): 575-581. DOI: 10.1038/s41746-023-00852-5

Quintero B, Balogh Q, et al. "Automated Program Repair on Real Bug Reports: a case study". Proc FAccT 20(2020): 586-593. DOI: 10.1016/j.mayocpiqo.2022.09.004

Ristic C, Castellani R, et al. "Automated Program Repair on Real Bug Reports: a survey". Nat Mach Intell 22(2023): 597-605. DOI: 10.1177/08404704241291226

Szabo J, Jankowiak S, et al. "Automated Program Repair on Real Bug Reports: a comparative analysis". J Mach Learn Res 24(2014): 608-617. DOI: 10.1093/medlaw/fwad013

Tarasov Q, Quintero T, et al. "Automated Program Repair on Real Bug Reports: a field deployment report". IEEE Trans Pattern Anal Mach Intell 26(2017): 619-629. DOI: 10.2196/89834

Urbanek Z, Zielinski U, et al. "Automated Program Repair on Real Bug Reports: a benchmark study". Proc NeurIPS 28(2020): 630-641. DOI: 10.1007/s10916-024-02104-9

Vasquez A, Alvez V, et al. "Automated Program Repair on Real Bug Reports: a systematic evaluation". Proc ICML 30(2023): 641-653. DOI: 10.3389/fdgth.2026.1896184

Wojcik H, Hendriksen W, et al. "Automated Program Repair on Real Bug Reports: a empirical analysis". Proc ICLR 32(2014): 652-665. DOI: 10.3390/diagnostics15162117

Zielinski O, Ostrowski Z, et al. "Automated Program Repair on Real Bug Reports: a ablation study". ACM Trans Intell Syst 34(2017): 663-677. DOI: 10.4103/jrpp.jrpp_20_65

Andric V, Vasquez A, et al. "Automated Program Repair on Real Bug Reports: a reproducibility study". Data Min Knowl Discov 36(2020): 674-680. DOI: 10.3389/fdgth.2026.1763884

Downloads

Published

2024-05-12

Issue

Vol. 3 No. 1 (2024): vol 3 issue 1 2024

Section

Articles

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.