Annotation Disagreement as Signal Rather Than Noise: A Technical Note on Soft Label Aggregation

Authors

  • Milena Dimitrakiev Department of Data Science, Dobrudzha Institute of Applied Artificial Intelligence, Dobrich, Bulgaria Author
  • Aivars Krumins Centre for Machine Learning Research, Selonia Centre for Machine Intelligence, Jekabpils, Latvia Author
  • Zoe Zavallis Department of Statistics, Marathasa Centre for Machine Intelligence, Kalopanayiotis, Cyprus Author

DOI:

https://doi.org/10.5281/zenodo.22657226

Keywords:

Annotation Disagreement as Signal Rather Than Noise; Applied machine learning; Evidence synthesis; Production deployment; MLOps

Abstract

Annotation Disagreement as Signal Rather Than Noise is an area of active development in applied machine learning, with
evidence that has accumulated unevenly across its component questions. We review the current evidence base for annotation
disagreement as signal rather than noise, spanning methodological, implementation, governance, and cost dimensions, focusing
on where practice has converged and where it continues to vary, and set out the practical considerations relevant to a team
considering its adoption. We conclude that selective, context-specific use is best supported by the evidence currently available,
rather than uniform adoption or avoidance

References

Alvez D, Dumitrescu A, et al. "Annotation Disagreement as Signal Rather Than Noise: a benchmark study". J Mach Learn Res 8(2014): 80-86. DOI: 10.4103/jrpp.jrpp_20_65

Brankovic K, Kowalczyk B, et al. "Annotation Disagreement as Signal Rather Than Noise: a systematic evaluation". IEEE Trans Pattern Anal Mach Intell 10(2017): 91-98. DOI: 10.3389/fdgth.2026.1763884

Castellani R, Ristic C, et al. "Annotation Disagreement as Signal Rather Than Noise: a empirical analysis". Proc NeurIPS 12(2020): 102-110. DOI: 10.2196/48297

Dumitrescu A, Andric D, et al. "Annotation Disagreement as Signal Rather Than Noise: a ablation study". Proc ICML 14(2023): 113-122. DOI: 10.1007/s00330-025-12031-z

Eriksdottir B, Brankovic E, et al. "Annotation Disagreement as Signal Rather Than Noise: a reproducibility study". Proc ICLR 16(2014): 124-134. DOI: 10.3389/fdgth.2026.1699005

Farkas I, Ionescu F, et al. "Annotation Disagreement as Signal Rather Than Noise: a large-scale evaluation". ACM Trans Intell Syst 18(2017): 135-146. DOI: 10.3389/fdgth.2026.1834800

Grabowski P, Petrescu G, et al. "Annotation Disagreement as Signal Rather Than Noise: a case study". Data Min Knowl Discov 20(2020): 146-158. DOI: 10.2196/69678

Hendriksen W, Wojcik H, et al. "Annotation Disagreement as Signal Rather Than Noise: a survey". Artif Intell 22(2023): 157-170. DOI: 10.7759/cureus.46454

Ionescu F, Foscarini I, et al. "Annotation Disagreement as Signal Rather Than Noise: a comparative analysis". Knowl Based Syst 24(2014): 168-182. DOI: 10.1016/j.ssci.2025.107028

Jankowiak G, Grabowski J, et al. "Annotation Disagreement as Signal Rather Than Noise: a field deployment report". IEEE Trans Neural Netw Learn Syst 26(2017): 179-185. DOI: 10.2196/48633

Kowalczyk N, Novotna K, et al. "Annotation Disagreement as Signal Rather Than Noise: a benchmark study". Proc KDD 28(2020): 190-197. DOI: 10.1371/journal.pone.0305949

Lindqvist U, Urbanek L, et al. "Annotation Disagreement as Signal Rather Than Noise: a systematic evaluation". Proc AAAI 30(2023): 201-209. DOI: 10.1371/journal.pdig.0000514

Marchetti D, Doncheva M, et al. "Annotation Disagreement as Signal Rather Than Noise: a empirical analysis". IEEE Trans Knowl Data Eng 32(2014): 212-221. DOI: 10.17219/acem/208596

Novotna E, Eriksdottir N, et al. "Annotation Disagreement as Signal Rather Than Noise: a ablation study". J Artif Intell Res 34(2017): 223-233. DOI: 10.7759/cureus.104086

Ostrowski L, Lindqvist O, et al. "Annotation Disagreement as Signal Rather Than Noise: a reproducibility study". ACM Comput Surv 36(2020): 234-245. DOI: 10.1038/s41746-023-00852-5

Petrescu S, Szabo P, et al. "Annotation Disagreement as Signal Rather Than Noise: a large-scale evaluation". Neural Comput Appl 38(2023): 245-257. DOI: 10.1016/j.mayocpiqo.2022.09.004

Quintero B, Balogh Q, et al. "Annotation Disagreement as Signal Rather Than Noise: a case study". Expert Syst Appl 40(2014): 256-269. DOI: 10.1177/08404704241291226

Ristic C, Castellani R, et al. "Annotation Disagreement as Signal Rather Than Noise: a survey". Inf Sci 42(2017): 267-281. DOI: 10.1093/medlaw/fwad013

Szabo J, Jankowiak S, et al. "Annotation Disagreement as Signal Rather Than Noise: a comparative analysis". Pattern Recognit 44(2020): 278-284. DOI: 10.2196/89834

Tarasov Q, Quintero T, et al. "Annotation Disagreement as Signal Rather Than Noise: a field deployment report". IEEE Access 46(2023): 289-296. DOI: 10.1007/s10916-024-02104-9

Urbanek Z, Zielinski U, et al. "Annotation Disagreement as Signal Rather Than Noise: a benchmark study". ACM Trans Softw Eng Methodol 8(2014): 300-308. DOI: 10.3389/fdgth.2026.1896184

Vasquez A, Alvez V, et al. "Annotation Disagreement as Signal Rather Than Noise: a systematic evaluation". Proc MLSys 10(2017): 311-320. DOI: 10.3390/diagnostics15162117

Wojcik H, Hendriksen W, et al. "Annotation Disagreement as Signal Rather Than Noise: a empirical analysis". Proc FAccT 12(2020): 322-332. DOI: 10.4103/jrpp.jrpp_20_65

Zielinski O, Ostrowski Z, et al. "Annotation Disagreement as Signal Rather Than Noise: a ablation study". Nat Mach Intell 14(2023): 333-344. DOI: 10.3389/fdgth.2026.1763884

Andric V, Vasquez A, et al. "Annotation Disagreement as Signal Rather Than Noise: a reproducibility study". J Mach Learn Res 16(2014): 344-356. DOI: 10.2196/48297

Balogh E, Emmerich B, et al. "Annotation Disagreement as Signal Rather Than Noise: a large-scale evaluation". IEEE Trans Pattern Anal Mach Intell 18(2017): 355-368. DOI: 10.1007/s00330-025-12031-z

Chiriac F, Farkas C, et al. "Annotation Disagreement as Signal Rather Than Noise: a case study". Proc NeurIPS 20(2020): 366-380. DOI: 10.3389/fdgth.2026.1699005

Doncheva M, Marchetti D, et al. "Annotation Disagreement as Signal Rather Than Noise: a survey". Proc ICML 22(2023): 377-383. DOI: 10.3389/fdgth.2026.1834800

Emmerich T, Tarasov E, et al. "Annotation Disagreement as Signal Rather Than Noise: a comparative analysis". Proc ICLR 24(2014): 388-395. DOI: 10.2196/69678

Foscarini C, Chiriac F, et al. "Annotation Disagreement as Signal Rather Than Noise: a field deployment report". ACM Trans Intell Syst 26(2017): 399-407. DOI: 10.7759/cureus.46454

Alvez D, Dumitrescu A, et al. "Annotation Disagreement as Signal Rather Than Noise: a benchmark study". Data Min Knowl Discov 28(2020): 410-419. DOI: 10.1016/j.ssci.2025.107028

Brankovic K, Kowalczyk B, et al. "Annotation Disagreement as Signal Rather Than Noise: a systematic evaluation". Artif Intell 30(2023): 421-431. DOI: 10.2196/48633

Castellani R, Ristic C, et al. "Annotation Disagreement as Signal Rather Than Noise: a empirical analysis". Knowl Based Syst 32(2014): 432-443. DOI: 10.1371/journal.pone.0305949

Dumitrescu A, Andric D, et al. "Annotation Disagreement as Signal Rather Than Noise: a ablation study". IEEE Trans Neural Netw Learn Syst 34(2017): 443-455. DOI: 10.1371/journal.pdig.0000514

Eriksdottir B, Brankovic E, et al. "Annotation Disagreement as Signal Rather Than Noise: a reproducibility study". Proc KDD 36(2020): 454-467. DOI: 10.17219/acem/208596

Farkas I, Ionescu F, et al. "Annotation Disagreement as Signal Rather Than Noise: a large-scale evaluation". Proc AAAI 38(2023): 465-479. DOI: 10.7759/cureus.104086

Grabowski P, Petrescu G, et al. "Annotation Disagreement as Signal Rather Than Noise: a case study". IEEE Trans Knowl Data Eng 40(2014): 476-482. DOI: 10.1038/s41746-023-00852-5

Hendriksen W, Wojcik H, et al. "Annotation Disagreement as Signal Rather Than Noise: a survey". J Artif Intell Res 42(2017): 487-494. DOI: 10.1016/j.mayocpiqo.2022.09.004

Ionescu F, Foscarini I, et al. "Annotation Disagreement as Signal Rather Than Noise: a comparative analysis". ACM Comput Surv 44(2020): 498-506. DOI: 10.1177/08404704241291226

Jankowiak G, Grabowski J, et al. "Annotation Disagreement as Signal Rather Than Noise: a field deployment report". Neural Comput Appl 46(2023): 509-518. DOI: 10.1093/medlaw/fwad013

Kowalczyk N, Novotna K, et al. "Annotation Disagreement as Signal Rather Than Noise: a benchmark study". Expert Syst Appl 8(2014): 520-530. DOI: 10.2196/89834

Lindqvist U, Urbanek L, et al. "Annotation Disagreement as Signal Rather Than Noise: a systematic evaluation". Inf Sci 10(2017): 531-542. DOI: 10.1007/s10916-024-02104-9

Marchetti D, Doncheva M, et al. "Annotation Disagreement as Signal Rather Than Noise: a empirical analysis". Pattern Recognit 12(2020): 542-554. DOI: 10.3389/fdgth.2026.1896184

Novotna E, Eriksdottir N, et al. "Annotation Disagreement as Signal Rather Than Noise: a ablation study". IEEE Access 14(2023): 553-566. DOI: 10.3390/diagnostics15162117

Ostrowski L, Lindqvist O, et al. "Annotation Disagreement as Signal Rather Than Noise: a reproducibility study". ACM Trans Softw Eng Methodol 16(2014): 564-578. DOI: 10.4103/jrpp.jrpp_20_65

Petrescu S, Szabo P, et al. "Annotation Disagreement as Signal Rather Than Noise: a large-scale evaluation". Proc MLSys 18(2017): 575-581. DOI: 10.3389/fdgth.2026.1763884

Quintero B, Balogh Q, et al. "Annotation Disagreement as Signal Rather Than Noise: a case study". Proc FAccT 20(2020): 586-593. DOI: 10.2196/48297

Ristic C, Castellani R, et al. "Annotation Disagreement as Signal Rather Than Noise: a survey". Nat Mach Intell 22(2023): 597-605. DOI: 10.1007/s00330-025-12031-z

Szabo J, Jankowiak S, et al. "Annotation Disagreement as Signal Rather Than Noise: a comparative analysis". J Mach Learn Res 24(2014): 608-617. DOI: 10.3389/fdgth.2026.1699005

Tarasov Q, Quintero T, et al. "Annotation Disagreement as Signal Rather Than Noise: a field deployment report". IEEE Trans Pattern Anal Mach Intell 26(2017): 619-629. DOI: 10.3389/fdgth.2026.1834800

Urbanek Z, Zielinski U, et al. "Annotation Disagreement as Signal Rather Than Noise: a benchmark study". Proc NeurIPS 28(2020): 630-641. DOI: 10.2196/69678

Vasquez A, Alvez V, et al. "Annotation Disagreement as Signal Rather Than Noise: a systematic evaluation". Proc ICML 30(2023): 641-653. DOI: 10.7759/cureus.46454

Wojcik H, Hendriksen W, et al. "Annotation Disagreement as Signal Rather Than Noise: a empirical analysis". Proc ICLR 32(2014): 652-665. DOI: 10.1016/j.ssci.2025.107028

Zielinski O, Ostrowski Z, et al. "Annotation Disagreement as Signal Rather Than Noise: a ablation study". ACM Trans Intell Syst 34(2017): 663-677. DOI: 10.2196/48633

Andric V, Vasquez A, et al. "Annotation Disagreement as Signal Rather Than Noise: a reproducibility study". Data Min Knowl Discov 36(2020): 674-680. DOI: 10.1371/journal.pone.0305949

Downloads

Published

2022-05-12

Issue

Vol. 1 No. 1 (2022): Vol 1 Issue 1 2022

Section

Articles

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.