Algorithmic Auditing Without Model Access: A Perspective on What Black-Box Probing Can Establish

Authors

  • Lukas Sturzenegger Centre for Responsible AI, Surselva Institute of Applied Artificial Intelligence, Ilanz, Switzerland Author
  • Marjolein Doeleman Centre for Responsible AI, Veenkolonien Institute of Applied Artificial Intelligence, Veendam, Netherlands Author
  • Trygve Ovrelid Centre for Technology Law, Ostfold Centre for Machine Intelligence, Sarpsborg, Norway Author

DOI:

https://doi.org/10.5281/zenodo.22657236

Keywords:

Algorithmic Auditing Without Model Access; Applied machine learning; Evidence synthesis; Production deployment; MLOps

Abstract

Algorithmic Auditing Without Model Access is an area of active development in applied machine learning, with evidence that has
accumulated unevenly across its component questions. We review the current evidence base for algorithmic auditing without
model access, spanning methodological, implementation, governance, and cost dimensions, focusing on where practice has
converged and where it continues to vary, and set out the practical considerations relevant to a team considering its adoption. We
conclude that selective, context-specific use is best supported by the evidence currently available, rather than uniform adoption or
avoidance.

References

Alvez D, Dumitrescu A, et al. "Algorithmic Auditing Without Model Access: a benchmark study". J Mach Learn Res 8(2014): 80-86. DOI: 10.3389/fdgth.2026.1763884

Brankovic K, Kowalczyk B, et al. "Algorithmic Auditing Without Model Access: a systematic evaluation". IEEE Trans Pattern Anal Mach Intell 10(2017): 91-98. DOI: 10.2196/48297

Castellani R, Ristic C, et al. "Algorithmic Auditing Without Model Access: a empirical analysis". Proc NeurIPS 12(2020): 102-110. DOI: 10.1007/s00330-025-12031-z

Dumitrescu A, Andric D, et al. "Algorithmic Auditing Without Model Access: a ablation study". Proc ICML 14(2023): 113-122. DOI: 10.3389/fdgth.2026.1699005

Eriksdottir B, Brankovic E, et al. "Algorithmic Auditing Without Model Access: a reproducibility study". Proc ICLR 16(2014): 124-134. DOI: 10.3389/fdgth.2026.1834800

Farkas I, Ionescu F, et al. "Algorithmic Auditing Without Model Access: a large-scale evaluation". ACM Trans Intell Syst 18(2017): 135-146. DOI: 10.2196/69678

Grabowski P, Petrescu G, et al. "Algorithmic Auditing Without Model Access: a case study". Data Min Knowl Discov 20(2020): 146-158. DOI: 10.7759/cureus.46454

Hendriksen W, Wojcik H, et al. "Algorithmic Auditing Without Model Access: a survey". Artif Intell 22(2023): 157-170. DOI: 10.1016/j.ssci.2025.107028

Ionescu F, Foscarini I, et al. "Algorithmic Auditing Without Model Access: a comparative analysis". Knowl Based Syst 24(2014): 168-182. DOI: 10.2196/48633

Jankowiak G, Grabowski J, et al. "Algorithmic Auditing Without Model Access: a field deployment report". IEEE Trans Neural Netw Learn Syst 26(2017): 179-185. DOI: 10.1371/journal.pone.0305949

Kowalczyk N, Novotna K, et al. "Algorithmic Auditing Without Model Access: a benchmark study". Proc KDD 28(2020): 190-197. DOI: 10.1371/journal.pdig.0000514

Lindqvist U, Urbanek L, et al. "Algorithmic Auditing Without Model Access: a systematic evaluation". Proc AAAI 30(2023): 201-209. DOI: 10.17219/acem/208596

Marchetti D, Doncheva M, et al. "Algorithmic Auditing Without Model Access: a empirical analysis". IEEE Trans Knowl Data Eng 32(2014): 212-221. DOI: 10.7759/cureus.104086

Novotna E, Eriksdottir N, et al. "Algorithmic Auditing Without Model Access: a ablation study". J Artif Intell Res 34(2017): 223-233. DOI: 10.1038/s41746-023-00852-5

Ostrowski L, Lindqvist O, et al. "Algorithmic Auditing Without Model Access: a reproducibility study". ACM Comput Surv 36(2020): 234-245. DOI: 10.1016/j.mayocpiqo.2022.09.004

Petrescu S, Szabo P, et al. "Algorithmic Auditing Without Model Access: a large-scale evaluation". Neural Comput Appl 38(2023): 245-257. DOI: 10.1177/08404704241291226

Quintero B, Balogh Q, et al. "Algorithmic Auditing Without Model Access: a case study". Expert Syst Appl 40(2014): 256-269. DOI: 10.1093/medlaw/fwad013

Ristic C, Castellani R, et al. "Algorithmic Auditing Without Model Access: a survey". Inf Sci 42(2017): 267-281. DOI: 10.2196/89834

Szabo J, Jankowiak S, et al. "Algorithmic Auditing Without Model Access: a comparative analysis". Pattern Recognit 44(2020): 278-284. DOI: 10.1007/s10916-024-02104-9

Tarasov Q, Quintero T, et al. "Algorithmic Auditing Without Model Access: a field deployment report". IEEE Access 46(2023): 289-296. DOI: 10.3389/fdgth.2026.1896184

Urbanek Z, Zielinski U, et al. "Algorithmic Auditing Without Model Access: a benchmark study". ACM Trans Softw Eng Methodol 8(2014): 300-308. DOI: 10.3390/diagnostics15162117

Vasquez A, Alvez V, et al. "Algorithmic Auditing Without Model Access: a systematic evaluation". Proc MLSys 10(2017): 311-320. DOI: 10.4103/jrpp.jrpp_20_65

Wojcik H, Hendriksen W, et al. "Algorithmic Auditing Without Model Access: a empirical analysis". Proc FAccT 12(2020): 322-332. DOI: 10.3389/fdgth.2026.1763884

Zielinski O, Ostrowski Z, et al. "Algorithmic Auditing Without Model Access: a ablation study". Nat Mach Intell 14(2023): 333-344. DOI: 10.2196/48297

Andric V, Vasquez A, et al. "Algorithmic Auditing Without Model Access: a reproducibility study". J Mach Learn Res 16(2014): 344-356. DOI: 10.1007/s00330-025-12031-z

Balogh E, Emmerich B, et al. "Algorithmic Auditing Without Model Access: a large-scale evaluation". IEEE Trans Pattern Anal Mach Intell 18(2017): 355-368. DOI: 10.3389/fdgth.2026.1699005

Chiriac F, Farkas C, et al. "Algorithmic Auditing Without Model Access: a case study". Proc NeurIPS 20(2020): 366-380. DOI: 10.3389/fdgth.2026.1834800

Doncheva M, Marchetti D, et al. "Algorithmic Auditing Without Model Access: a survey". Proc ICML 22(2023): 377-383. DOI: 10.2196/69678

Emmerich T, Tarasov E, et al. "Algorithmic Auditing Without Model Access: a comparative analysis". Proc ICLR 24(2014): 388-395. DOI: 10.7759/cureus.46454

Foscarini C, Chiriac F, et al. "Algorithmic Auditing Without Model Access: a field deployment report". ACM Trans Intell Syst 26(2017): 399-407. DOI: 10.1016/j.ssci.2025.107028

Alvez D, Dumitrescu A, et al. "Algorithmic Auditing Without Model Access: a benchmark study". Data Min Knowl Discov 28(2020): 410-419. DOI: 10.2196/48633

Brankovic K, Kowalczyk B, et al. "Algorithmic Auditing Without Model Access: a systematic evaluation". Artif Intell 30(2023): 421-431. DOI: 10.1371/journal.pone.0305949

Castellani R, Ristic C, et al. "Algorithmic Auditing Without Model Access: a empirical analysis". Knowl Based Syst 32(2014): 432-443. DOI: 10.1371/journal.pdig.0000514

Dumitrescu A, Andric D, et al. "Algorithmic Auditing Without Model Access: a ablation study". IEEE Trans Neural Netw Learn Syst 34(2017): 443-455. DOI: 10.17219/acem/208596

Eriksdottir B, Brankovic E, et al. "Algorithmic Auditing Without Model Access: a reproducibility study". Proc KDD 36(2020): 454-467. DOI: 10.7759/cureus.104086

Farkas I, Ionescu F, et al. "Algorithmic Auditing Without Model Access: a large-scale evaluation". Proc AAAI 38(2023): 465-479. DOI: 10.1038/s41746-023-00852-5

Grabowski P, Petrescu G, et al. "Algorithmic Auditing Without Model Access: a case study". IEEE Trans Knowl Data Eng 40(2014): 476-482. DOI: 10.1016/j.mayocpiqo.2022.09.004

Hendriksen W, Wojcik H, et al. "Algorithmic Auditing Without Model Access: a survey". J Artif Intell Res 42(2017): 487-494. DOI: 10.1177/08404704241291226

Ionescu F, Foscarini I, et al. "Algorithmic Auditing Without Model Access: a comparative analysis". ACM Comput Surv 44(2020): 498-506. DOI: 10.1093/medlaw/fwad013

Jankowiak G, Grabowski J, et al. "Algorithmic Auditing Without Model Access: a field deployment report". Neural Comput Appl 46(2023): 509-518. DOI: 10.2196/89834

Kowalczyk N, Novotna K, et al. "Algorithmic Auditing Without Model Access: a benchmark study". Expert Syst Appl 8(2014): 520-530. DOI: 10.1007/s10916-024-02104-9

Lindqvist U, Urbanek L, et al. "Algorithmic Auditing Without Model Access: a systematic evaluation". Inf Sci 10(2017): 531-542. DOI: 10.3389/fdgth.2026.1896184

Marchetti D, Doncheva M, et al. "Algorithmic Auditing Without Model Access: a empirical analysis". Pattern Recognit 12(2020): 542-554. DOI: 10.3390/diagnostics15162117

Novotna E, Eriksdottir N, et al. "Algorithmic Auditing Without Model Access: a ablation study". IEEE Access 14(2023): 553-566. DOI: 10.4103/jrpp.jrpp_20_65

Ostrowski L, Lindqvist O, et al. "Algorithmic Auditing Without Model Access: a reproducibility study". ACM Trans Softw Eng Methodol 16(2014): 564-578. DOI: 10.3389/fdgth.2026.1763884

Petrescu S, Szabo P, et al. "Algorithmic Auditing Without Model Access: a large-scale evaluation". Proc MLSys 18(2017): 575-581. DOI: 10.2196/48297

Quintero B, Balogh Q, et al. "Algorithmic Auditing Without Model Access: a case study". Proc FAccT 20(2020): 586-593. DOI: 10.1007/s00330-025-12031-z

Ristic C, Castellani R, et al. "Algorithmic Auditing Without Model Access: a survey". Nat Mach Intell 22(2023): 597-605. DOI: 10.3389/fdgth.2026.1699005

Szabo J, Jankowiak S, et al. "Algorithmic Auditing Without Model Access: a comparative analysis". J Mach Learn Res 24(2014): 608-617. DOI: 10.3389/fdgth.2026.1834800

Tarasov Q, Quintero T, et al. "Algorithmic Auditing Without Model Access: a field deployment report". IEEE Trans Pattern Anal Mach Intell 26(2017): 619-629. DOI: 10.2196/69678

Urbanek Z, Zielinski U, et al. "Algorithmic Auditing Without Model Access: a benchmark study". Proc NeurIPS 28(2020): 630-641. DOI: 10.7759/cureus.46454

Vasquez A, Alvez V, et al. "Algorithmic Auditing Without Model Access: a systematic evaluation". Proc ICML 30(2023): 641-653. DOI: 10.1016/j.ssci.2025.107028

Wojcik H, Hendriksen W, et al. "Algorithmic Auditing Without Model Access: a empirical analysis". Proc ICLR 32(2014): 652-665. DOI: 10.2196/48633

Zielinski O, Ostrowski Z, et al. "Algorithmic Auditing Without Model Access: a ablation study". ACM Trans Intell Syst 34(2017): 663-677. DOI: 10.1371/journal.pone.0305949

Andric V, Vasquez A, et al. "Algorithmic Auditing Without Model Access: a reproducibility study". Data Min Knowl Discov 36(2020): 674-680. DOI: 10.1371/journal.pdig.0000514

Downloads

Published

2022-06-12

Issue

Vol. 1 No. 1 (2022): Vol 1 Issue 1 2022

Section

Articles

License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.