> Agent view (Markdown) of https://www.nycbiasaudit.com/resources/llm-resume-screening-bias-testing
> Publisher: Holistic AI (https://www.holisticai.com). Site index for agents: https://www.nycbiasaudit.com/llms.txt · Everything in one file: https://www.nycbiasaudit.com/llms-full.txt

# Testing LLM resume screeners for bias: counterfactual audit design for generative AI in hiring

Research paper · 18 pages · Published 2026-10-21 · By Holistic AI

Published audits show that LLM screeners react to candidate names, to the order resumes appear in a prompt, and to how the prompt is worded. This paper sets out a counterfactual test design and a reproducible harness that separate those effects and report them in the format NYC Local Law 144 expects. A worked example on a synthetic screener shows the harness recovering injected bias and exposing how careless test designs overstate or understate it.

## Key numbers

- **63.5%**: Pairwise choices that went to the first listed candidate across 22 LLMs (Rozado, 2025)
- **85.1%**: Race tests in which embedding retrievers preferred White associated names (Wilson and Caliskan, 2024)
- **361k**: Resumes scored by five LLMs in one intersectional study (An et al., 2025)
- **6**: Original charts and diagrams, including a test harness architecture

## What is inside

- A review of five published LLM hiring bias studies and what they mean for audits
- When an LLM screener counts as an AEDT under NYC Local Law 144
- Counterfactual design rules: signal injection, order counterbalancing, repeats and prompt grids
- A reproducible test harness with architecture and pseudocode
- A worked example on a synthetic screener, plus mitigation options and checklists

## Who it is for

- ML engineers and HR tech product teams
- AI governance and risk leads
- Bias auditors and employment counsel

The full PDF is free. Request it with the form at https://www.nycbiasaudit.com/resources/llm-resume-screening-bias-testing.

---

Holistic AI carries out independent Local Law 144 bias audits. What the service includes: https://www.nycbiasaudit.com/services.md. To commission a bias audit or ask a question, book a call at https://www.nycbiasaudit.com/demo or email we@holisticai.com.