Every HR leader evaluating an AI hiring tool has heard some version of the same reassurance: “our model has been tested for bias.” It’s become a standard line in vendor pitches, backed by fairness benchmarks and compliance checklists. But there’s new research suggesting that reassurance may be answering the wrong question.
A study out of Princeton and the University of Chicago tested how large language models — including popular tools like ChatGPT, Claude, and Gemini — behave when asked to make repeated hiring decisions over time. The experiment was designed to strip out any real differences between candidate groups; every group-job pairing came with the same independently-sampled 90% probability of success, so there were no actual gaps to “discriminate” on.
The models found a way anyway. Across repeated rounds, they began sorting fictional demographic groups into different categories of jobs despite having no on-paper signal to justify it. The researchers connected this to a familiar pattern: an early, essentially random success or failure nudges the model toward repeating a group-job pairing that seemed to work, rather than trying out alternatives. It’s the same dynamic behavioral scientists have documented in people. The difference is that this bias isn’t imported from distinct training data or historical hiring records that carry old human biases with them. It’s manufactured fresh, straight from the model’s own decision-making history.
That distinction matters more than it might first appear. Most fairness testing that vendors point to measure whether a model reproduces existing bias. This study measured whether a model builds a new one, from scratch, simply by making decisions over time. A tool can pass every standard benchmark and still develop this kind of drift once it’s actually deployed and building its own history.
The fixes that didn’t work (and the one that did)
The researchers tried some obvious interventions to break the pattern. Asking the model to reason step-by-step before making a final call barely moved the needle. Neither did increasing randomness in its outputs, shortening how much decision history it could reference, or removing incentive structures from the task.
What did work was a more deliberate direction. Researchers tried giving the model an explicit, measurable diversity objective, rather than a general instruction to “be fair.” And it worked — to some extent. When they reran the test with groups that genuinely did perform differently, forcing the same diversity objective actually lowered overall outcomes. In other words, this intervention only helps when the underlying population truly is equivalent — and determining that requires a human judgment call.
What this means for hiring decisions
This was a controlled, synthetic study, not a live tool deployment, and its authors have been clear that the mechanism itself, rather than any specific numbers, is the finding we should be paying attention to. But the implication is hard to ignore.
As hiring tools move from relatively simple “scorecards” toward agentic systems that learn and adapt across many decisions, the opportunity for this kind of self-generated bias grows too. For HR leaders, that means we need to be rethinking the conversation when we’re looking at AI systems to integrate with our hiring.
The right question isn’t only “has this been tested for bias.” That doesn’t take the full measure of a system. We need to ask something more like, “How does this system behave over hundreds of decisions, and does it optimize for measurable, equitable outcomes, or just a general instruction to be fair?”
It also reinforces something we’ve said consistently as AI takes on more of the hiring workflow. These systems still need experienced human judgment in the loop, not as a compliance formality, but because determining when candidates truly are equivalent is a call no model can make for itself. Whether you’re handling it internally or working with a trusted search partner like Blue Rock Human Capital, keeping a people-first perspective can help your organization remain a trusted place where people want to work and thrive.