AI Hiring Audit: How We Check a Judge With a Second Model

An AI hiring audit is a check on the decisions an AI makes about jobs and people, and the phrase covers two different tests. A bias audit, the kind New York City's Local Law 144 requires, asks whether outcomes differ across sex and race groups. An agreement audit asks whether a second reader, handed the same decision blind, lands in the same place. This post is about the second kind. On 20 October 2026, a separate, stronger model re-judged 601 decisions made by the AI judge in ApplyEze Autopilot, without seeing the verdict, and agreed on 502, or 83.5%. Below is how that worked, how to read a number like that, and a version any team can run on a tool it pays for, with the formulas.

Why a decision that sends applications needs checking

A judge's verdict decides whether an application goes out. In the 14 days to 9 November 2026 it scored 259,378 roles, and 50,609 reached 6 or higher, the default pass mark. No person can read that many, and the design does not ask anyone to: runs are applied or released, and never wait for a person.

So the decision has to be checked another way. A wrong yes sends an application under someone's name. A wrong no is quieter: the role just does not go out. Sampling those decisions and re-reading them blind is how you see either kind.

Autopilot is set up today for staffing and recruiting teams that work with ApplyEze. If you are job hunting on your own, you can start with 50 free applications on sign-up, no card needed. You pick the matches and queue them in the desktop app, and ApplyEze fills and submits each form. No judge decides for you on that path, so this audit does not describe it.

How the blind audit was run

1. Take a sample of decisions. The sample was 601 of the judge's decisions, including roles it passed and roles it turned down.
2. Hide the verdict. The checker did not see the judge's score or reason. That is what makes it blind. Show a reader the first answer and it can sway the second.
3. Use a separate reader. A different, stronger model judged each role against the person.
4. Count the matches. It agreed on 502 and disagreed on 99. So 502 / 601 = 83.5%, and 99 / 601 = 16.5%.
5. Date it. The result belongs to 20 October 2026, and to the judge as it stood that day. An audit is a dated photograph. The judge has changed since, including the years-of-experience lines in its brief, which went in on 4 November 2026.

The chart holds three separate checks. The first is this audit. The second asked a model of 193 applications on 26 October 2026 whether a recruiter would have sent each one: hand-picked applications scored 69% and Autopilot's scored 84%. The sample is small, and it is a model's reading, not a recruiter's; the post on that audit has the detail. The third checked 84 sampled resumes on 4 November 2026 and found 0 invented employers, dates, degrees or numbers. ApplyEze brings the posting's named skills and tools onto your skills line, even when your own record does not list them. There is no step to remove one before a live run, so use a dry run to see what it writes, and be ready to discuss anything on the page. The resume audit covers what it tested.

What 83.5% means

It means a stronger model, reading the same brief with no sight of the verdict, landed in the same place on 502 of 601 decisions. It is agreement, not accuracy, because the checker is not an answer key.

If the 601 were a random draw, sampling alone would put about 3 points of uncertainty either way on that figure (our arithmetic: 1.96 times the square root of 0.835 times 0.165 divided by 601).

Agreement depends on how lopsided the decisions are. This applies to any figure from any vendor, ours included, and it is the most useful thing to know before you read one. Picture two readers who never look at a role, and who each simply turn down a set share at random. They will still agree often, because they often both say no. Statisticians correct for this with Cohen's kappa, which compares observed agreement with the agreement chance would give: 0 is what chance alone produces, and 1 is perfect. Here is an illustrative 85% agreement at different mixes, assuming both readers turn down the same share (our arithmetic):

The same headline can mean very different things depending on the mix. So when a vendor quotes agreement, ask for the mix alongside it, and for how the yes decisions fared on their own. The worked example further down shows why.

What the disagreements could be

There were 99. Taken together they could be any blend of these, and they do not mean the same thing:

• The first reader was wrong. It missed something the checker saw.
• The checker was wrong. "Stronger" does not mean correct.
• The role sat on the line. A real 5 against a real 6 is close to a coin toss for any careful reader, and the bar turns it into a yes or a no.
• The brief was ambiguous. A generic title or a thin description can be read two ways, and neither reader is at fault.

Direction matters as well. A wrong pass sends an application, and a wrong turn-down loses one, and those call for different fixes. That is why the formulas below split the two.

Why this is not a recruiter panel

A model is checking a model. Two language models can share blind spots. Neither has talked to the hiring manager, or knows that one employer's "senior" is another's mid-level.

Agreement is not outcome. A model's approval and an employer's reply measure different things, and the second is the one you want. So we track it separately. At one staffing team, ApplyEze's callback rate rose from 1.5 per 1,000 applications in late September to 3.2 in late October, and human callbacks per candidate per month rose from about 2 in spring to 7 for October applications, with volume more than tripled. A human callback is a screening call, an interview or an offer. Part of the rise is better counting, and results for any one person will differ.

A panel is not a gold standard either. In a study by interviewing.io, 76 technical recruiters judged 30 engineers' resumes each. Two recruiters agreed on whether to interview a candidate 64% of the time, and they chose correctly, against the candidate's first interview result, 55% of the time. They said yes to 62% overall, and at that rate two readers guessing would agree about 53% of the time (our arithmetic). That is engineering only, from a company that sells interview practice, and it is a different task, so do not set 64% beside 83.5%. It shows what a panel can give you: human context, not a final answer. A panel is still worth running if you can afford one.

It is not a bias audit. This is our own check on our own judge. Under Local Law 144, a bias audit must calculate selection or scoring rates and the impact ratio across sex, race or ethnicity and their intersections, and the auditor must be independent of the employer and the vendor. An agreement audit answers a different question, and it says nothing about fairness between groups. What the judge is never shown covers that side.

Those legal audits have limits of their own, which is worth knowing if a vendor waves one at you. The same FAQ says the law "does not require any specific actions based on the results". A 2024 study checked 391 employers and found 18 audit reports and 13 transparency notices, though its authors warn that a missing report cannot be read as a violation. A December 2025 New York State Comptroller audit called enforcement "ineffective", and DLA Piper's summary reports the city had found one issue in 32 audits where the Comptroller's team found at least 17 potential ones.

How to run this test on any tool

You need an export of decisions with the inputs the tool saw, a spreadsheet and a second reader. A careful colleague works, and two are better.

1. Sample at random, split by verdict. Draw at least 100 yes decisions and 100 no decisions. If you sample in proportion, a tool that rejects most things gives you almost no yeses to check. 100 yeses gives you roughly 10 points of uncertainty on the share that hold up, and 300 gets you near 6.
2. Keep the inputs exact. Give your reader what the tool received. If you also want to test the input itself, repeat the exercise with the full posting and compare.
3. Hide the verdict and shuffle. The reader must not know the tool's answer, or which pile a decision came from.
4. Write the rule first. Agree on what counts as a yes before reading anything, in words. Otherwise you are measuring two different questions.
5. Fill in four counts, then apply these formulas.

A = both said yes
B = tool yes, reader no
C = tool no, reader yes
D = both said no
N = A + B + C + D

Agreement     = (A + D) / N
Chance        = ((A + B) / N) * ((A + C) / N) + ((C + D) / N) * ((B + D) / N)
Kappa         = (Agreement - Chance) / (1 - Chance)
Yes confirmed = A / (A + B)
No confirmed  = D / (C + D)

Here is a worked example. These numbers are invented to show the math. They are not from our audit. Out of 400 decisions, A = 60, B = 40, C = 30 and D = 270.

• Agreement = 330 / 400 = 82.5%, which sounds fine.
• Chance = 0.25 x 0.225 + 0.75 x 0.775 = 63.75%.
• Kappa = (0.825 - 0.6375) / 0.3625 = 0.52, which McHugh's scale calls weak.
• No confirmed = 270 / 300 = 90%.
• Yes confirmed = 60 / 100 = 60%.

That last line is why you run this. A tool can post 82.5% agreement while 4 of every 10 of its yes decisions fail a second read, and those are the decisions that act. If you drew more of one verdict than the tool really produces, weight the results back to its real mix before you quote an overall agreement rate. Yes confirmed and No confirmed do not need that correction.

Three more steps turn one number into a real check. Read every disagreement, and sort them into the four kinds above. Run the tool twice on the same inputs, because a tool that disagrees with itself has a problem that no second reader can diagnose. Repeat after each change to the model, the prompt or the inputs, since an old audit describes an old system. If you can get at outcomes, compare callbacks by verdict as well, knowing it takes far more data than you might expect. The pass mark post works through the sample sizes.

A job seeker can run a small version too. Take 30 roles a tool recommended, score them yourself in advance using a fixed rule, and count how many hold up. You will learn quickly whether the tool's idea of a fit resembles yours.

What to ask for, and what we publish

Whichever tool you are auditing, ask the vendor for these. Any vendor with a real audit can answer most of them in a sentence.

• the mix of passes and turn-downs in the sample
• how disagreements split between wrong passes and wrong turn-downs
• agreement by score and by field
• how much the tool varies between two runs of the same input
• any test of differences between groups
• who chose the reader, and who ran the check

ApplyEze publishes how many roles it turns down, a blind check on that decision, an audit of applications, an audit of resumes and completion by system. Across 259,378 roles, about 4 of every 5 on the shortlist were turned down, and a second model agreed on 502 of 601 decisions. On the rival pages we read on 9 November 2026, we did not find figures like these. That puts us ahead in what is shown, which is a different thing from results.

Those are team numbers, from a system that is not part of the free path. If you are one person, the same habit still applies: read what goes out in your name. A dry run fills the form and stops before Submit, every run leaves a receipt, and you pick and queue the roles yourself, on a computer that stays on and awake while they run, with an inbox connected for Workday and Oracle. Your first 50 applications are free, no card needed, so you can audit it yourself on real postings.