Human review of AI email labels means a model proposes what an email is, a person decides whether that proposal changes anything, and the record keeps both. It exists because a label is cheap and its consequences are not. A wrong "interview" tells a recruiter and a candidate about a conversation that does not exist. This post shows what that review looks like from the reviewer's chair, and what it means for the numbers. Every count we publish says "found", and here is why.
Here is how callback detection works where it is switched on. It is rolling out with staffing teams first and is not part of self-serve accounts yet. So this describes a review screen used at staffing teams, and a self-serve user will not see it.
The AI only suggests
The AI only suggests. A person confirms each callback before the application changes. For each email the model returns one of six categories, a screening flag, any event time and deadline as written, a short evidence quote, the application it thinks matches, and the company and role named. It returns no certainty number, so nothing can quietly pass on a high one.
That is one of the available designs, and the differences are worth knowing whichever tool you use.
| Design | What moves your tracker | Example |
|---|---|---|
| Moves on confidence | The software, when it judges the match strong | Tsenta: statuses move "when the match is confident" |
| Suggests, and you approve | You, one by one or in bulk | Simplify: "Choose Approve to apply it, Decline to dismiss it, or Approve All" |
| Suggests, and a reviewer confirms | A recruiter or admin | ApplyEze, where callback detection is on |
Moving on confidence means nothing waits for a person. Our design puts a person between the label and the record, so a wrong label cannot change an application by itself. It does not email or text you. You see new callbacks when you open the page.
One thing happens without a person, and it is not an AI label. When a rule identifies a receipt and it matches one application decisively, an unconfirmed submission can become a confirmed one, with Undo.
What a reviewer sees
Reviewers are company admins and recruiters. The Callbacks page in the staffing portal has tabs for Needs review, Confirmed, Not callbacks and All, and a sidebar count of how many need review. For each callback, the reviewer gets:
- the company and role, the candidate, the applied date and who applied;
- why it matched;
- "What we read": the time, deadline and round the model pulled out;
- From, To, Subject and Received;
- the email as plain text, with no images or HTML. On Gmail inboxes there is an Open in Gmail button.
The point of that list is that the evidence sits beside the suggestion. A reviewer can read the sentence the label came from without leaving the page. Plain text has a side benefit: no remote image loads when a reviewer opens it.
The chart shows the whole path: the inbox is scanned, rules settle most mail, an AI labels what is left, it is tied to an application, and a person confirms the stage. This post is about the last step.
The four decisions
| Decision | What it does |
|---|---|
| Confirm | Moves the application to that stage. The model's original label is kept |
| Not a callback, or Duplicate | The card leaves the review list. The email text is hidden, not deleted |
| Wrong application | Pick another of the candidate's sent applications. The card moves there and is confirmed |
| Undo | For 10 seconds in a toast, and later from history. Restores what was there |
A card with no matched application reads "Not matched to an application yet", and the reviewer has to pick one before confirming. Our post on how a reply gets matched works through why that happens so often, with the math.
Correcting a stage or an application
A yes or no would not be enough, because the common error is a right idea with the wrong detail.
Sample, written for this post. An email from a recruiter says "Could you do a quick 15 minute chat on Thursday?" The model labels it interview. The reviewer reads it, changes the stage to screening, and confirms.
The kinds are defined in job application reply types, and the line between a screening and an interview takes judgment. So a reviewer can change the stage, time, deadline or round before confirming. Two details matter.
First, the model's label is kept next to the correction. That is what makes it possible to measure how often the two disagree.
Second, a confirm does not overwrite a reply with a later time already recorded on that application. An older email confirmed late cannot move someone backward.
Reminders and reschedules
Reminders and reschedules fold into the same card instead of making a new one. A folded email can update the card's time or deadline and mark it rescheduled, and the card shows "+N follow-ups folded in".
The reason is the reviewer's attention. A person asked to judge the same interview four times is likely to stop reading by the third, and a queue that trains people to click quickly is a bad queue. An email that matched nothing can still appear as a duplicate, and that is what the Duplicate option is for. A card someone dismissed still absorbs later mail of the same kind in its thread, without reopening.
What the research says about review screens
"A human checks it" is where most descriptions stop, and the design of the screen decides how much that check is worth. In a 2025 preprint, Beck, Eckman, Kern and Kreuter ran a randomized experiment with 2,784 annotators reviewing AI suggestions on emissions tables, and found:
- Annotators fixed plain errors such as spelling 82% of the time, and errors that needed a conceptual grasp of the rules only 31% of the time.
- People favorable to AI let more mistakes through. Those skeptical of it caught errors more reliably and scored higher on accuracy, and attitude was the strongest predictor of performance, ahead of demographics.
- Requiring corrections reduced engagement and raised acceptance of wrong suggestions, in the authors' reading, though overall accuracy was slightly higher in that group (68% against 66%), and their regression found it unchanged.
That is crowd work on tables, in an arXiv preprint, so it does not measure email review. It does show where to put the effort. Screening or interview is closer to the conceptual kind of error, and that kind is the one a tired reviewer is most likely to wave through, which is why the screen shows the email beside the label and lets a reviewer correct the stage.
The EU AI Act names the same risk. Article 14 says people overseeing high-risk systems should "remain aware of the possible tendency of automatically relying or over-relying on the output", and should be able to "disregard, override or reverse" it. We are not saying sorting job email falls under that law. It is a useful test for any review screen, and it is why the original label, the correction and Undo all exist here.
If you run any AI review queue, five habits follow from all this: show the source next to the label, make correcting as easy as accepting, keep the machine's answer, collapse repeats, and sample your reviewers' work now and then against a second reader.
Why our counts say found
This is the part that matters if you read numbers for a living. Here is what each one is.
| What we publish | What stands behind it | Did a person review it? |
|---|---|---|
| 801 replies: 465 assessments, 196 screenings, 140 interviews | Machine labels on email received 4 Oct to 8 Nov 2026 | No |
| 418 of those tied to one ApplyEze application | The same labels, plus the matching step | No |
| 336 human callbacks by day and hour | Screening plus interview from the same labels | No |
The chart shows that split of 801, across about 29 candidates at one staffing team. They were found when the scanner first ran on 3 November and read back to 4 October. No person reviewed them. So we say "found", and "confirmed" would be false. The scanner did not watch that window as it happened: most of it was read in hindsight. And because nobody checked, some of the 801 may be wrong in either direction.
That word is a small discipline, and it changes how much a number can be trusted. On the pages we read on 9 November 2026, Tsenta's pricing page, Simplify's email article and Wonsulting's tracker page describe reply tracking and give no count of replies and no match rate. That is not a fault in a product page, and they may publish more elsewhere. It does mean you should ask four things of any reply number from any tool, ours included:
- Who labelled it: a rule, a model or a person?
- Did anyone check the label after that?
- What is the denominator, and over what dates?
- Are automatic emails, such as test invitations and receipts, counted as replies?
Results for any one person will differ, and these counts are no promise of what any reader will get.
Who can read an email, and what is logged
A recruiter sees their assigned candidates, and a company admin sees the whole company. Each company sees only its own candidates, enforced in the database. Candidates at those teams have no such page.
- Each decision records who and when.
- Each read of an email's text is logged.
- The text a reviewer sees is not redacted, because they have to judge it. It is kept for 90 days.
- Function logs hold counts and codes only.
- Marking something "not a callback" hides the text and does not delete it. Undo shows it again.
The longer account of what is stored and what disconnecting removes is in connecting an inbox to a job tool, and callback detection has the full path.
On a self-serve account you see every application ApplyEze sends, and you record employer replies yourself on each one. That makes you the human in the loop, and your Heard back and Interviews counts come from what you record. You queue the roles in the desktop app, which runs on your computer while it is on and awake, and ApplyEze fills and submits each form. If you would rather be the reviewer than the form filler, your first 50 applications are free, no card needed.