How to Run a Fair Trial of an AI Screening Tool on a Closed Role

A fair trial of an AI screening tool uses a role you have already filled, criteria frozen before the run, and consultant scores the tool never sees. Judge it on four things: where the placed candidate lands, what it does to the rejected pile, whether a second run agrees, and whether a changed name changes the ranking.

A live role cannot be graded, because nobody knows yet who was right. A closed role can: you know who was placed, who the client turned down and how the placement went. For an agency, that is the cheapest honest test of a screening tool, and it costs one afternoon of a consultant's time.

This page gives ten questions to settle before you load a single CV, each with what a good answer sounds like and what a red flag sounds like.

Why do most AI screening trials tell you nothing?

They run on a live role, with criteria nobody wrote down, and judge the tool by whether the top five look right. A shortlist that looks right can still be hiding the people you needed. A Harvard Business School and Accenture survey of 2,275 executives in the US, UK and Germany, fielded in early 2020, found that 88% of employers agreed qualified high-skills candidates are vetted out because they do not match the exact criteria in the job description. The figure was 94% for middle-skills roles. That survey is older than a year, but the mechanism it describes is the one a trial has to go looking for.

When we looked for guidance on piloting a screening tool in November 2026, the vendor guide we read in full, Manatal's September 2026 piece on AI resume screening, was a buying checklist of features, bias controls and integrations. It recommends audits and a human second look at borderline candidates. It has no trial protocol, and nothing for an agency that needs to show a client why a shortlist can be trusted. The ten questions below are that protocol.

Which questions set the trial up before anything runs?

Choose the role, write the criteria and decide how scoring stays blind, all before the tool sees a CV. Each choice you make after seeing results is a choice the tool's output has influenced.

Which closed role should we use?

Good answer: a role you filled in the last six to twelve months, with the full applicant pool still on file and a known outcome: who was placed, who the client rejected, and how long the placement lasted. Pick a role of the kind you place regularly, and pick it before you look at any results. Red flag: choosing the role that went well, or letting the vendor choose.

Who writes the criteria, and when?

Good answer: the criteria and the evidence that counts as strong, written down and dated before the run, and the same text given to the tool and to your consultants. Red flag: adjusting the criteria after you see the ranking. A tool should also push back on a vague criterion rather than rank against it; ask what it does with "strong communicator".

How do we keep the scoring blind?

Good answer: two consultants score the same CVs independently and write their scores down before anyone sees the tool's ranking, and the tool is never told who was placed or who the client rejected. Two scorers also show you how much your own people disagree, which is the yardstick the tool should be held to. Red flag: a reviewer who has seen the ranking first, because that reviewer is no longer an independent check.

Which questions test what the tool actually produces?

Ask where the placed candidate lands, what happens to the people your team turned away, whether the tool agrees with itself, and whether it catches a planted problem. These four tests read the output directly, so none of them depends on how well the demo went.

Where does the placed candidate land?

Good answer: the placed candidate sits in the tool's top ten of the pool, with the evidence for that position shown beside the name. One placement is one data point, so if two or three closed roles are available, run them all. Red flag: judging the tool on a single hit or a single miss, or on a ranking with no reasons attached.

What does it do to the candidates we rejected?

Good answer: you pull ten candidates from your own reject pile, including any the client turned down, and see where the tool puts them. Then you read every case where it ranks a rejected person high and decide, with the CV in front of you, who was right. That is the test for the false negatives the Harvard Business School survey describes. Red flag: checking only the top of the list, or a tool that removes candidates so you never see the bottom of the pool.

Does a second run agree with the first?

Good answer: you run the same role twice and compare the two top tens. Decide the overlap you will accept, and write it down before the first run. Any difference should be explainable from the evidence. Red flag: a top ten that shifts between runs with no reason you can read, because a score that changes on its own is a score you cannot defend to a client.

Does it catch a claim we planted?

Good answer: you add two or three test CVs with a problem you put there, such as overlapping tenures or a team size that does not fit the title, and the tool flags each one to a person with the reason attached. For how the same checks work before a client submission, see how staffing agencies verify candidate claims. Red flag: silence, or an automatic rejection. A consistency check is also not background verification, and neither replaces the formal check after an offer.

Which questions test fairness, data and cost?

Run a name swap, settle what happens to the CVs you upload, and agree the cost and the stopping rule in writing. These are the three things agencies tend to find out after signing.

Does the ranking change when only the name changes?

Good answer: you take ten to twenty CVs, run them once as they are and once with the names swapped for names from a different region or community that is common in your pool, and the top-ten membership does not move. A University of Washington study from 2024 ran more than three million comparisons across over 550 resumes and three language models, and found white-associated names preferred 85% of the time against 9% for Black-associated names. It used US names and research models, so the result does not transfer to India. The test does. Red flag: a vendor that cannot tell you whether it has run this test itself.

The US rule of thumb for comparing groups is the four-fifths rule: a selection rate below 80% of the highest group's rate is generally treated as evidence of adverse impact, and the same regulation says small numbers may not be statistically significant. A trial of a few dozen CVs is small. You may not hold group data on candidates at all, and a trial is a poor reason to start collecting it, which is why the name swap needs none. What a formal audit checks is covered in what NYC Local Law 144 actually checks.

What happens to the CVs we upload?

Good answer: a written statement of where the files are stored, whether they are used to train anything, and deletion at the end of the trial, confirmed in writing. Old CVs are personal data collected for a placement, so check whether your candidate consent notice covers use in a tool evaluation, and remove names and contact details where the tool does not need them. This is not legal advice; our DPDP explainer sets out how the Act treats candidate data. Red flag: "we are compliant" with no document behind it.

What does the trial cost, and who ends it?

Good answer: a trial with a written scope, a spend cap and an end date, and a rule you wrote down in advance for what counts as a pass. Convert any price into the cost of one role at your own volume; how much AI interviewing costs in India shows the arithmetic. Red flag: a pilot offered only after an annual contract is signed.

Where do agencies usually get a trial wrong?

They judge the tool on one placement, let the vendor pick the role, and treat a flag as proof. Each mistake is the same one: the trial was designed after the result was known.

A second mistake is stopping at the top ten. The tool also earns its place by what it does to the rest of the pool, and the reject pile is where you find out. The third is skipping the stopping rule. If you have not written down what a pass looks like, every result will look like a pass to someone who wants one. The companion list for interview tools is questions to ask before buying an AI interview tool in India. If your real question is whether a general chatbot will do instead, read whether a staffing agency can just use ChatGPT to screen candidates.

How AgentR answers this checklist, and where its answers are weak

AgentR reads every application in full, checks what candidates claim against the public record, and runs structured interviews scored against a rubric written before anyone applied. The team gets a ranked shortlist with the evidence attached, nobody is auto-rejected, and every decision stays with a person. AgentR is in private beta, so there are no customer references and no outcome data to offer. Here is how it answers the ten questions, using what its public pages say as of November 2026.

• Closed role: the first role is free, up to 50 applicants reviewed and 5 interviews, with no card. A closed role with a bigger pool needs a sample, so decide the rule before you run: include the placed candidate and the ones the client rejected, fill the rest at random, and write the rule down. The pages we read do not say how old CVs are loaded, so ask before you start.
• Criteria first: the shortlisting page says criteria that are too vague are sent back before anything is ranked, and that your criteria carry 45% of the score, with 40% from 20 scored signals and 15% from the claim check.
• Blind scoring: that is your process. The tool cannot enforce it.
• Placed candidate and reject pile: the shortlist is an ordering, not a filter, so the whole pool stays visible with scores. That makes the reject-pile test possible.
• Repeat run: weak. We have no published run-to-run agreement figure, so run it and tell us what you find.
• Planted claim: verification checks claims for consistency against public and organisational signals, sends flags to a person, and does not penalise a thin public footprint. It does not contact employers or institutions and is not background verification.
• Name swap: weak. We have no bias-audit or accuracy figure to quote here, so run the swap yourself.
• Uploaded CVs: the privacy policy and platform security pages exist, but the pricing page carries no retention or deletion terms. Get them in writing.
• Cost and stopping rule: after the free role, the pricing page lists ₹25 per applicant reviewed and ₹10 per interview minute, before GST, with an estimate before a role goes live and a spend cap per role. A plan at ₹11,999 a month buys ₹12,000 of usage, and unused paid balance can be refunded on request. See pricing for the current figures.

Pick one role you have already filled and run it through AgentR. The first role is on us.

See how the shortlist is scored and ranked on the better shortlisting page.