A recruiter opens the dashboard on Monday morning.
There are 642 new applications.
Some candidates are clearly qualified. Some are not. Some have unusual career paths. Some have strong skills but imperfect resumes. Hiring managers want shortlists by the end of the day.
Then the vendor demo looks tempting.
An AI applicant tracking system promises faster screening, smarter matching, automatic ranking, better candidate communication, and fewer manual tasks.
That sounds useful.
It can be useful.
But an AI ATS is not just another HR tool. It can influence who gets seen, who gets shortlisted, and who quietly disappears from the process.
That makes the buying decision bigger than software.
It is also a decision about fairness, compliance, data protection, candidate trust, and hiring quality.
Before signing a contract, ask these 10 questions.
What Is an AI ATS?
An AI ATS is an applicant tracking system that uses artificial intelligence, machine learning, natural language processing, automation, or predictive analytics to support recruitment workflows.
A traditional ATS stores applications, tracks hiring stages, manages job postings, and keeps candidate records organized.
An AI ATS may go further. It may parse resumes, recommend candidates, rank applicants, generate interview questions, summarize notes, identify skills, or automate communication.
The important question is not whether the system uses AI.
The important question is where AI affects the hiring decision.
If the AI only schedules interviews, the risk is lower. If it ranks candidates or recommends rejection, the risk is much higher.
The U.S. Equal Employment Opportunity Commission has made clear that employers can still face discrimination risks when AI tools are used in employment decisions. Its guidance on AI and employment discrimination explains that existing federal employment laws continue to apply when automated systems are involved.
In the EU, employment-related AI systems can fall into the high-risk category under the AI Act, with obligations around governance, transparency, human oversight, accuracy, and risk management. The European Commission’s AI Act implementation timeline shows that major rules, including many high-risk AI obligations, are being phased in across 2026 and 2027.
So evaluation should go beyond features.
It should test accountability.
1. What hiring problem are we trying to solve?
Start with the problem, not the product.
“We need AI for recruitment” is not a strategy.
A better statement sounds like this:
“We need to reduce manual screening time for high-volume roles while preserving recruiter review, candidate transparency, and compliance reporting.”
That gives the buying team something to test.
Different hiring problems require different tools. A company struggling with scheduling may not need AI candidate ranking. A staffing agency managing thousands of applicants may need automated matching, but only with strong audit controls. A global employer may care more about country-specific workflows, documentation, and onboarding handoffs.
Before the demo, define the business need.
Is the goal speed, quality, consistency, compliance, candidate experience, or cost reduction?
If the vendor cannot connect its AI features to that goal, the tool may create more complexity than value.
2. Which hiring stages does the AI influence?
Ask the vendor to map every AI feature by hiring stage.
- Does the AI help with sourcing?
- Does it screen resumes?
- Does it rank applicants?
- Does it recommend interview questions?
- Does it summarize interviews?
- Does it reject candidates automatically?
This distinction matters.
AI used for administrative support is different from AI used for selection.
A useful evaluation table may look like this:
| AI Feature |
Risk Level |
Why It Matters |
|
Interview scheduling |
Low | Mostly administrative |
|
Resume parsing |
Medium | Errors can affect candidate records |
|
Candidate matching |
High | May shape who recruiters review |
|
Automated rejection |
High |
May exclude qualified candidates |
|
Interview scoring |
High | May influence selection decisions |
|
Video or speech analysis |
Very high | Can raise privacy, bias, and accessibility concerns |
The buyer should ask one simple question:
Does the AI assist a human, or replace human judgment?
A responsible system should help recruiters work better. It should not hide important decisions behind automation.
3. Can the vendor explain how recommendations are made?
Recruiters do not need to understand every technical detail.
They do need a practical explanation.
If the system ranks one candidate above another, why?
Is it based on skills, keywords, experience, certifications, assessments, historical hiring data, or a mix of factors?
Can recruiters see the reason for a recommendation?
Can they override it?
Is the override logged?
Be careful with vague claims such as:
“Our AI finds the best candidates.”
“The system learns from your hiring history.”
“Our algorithm is proprietary, so we cannot explain it.”
That last one is especially important. A vendor can protect intellectual property and still provide meaningful transparency.
If a system learns from historical hiring patterns, it may also learn from historical bias. If previous hires came mostly from certain universities, locations, industries, or demographic groups, the model may treat those patterns as signals of quality.
Good AI ATS vendors should be able to explain what the tool does, what it does not do, what data it uses, and what its limitations are.
4. How does the vendor test for bias and adverse impact?
Do not accept “our AI is unbiased” as an answer.
No serious vendor should say that.
Bias can enter through training data, job descriptions, resume parsing, skills inference, location filters, education requirements, employment gap assumptions, or assessment design.
Ask for evidence.
Useful documentation may include bias audit reports, adverse impact testing, validation summaries, accessibility testing, data quality reviews, model documentation, and remediation procedures.
The testing should also match your use case.
A bias audit for one country or role type does not automatically prove the system is safe for every job in every market.
New York City’s automated employment decision tools law gives a practical example of where regulation is heading. The city’s AEDT rules require covered employers and employment agencies to meet bias audit and notice requirements before using certain automated tools in hiring or promotion.
Even when that specific law does not apply, the principle is useful.
If AI influences hiring, fairness needs to be tested, documented, and monitored.
5. What candidate data does the system use?
Recruitment data is sensitive.
An ATS may process resumes, contact details, work history, salary expectations, interview notes, assessment results, references, location, immigration information, and accommodation requests.
An AI ATS may also create inferred data, such as skills predictions, match scores, seniority estimates, or likelihood-to-accept indicators.
Ask for a clear data map.
What data is collected?
What data is inferred?
Is sensitive data used in scoring?
Is customer data used to train vendor models?
Can you opt out of model training?
Where is data stored?
Who are the subprocessors?
How long is data retained?
Can candidates request deletion or correction?
For UK and EU-related hiring, privacy issues are especially important. The UK Information Commissioner’s Office has reviewed AI tools used in recruitment and warned that these tools can create risks for privacy and information rights, even when they offer operational benefits.
The safest buying approach is simple.
Know what data goes in.
Know what the AI does with it.
Know who can access it.
Know how it can be deleted.
6. Does the system preserve human oversight?
AI should support recruiters, not remove accountability.
Look for features that keep humans in control:
|
Oversight Feature |
Why It Matters |
|
Recruiter review before rejection |
Prevents blind automation |
|
Score explanations |
Helps recruiters understand recommendations |
|
Override controls |
Allows human judgment |
|
Audit logs |
Supports compliance and review |
| Configurable workflows | Adapts to different countries and roles |
|
Escalation paths |
Helps with edge cases and accommodations |
The system should also make it possible to pause automation.
That may sound inefficient, but responsible hiring sometimes needs friction.
A candidate may have a career gap due to caregiving, migration, military service, illness, study, or informal work. A rigid model may treat that as a weakness. A human recruiter may understand the context.
A good AI ATS follows this pattern:
AI organizes.
AI suggests.
AI explains.
Humans decide.
7. Does the ATS fit real hiring workflows?
A demo is clean.
Real hiring is not.
Before buying, test the system against real scenarios:
- High-volume hiring
- Executive search
- Contractor onboarding
- International remote roles
- Internal mobility
- Campus recruitment
- Staffing agency submissions
- Employer of Record workflows
- Multi-language applications
- Referral candidates
- Rehires
- Roles with licensing requirements
- Roles requiring right-to-work checks
For global organizations, this is critical.
The ATS may need to support different worker types, including employees, contractors, freelancers, agency workers, and Employer of Record hires. It may also need to integrate with onboarding, compliance documentation, payroll, workforce management, and payment systems.
This is where platforms such as TFY can become relevant to the broader workforce stack. An AI ATS may help identify candidates faster, but the business still needs to manage classification, onboarding, compliance, contracts, payroll, and cross-border workforce administration.
Recruitment does not end at selection.
The technology should support what happens next.
8. What candidate experience does the AI create?
Candidates experience hiring technology before they experience your company culture.
A poor AI ATS can make people feel ignored, filtered, or confused.
A good one should make the process clearer.
Ask what candidates will see.
Are they told when AI is used?
Can they correct parsed resume data?
Can they request accommodation?
Can they contact a human?
Are chatbot answers reviewed?
Are rejection messages respectful?
Is the application mobile-friendly?
Does the system support accessibility and multiple languages?
Candidate experience matters for employer brand, but it also matters for fairness.
If the AI misreads a resume, candidates should have a way to correct it. If a chatbot gives incorrect information, there should be human support. If an assessment creates accessibility barriers, there should be an alternative process.
Automation should not trap people inside a process they cannot question.
9. What compliance obligations apply in our markets?
AI hiring rules are changing quickly.
In the United States, federal anti-discrimination laws still apply when AI is used in employment decisions. The EEOC’s role in AI enforcement focuses on whether technology contributes to unlawful discrimination in hiring, promotion, termination, or other employment practices.
In New York City, covered AEDTs require bias audits and candidate notices. In the EU, the AI Act creates obligations for high-risk systems, including many employment-related uses. In the UK, data protection rules apply to recruitment data and automated processing.
Ask vendors:
- Which jurisdictions do you support?
- Can we configure notices by country?
- Can we disable AI features in certain markets?
- Can we export audit logs?
- Can we manage retention schedules by location?
- Do you provide documentation for legal review?
- Do you notify customers when AI models materially change?
Do not rely on “fully compliant” marketing language.
Compliance depends on the tool, the configuration, the market, and how the employer uses it.
10. How will we measure ROI and risk together?
Many AI ATS vendors sell speed.
Speed matters, but it is not enough.
A hiring process can become faster and still become worse.
Measure three categories.
Efficiency
Track time to shortlist, time to interview, time to hire, recruiter hours saved, scheduling time reduced, and cost per hire.
Quality
Track interview-to-offer ratio, offer acceptance rate, hiring manager satisfaction, candidate satisfaction, new hire retention, and skills match accuracy.
Risk
Track adverse impact indicators where legally permitted, candidate complaints, override rates, automated rejection rates, parsing errors, audit findings, and data deletion response times.
The best question is not:
Did the AI make hiring faster?
The better question is:
Did it help us hire faster, fairer, and with more control?
A Simple AI ATS Buying Scorecard
Use this scorecard when comparing vendors.
|
Evaluation Area |
Question | Score |
|
Business fit |
Does it solve a clear hiring problem? | 1 to 5 |
|
Transparency |
Can the vendor explain recommendations? | 1 to 5 |
|
Bias testing |
Is fairness tested and documented? | 1 to 5 |
|
Human oversight |
Can recruiters review and override? | 1 to 5 |
|
Data protection |
Is candidate data handled responsibly? | 1 to 5 |
|
Compliance |
Does it support relevant market rules? | 1 to 5 |
|
Workflow fit |
Does it match real hiring scenarios? | 1 to 5 |
|
Integrations |
Does it connect with HR, payroll, EOR, and onboarding tools? | 1 to 5 |
|
Candidate experience |
Is the process clear and accessible? | 1 to 5 |
|
ROI |
Does value include quality and risk, not just speed? |
1 to 5 |
A vendor with strong features but weak transparency, auditability, or privacy controls should not move forward without deeper review.
Final Thoughts
An AI ATS can be a valuable tool.
It can reduce manual work, improve consistency, speed up shortlisting, and help recruiters focus on better candidate conversations.
But it can also create risk if it is opaque, poorly tested, badly configured, or used without human oversight.
The right question is not whether the system has AI.
The right question is whether your organization can understand it, control it, monitor it, and explain it.
Before buying, ask the hard questions.
- What problem are we solving?
- Where does AI influence hiring?
- How are recommendations made?
- How is bias tested?
- What data is used?
- Can humans override the system?
- Does it fit our real workflows?
- What does the candidate experience look like?
- Which laws apply?
- How will we measure ROI and risk together?
AI should not make hiring less human.
Used well, it should help hiring teams make better, faster, fairer, and more accountable decisions.
