What 5 Chief People Officers Really Think About AI Interviews: Honest Feedback from Senior HR Leaders

We used our AI interviewer to screen candidates for a Chief People Officer role. Then we asked them all for brutally honest feedback. Here's what 5 senior HR leaders told us—the good, the bad, and what we're fixing.

We recently did something that felt risky.

We used our AI agents to screen candidates for a Chief People Officer position at a major firm in Dubai. These weren't junior recruiters or entry-level coordinators—these were VPs of HR, Group Heads of Talent, and seasoned HR leaders with 20+ years of experience across multinational organizations.

Then we did something even riskier:

We asked every single one of them for honest feedback about being interviewed by AI.

Five responded. Not with generic pleasantries, but with detailed, thoughtful critiques that were sometimes uncomfortable to read.

Here's what they told us.

Who Gave Us Feedback

Before we dive into their responses, context matters.

The five people who took the time to share their experience weren't random candidates. They were:

• A Vice President of HR at a 15-country multinational retail operation with 20+ years of GCC experience

• A Group Head of Talent & Culture with 23+ years across Coca-Cola, AstraZeneca, and PepsiCo

• An HR Director with multi-regional experience spanning EMEA, Turkey, MENA, and Israel, specializing in transformation and M&A

• An AVP-level transformation specialist with deep expertise in professional services

• A Group Head managing HR operations across MEA and APAC

Combined, these five leaders have probably hired thousands of people. They know what good interviewing looks like. They've sat on both sides of the table more times than most of us can count.

When people like this give you feedback, you listen.

What Worked: The Surprising Positives

Let's start with what we got right, because honestly, we weren't sure what to expect.

1. Consistency and Structure

Every single respondent mentioned this.

"Structured, consistent, and removed any potential bias from initial screening. It ensures fairness and allows candidates to provide thoughtful, uninterrupted responses." — VP of HR, 20+ years, multinational retail

"Highly structured and helped keep answers focused — much more consistent than traditional first-round screening." — HR Director, multi-regional transformation

Traditional interviews vary wildly depending on who's conducting them. One interviewer asks behavioral questions. Another goes with gut feeling. A third gets distracted and cuts the interview short.

Our AI asked every candidate well-sequenced competency-based questions. No favoritism. No bias. No bad days affecting judgment.

2. Fairness Through Removal of Bias

This came up repeatedly, and it mattered more than we expected.

"Creates fairness and removes unnecessary bias when designed well." — HR Director, transformation specialist

"The neutrality, the clarity of questions, and the ability to reflect before answering created a positive experience." — VP of HR, multinational retail

Think about what bias looks like in traditional interviews:

• Halo effect (candidate went to the same university as the interviewer)

• Similarity bias (candidate reminds interviewer of themselves)

• Recency bias (last candidate gets more favorable evaluation)

• Visual bias (unconscious reactions to appearance, accent, age)

AI interviews don't have these. They can't. And candidates noticed.

3. Professional Design and User Experience

Even first-time AI interview participants were impressed.

"Overall found this an interesting experience. First time doing an AI interview and it exceeded my expectations. The technology was good... it worked well. Questions were well drafted competency based/behavioural questions and well structured." — Group Head, 23+ years experience

"The overall design of the interview experience is very professional and while I was a bit nervous before starting it, once I started talking to 'Jason' [our AI interviewer] the conversation flowed smoothly." — Group Head, Talent & Culture

We'd spent months refining the voice, the pacing, the question logic. It was validating to hear it worked.

4. Better Than Traditional Screening

Here's the kicker: 100% of respondents said they would use this product for their own hiring.

Not "maybe." Not "I'll think about it."

They would use it.

"Yes, I would consider using a similar tool in talent acquisition. It can bring efficiency, standardization, and scalability — especially in high-volume or multi-country environments." — VP of HR, 15-country operation

"Yes, I would absolutely consider using AI-based screening as part of a hiring process, especially for volume or early-stage leadership pipelines." — HR Director, multi-regional

Even the candidates who had critical feedback (more on that in a moment) saw the value clearly enough to want to use it themselves.

What Didn't Work: The Uncomfortable Truth

Now for the hard part.

Because while the core value proposition landed, the execution had issues. Real ones. The kind that make candidates frustrated even if they can see the potential.

Issue #1: Device Compatibility Created Friction

"Would have been better to share that I needed a laptop and can't take it from my phone but once this was sorted, it worked well." — Group Head, Talent & Culture

This seems small, but it's not.

Imagine being a busy HR leader squeezing in an interview between meetings. You click the link on your phone, ready to go. And... it doesn't work properly. You have to scramble to find a laptop, log back in, restart.

First impressions matter. And ours wasn't great for mobile users.

What we're fixing: We made the AI interview available over mobile. And we have released an extensive guide on How to Prepare for an AI Interview.

Timeline: Implemented and fixed.

Issue #2: No Orientation or Warm-Up

"A short orientation or warm-up question could help candidates get comfortable with the format." — VP of HR, multinational retail

"I was a bit nervous before starting it." — Group Head, Talent & Culture

Even senior HR professionals felt anxious before their first AI interview. That makes sense—it's unfamiliar. But we did nothing to ease that anxiety.

We just jumped straight into competency questions.

What we're fixing: A 2-minute orientation at the start of every interview:

"Hi [Name], I'm Jason, your AI interviewer today. I'll be asking you about your experience and qualifications for [Role]. Take your time with answers—there's no rush. You can think before responding. Before we dive in, tell me briefly: what interested you most about this opportunity?"

That warm-up question won't be scored. It's just to help candidates get comfortable talking to an AI before the real questions begin.

Timeline: Shipping in next sprint (2 weeks).

Issue #3: No Re-Record Option

"Offering the option to re-record a response in case of technical issues would enhance the user experience." — VP of HR, multinational retail

What if your dog barks mid-answer? What if you just completely botch your response and want to try again?

In a live interview, you'd say "Sorry, can I rephrase that?" The interviewer would say "Of course."

Our AI doesn't offer that option. Once you answer, it moves on.

What we're fixing: We are going to work with clients to use the platform as interview prep so candidates can effectively have 2 interviews, the first one will be an interview prep and then we will do a second call to record their actual AI interview

Maximum 2 interviews to prevent gaming the system, but still allow for genuine interviews.

Issue #4: Awkward Conversational Flow

This was the most nuanced feedback - and arguably the most important.

"The only thing I missed is the back & forth conversation. There were some moments of silence and I wasn't sure how much it would have accommodated if I had asked them to rephrase the question or if I could redirect the question in a different way." — Group Head, Talent & Culture

Here's what was happening:

Candidate finishes answering → Silence → Next question begins.

In human conversation, silence means "I'm thinking."

In AI conversation, silence means "Is this thing broken?"

Our AI agent is capable of voice presence with Human like conversational capabilities and responses under 800 milliseconds. And if this candidate had asked questions about the job or anything related to the interview it will have responded. But we take this feedback on board and we will make our AI model more chatty to be as close as possible to real conversations.

What we're fixing (short-term):

Conversational acknowledgments between responses:

• "Got it, thanks."

• "That's helpful context."

• "I appreciate that perspective."

The Insight That Surprised Us

Here's what we didn't expect:

Many candidates gave us detailed product feedback.

And not generic responses like "Great experience, thanks!" but 200+ word analyses with specific improvement suggestions.

Why?

Because when candidates experience something truly new - something they can see has potential but isn't quite there yet - they actually care about improving it.

They're not just passively receiving an interview. They're participating in the evolution of hiring technology.

These weren't just candidates. They became stakeholders.

What This Means for AI in Hiring

If you're building or buying AI hiring tools, here are three lessons from our experience:

1. Candidates Don't Resist AI They Resist Bad AI

Not a single candidate said "I hate AI interviews" or "This is dehumanizing."

They said "This is good, AND here's how to make it better."

The core value proposition fairness, consistency, bias reduction was strong enough to overcome friction.

But that doesn't mean we get to ignore UX. The opposite, actually. Because if the core is strong, fixing the edges is what takes us from "interesting" to "indispensable."

2. UX Matters as Much as Technical Performance

Our AI worked perfectly from a technical standpoint. The speech recognition was accurate. The questions made sense. The scoring was consistent.

But the user experience had gaps.

No device warnings. No warm-up. No re-record option.

Those gaps matter as much as the technical performance. Maybe more.

Because hiring isn't just about accuracy. It's about experience. Candidates remember how they felt during the process. And they tell others.

3. Transparency Builds Trust in New Technology

We could have ignored this feedback. Spun it. Focused only on the positive quotes.

Instead, we're publishing it the good and the uncomfortable and explaining what we're fixing.

Why?

Because that's how you build trust when you're introducing unfamiliar technology.

You don't pretend it's perfect. You show that you listen, you iterate, and you care more about getting it right than being right.

This feedback gave us a roadmap to product-market fit.

A Note to the Five Who Responded

If you're reading this - thank you.

You chose to help us build something better.

That's the kind of generosity that makes the startup journey worthwhile. And it's the kind of customer-obsessed mindset we hope to embody as we grow.

We're implementing most improvements you suggested. Not because we have to, but because you took the time to show us what was broken.

That's a gift. And we won't waste it.

Try It Yourself

Curious what these candidates experienced?

We're offering a limited number of demo interviews for HR leaders who want to experience AI screening firsthand—no strings attached.

[Book a 15-Minute Demo] → Experience the interview from a candidate's perspective, then let's talk about your hiring challenges.

Or if you just want to chat about AI in recruiting:

[Schedule a Discovery Call] → No sales pitch. Just a conversation about where hiring tech is heading and whether it makes sense for your team.

The Bottom Line

AI interviewing works.

But only if you:

1. Listen to candidates (not just clients)

2. Ship improvements fast (don't get defensive)

3. Stay transparent about limitations (trust beats perfection)

We're building an AI recruiter that candidates actually want to interview with.

Not because it's flawless.

Because we care enough to fix what's broken.

About the Author: Adil is Co-founder & CEO of shortlistd.io, an AI-powered recruitment platform. He spent 20+ years in the recruitment industry before building technology to solve the problems he saw every day. He believes the best products are built by listening to users even when the feedback stings.