SAP's Chief Design Officer: AI Evals Fail Without Designers

Eighty-eight percent of researchers doubt the quality of AI-generated test personas — and that number alone should make companies rethink who runs their AI evaluations, according to SAP Chief Design Officer Arin Bhowmick.

In a new Fast Company essay, Bhowmick argues that AI evaluations — the tests that decide whether an AI feature ships — sit with engineers and product managers when they should also sit with designers. Evals look like the least designerly corner of product development: scripts, dashboards, percentages. But behind each one sits a decision about what "good" means for the person using the product.

"What should the AI do when it isn't sure? Which mistakes can people live with?" Bhowmick writes. "Designers make calls like these every day."

His case rests on five arguments.

Why can't numbers alone judge AI quality?

The hardest quality to quantify in evals, Bhowmick says, is whether the AI gave the user the right answer. Even user ratings mislead. A Stanford study found people trusted flattering AI answers more than answers that pushed back.

Judging "rightness" takes trained taste, he argues. Teams can train AI to grade other AI's outputs, but someone with judgment has to teach the grader what to flag — jargon a new user won't understand, or a cheerful reply to a complaint.

The takeaway: engineers can miss UX failures that a trained designer catches on sight.

What happens when the AI doesn't know the answer?

AI often sounds confident even when it's wrong. Oxford researchers have documented chatbots mixing good and bad medical advice, leaving users unable to tell the difference.

Hallucinations make products feel untrustworthy and push users to competitors, Bhowmick writes. His proposed eval criterion: when the AI doesn't know, it must say so and point the user somewhere useful.

"An AI not knowing the answer is just another dead end," he writes — and designers have handled error messages and dead ends for years.

Are synthetic test users good enough?

No, Bhowmick says. Some eval teams build test sets with AI-generated personas. He doesn't see that working, and cites a User Interviews survey in which 88% of researchers raised concerns about the quality and accuracy of synthetic users.

UX researchers, by contrast, know how to recruit real people — and they know how messy those people are. Pre-AI product research taught designers how users mumble, jump between topics, and change answers depending on mood or time of day.

The lesson in plain words: real users beat simulated ones, and designers know how to find them.

Who checks the eval's assumptions?

Every eval is built on assumptions about what users want, Bhowmick writes — and those assumptions are rarely written down. They usually come from business goals or from whatever is easiest to measure.

His example: a team building a support chatbot to cut costs may assume people prefer not talking to a human. Any handoff to a live agent then counts as a failed test, so the bot that never hands anyone over scores best — even when people get nowhere.

Designers write assumptions down and check them with real users before building a feature, he says. Evals deserve the same treatment.

When is an eval failure actually a design failure?

When an eval fails, teams tweak prompts or switch models. Sometimes that's wrong, Bhowmick argues — sometimes the product itself is the problem.

He quotes Hamel Husain, a machine learning engineer with more than 20 years of experience who now teaches AI evals, who wrote this summer: if an AI's answers are hard for a team to check, they're often hard for users too.

Bhowmick's illustration: a travel assistant books the wrong date because a user typed "next Friday," which can mean two different days. A better prompt won't fix that. Asking the user to confirm the exact date will — and that's a design decision.

Before spending another week refining prompts, he writes, teams should have a designer review the failed cases.

The bottom line

Engineers and product managers will keep running AI evals, Bhowmick concedes — as they should. But most decisions behind evals are about people. His advice to design leaders: help write the eval criteria before they're final.

"Deciding which mistakes users can live with is our job, too," he writes.

For companies shipping AI features, that means the cheapest fix for a failing eval may already sit on the design team's payroll.