Question 5LLMs, RAG and agents
Support says the AI assistant “is wrong sometimes.” How do you make that something you can fix?
Customers describe AI problems in feelings. They want to see you turn that into a number.
“Applied AI Knowledge: Large language models (LLMs), prompt engineering, frameworks like RAG and function calling, and building evals to validate agentic AI solutions.”Forward Deployed Architect · DevRev
The answerFour steps
- 1
Collect real failures from logs and from the support team
Say it like this
“I'd ask support for every bad answer from the last two weeks and pull the thumbs-down chats from the logs. Maybe that's 60 cases.”
Why it scores
Real failures first. Without examples you're fixing a rumor.
- 2
Sort them: wrong document found, right document but wrong answer, or the question wasn't answerable
Say it like this
“I sort each one: it found the wrong article, it found the right article but answered wrong, or the question needed account data it doesn't have.”
Why it scores
Groups by cause, because each group needs a different fix.
- 3
Build a test set from them and measure today's error rate
Say it like this
“I mix the 60 into 200 normal questions and measure: 11% wrong today.”
Why it scores
Mixes failures with normal traffic, so the baseline is an honest rate.
- 4
Fix the biggest group first and show the number moving
Say it like this
“Then I fix the biggest group first, archiving old pricing pages, re-run, and send support the result: 11% down to 5%.”
Why it scores
Fixes the biggest group and reports a number support can repeat.
StrongAll together
About a minute, out loud. The marked parts are the four steps; hover one to find it above.
Say it's a billing assistant at a telecom. I'd ask support for every bad answer from the last two weeks and pull the thumbs-down chats from the logs. Maybe that's 60 cases.1 I sort each one: it found the wrong article, it found the right article but answered wrong, or the question needed account data it doesn't have.2 Say 30 are the first kind, mostly old pricing pages. I mix the 60 into 200 normal questions and measure: 11% wrong today.3 Then I fix the biggest group first, archiving old pricing pages, re-run, and send support the result: 11% down to 5%.4
WeakThe answer that fails
Sounds fine. Common. Scores low.
“I'd look at the system prompt and make it stricter, maybe try a newer model, and ask support to tell us whenever they see more problems.”
Interviewer's note
Changing things blind. No examples and no baseline, so nobody can tell if anything got better.
PracticeTwo minutes, then they push back
Your turn
Support says the AI assistant “is wrong sometimes.” How do you make that something you can fix?
Answer like you're in the room: about two minutes, no notes. Say it out loud, even alone. It feels different from thinking it. Then check your answer against what interviewers score.