Question 3LLMs, RAG and agents
The model gives a confident wrong answer to an important customer. What now?
Tests how you handle trust, on top of the technical fix.
“Applied AI Knowledge: Large language models (LLMs), prompt engineering, frameworks like RAG and function calling, and building evals to validate agentic AI solutions.”Forward Deployed Architect · DevRev
The answerFour steps
- 1
Tell the customer the same day what you know and when they'll hear more
Say it like this
“Same day, I tell her we've seen it, that she shouldn't rely on that answer, and that she'll have the cause by tomorrow.”
Why it scores
Speaks up before having the full story. Silence after a wrong answer costs more trust than the error.
- 2
Reproduce it and find the cause (bad retrieval, missing data, prompt, model)
Say it like this
“Then I replay the exact question with the logged retrieval. Say the model found the 2023 draft instead of the signed amendment.”
Why it scores
Uses the logs to find the cause instead of guessing at the prompt.
- 3
Add the case to the test set so it can't come back unnoticed
Say it like this
“I add her question and five like it to the test set, so no later change brings it back quietly.”
Why it scores
Turns one incident into permanent tests. Sounds like someone who runs evals.
- 4
Add a guard where it matters: citations, confidence checks, human review
Say it like this
“for dates and money, the answer now shows the quoted clause next to it, so she can check it in two seconds.”
Why it scores
Puts the guard where errors cost most, and makes checking cheap for the user.
StrongAll together
About a minute, out loud. The marked parts are the four steps; hover one to find it above.
Say our contract assistant told their head of procurement that a supplier deal renews in March. It actually ended in January. Same day, I tell her we've seen it, that she shouldn't rely on that answer, and that she'll have the cause by tomorrow.1 Then I replay the exact question with the logged retrieval. Say the model found the 2023 draft instead of the signed amendment.2 So I fix the index first: drafts get tagged and ranked below signed versions. I add her question and five like it to the test set, so no later change brings it back quietly.3 And for dates and money, the answer now shows the quoted clause next to it, so she can check it in two seconds.4
WeakThe answer that fails
Sounds fine. Common. Scores low.
“I'd look into what went wrong, improve the prompt so the model is more careful, maybe add “only answer if you're sure,” and let the customer know once it's fixed.”
Interviewer's note
Guessed the cause, patched the prompt, told the customer last. Nothing stops it coming back.
PracticeTwo minutes, then they push back
Your turn
The model gives a confident wrong answer to an important customer. What now?
Answer like you're in the room: about two minutes, no notes. Say it out loud, even alone. It feels different from thinking it. Then check your answer against what interviewers score.