Question 2LLMs, RAG and agents
How do you know an LLM feature is good enough to ship?
Evals are what separate a demo from production. Many roles list them by name.
“Applied AI Knowledge: Large language models (LLMs), prompt engineering, frameworks like RAG and function calling, and building evals to validate agentic AI solutions.”Forward Deployed Architect · DevRev
The answerFour steps
- 1
Write 50–200 real examples with expected outcomes, with customer experts
Say it like this
“I'd sit with two of their lawyers and write 150 real clauses with the right answer for each.”
Why it scores
Test cases from the customer's own experts, with a number. Not “I'd test it thoroughly.”
- 2
Agree on a pass bar and the errors that are never acceptable
Say it like this
“Before we run anything, we agree the bar: 90% correct overall, and zero misses on liability caps.”
Why it scores
Splits errors into acceptable and never acceptable. Most candidates miss this line.
- 3
Score automatically where possible; review a sample by hand
Say it like this
“Most checks run automatically, but I review 20 by hand each week with a lawyer, because the automatic scorer can be wrong too.”
Why it scores
Doesn't fully trust a model grading a model. Sounds like someone who has run evals.
- 4
Re-run on every prompt or model change, and watch live traffic after launch
Say it like this
“after launch every answer a lawyer edits goes back into the test set.”
Why it scores
Turns real use into new tests, so quality holds after the demo.
StrongAll together
About a minute, out loud. The marked parts are the four steps; hover one to find it above.
Take a contract-review assistant at a law firm. I'd sit with two of their lawyers and write 150 real clauses with the right answer for each.1 Before we run anything, we agree the bar: 90% correct overall, and zero misses on liability caps.2 Some errors are fine; some never are. Most checks run automatically, but I review 20 by hand each week with a lawyer, because the automatic scorer can be wrong too.3 The set re-runs on every prompt or model change, and after launch every answer a lawyer edits goes back into the test set.4
WeakThe answer that fails
Sounds fine. Common. Scores low.
“I'd try a bunch of prompts myself, and once the answers look good and the customer likes the demo, we ship and monitor it.”
Interviewer's note
“Looks good” isn't a bar. Nothing to re-run when the model changes next month.
PracticeTwo minutes, then they push back
Your turn
How do you know an LLM feature is good enough to ship?
Answer like you're in the room: about two minutes, no notes. Say it out loud, even alone. It feels different from thinking it. Then check your answer against what interviewers score.