Question 6LLMs, RAG and agents

When would you fine-tune a model, and when is prompting or retrieval enough?

Customers often ask for fine-tuning first. They want to hear you pick the cheapest thing that works.

Asked for in 80% of live FDE postings

“Applied AI Knowledge: Large language models (LLMs), prompt engineering, frameworks like RAG and function calling, and building evals to validate agentic AI solutions.”

Forward Deployed Architect · DevRev

They scoreMeasures before choosingMatches fix to failureCounts the upkeep

The answerFour steps

1
Try clear prompts and examples first; measure against a test set
Say it like this
“First I write clear instructions with five examples and score it on 150 real questions. Say that gets 70%.”
Why it scores
Measures the cheap option first, so the choice rests on a number.
2
If the model lacks the customer's facts, add retrieval
Say it like this
“Most misses are facts it has never seen, like part numbers and torque specs. That's retrieval: index the manuals and cite them. Now we're at 88%.”
Why it scores
Puts facts in retrieval, where they can be updated and cited.
3
Fine-tune when you need a format or behaviour prompts can't hold, and you have good examples
Say it like this
“I'd fine-tune if every answer had to fit a strict format for their ticket system, prompts kept breaking it, and we had 1,000 good examples.”
Why it scores
Names a real reason to fine-tune and the data it needs.
4
Count the cost: training data, upkeep and re-testing on every model update
Say it like this
“And I'd price it: building that data, retraining when the base model changes, and re-running the tests each time.”
Why it scores
Counts the ongoing cost, which customers rarely hear about up front.

StrongAll together

About a minute, out loud. The marked parts are the four steps; hover one to find it above.

Say a customer wants to fine-tune on their product manuals so the bot “knows” their machines. First I write clear instructions with five examples and score it on 150 real questions. Say that gets 70%.¹ Most misses are facts it has never seen, like part numbers and torque specs. That's retrieval: index the manuals and cite them. Now we're at 88%.² Fine-tuning is a poor way to teach facts, and theirs change every quarter. I'd fine-tune if every answer had to fit a strict format for their ticket system, prompts kept breaking it, and we had 1,000 good examples.³ And I'd price it: building that data, retraining when the base model changes, and re-running the tests each time.⁴

WeakThe answer that fails

Sounds fine. Common. Scores low.

“If the customer has a lot of domain data, I'd fine-tune, because the model learns their terminology and gives more accurate answers than a general model with prompts.”

Interviewer's note

Fine-tuning to teach facts. Out of date when the manuals change, and no measurement behind the choice.