Question 1LLMs, RAG and agents
A customer wants a chatbot over their internal documents. How do you build it?
RAG is the most common first project. They want the whole path, from messy files to answers people trust.
“Applied AI Knowledge: Large language models (LLMs), prompt engineering, frameworks like RAG and function calling, and building evals to validate agentic AI solutions.”Forward Deployed Architect · DevRev
The answerFour steps
- 1
Collect and clean the documents; handle permissions per user
Say it like this
“First I'd drop stale drafts and duplicates, and carry each file's access list along, so people only get text they could open themselves.”
Why it scores
Cleans the source and keeps permissions. Most bad answers start in the files, not the model.
- 2
Split them into chunks, embed, and store with source links
Say it like this
“Then I split by headings into chunks of a few hundred tokens, embed them, and store each one with its source link and page.”
Why it scores
Chunks by structure and keeps source links, which makes citations and debugging possible.
- 3
Retrieve, then answer with citations; say “I don't know” when retrieval is weak
Say it like this
“the model answers only from the top results and cites them. If nothing scores above a threshold, it says “I couldn't find that” instead of guessing.”
Why it scores
Has a refusal path. A bot that guesses loses trust in its first week.
- 4
Build a test set of real questions with expected answers before launch
Say it like this
“I'd pull 100 real questions from their help desk tickets, write the right answer and source for each, and score every change against them.”
Why it scores
Test set from real tickets, not invented questions. Measures before launch.
StrongAll together
About a minute, out loud. The marked parts are the four steps; hover one to find it above.
Say it's an engineering firm with 12,000 files across SharePoint and Confluence. First I'd drop stale drafts and duplicates, and carry each file's access list along, so people only get text they could open themselves.1 Then I split by headings into chunks of a few hundred tokens, embed them, and store each one with its source link and page.2 At question time, the model answers only from the top results and cites them. If nothing scores above a threshold, it says “I couldn't find that” instead of guessing.3 Before anyone uses it, I'd pull 100 real questions from their help desk tickets, write the right answer and source for each, and score every change against them.4
WeakThe answer that fails
Sounds fine. Common. Scores low.
“I'd load the documents into a vector database with LangChain, pick a good embedding model, and connect GPT to it. Then I'd tune the prompt until the answers look right.”
Interviewer's note
All plumbing. Nothing on stale files, who can see what, or how we'd know the answers are right.
PracticeTwo minutes, then they push back
Your turn
A customer wants a chatbot over their internal documents. How do you build it?
Answer like you're in the room: about two minutes, no notes. Say it out loud, even alone. It feels different from thinking it. Then check your answer against what interviewers score.