CareerGuide 2 of 36
Career guide · Guide 2 of 36
Top 50 AI Engineer Interview Questions and Answers
Top 50 AI engineer interview questions with simple answers: AI agents, MCP, RAG pipelines, evaluation, guardrails, cost, latency and AI system design.
On this page
These 50 questions cover what AI engineer interviews ask about building and running AI applications: agents and tools, MCP, RAG pipelines, evaluation, security, cost and latency, and AI system design. Each answer is written in plain words and gets straight to the point. For questions about how language models themselves work (tokens, attention, fine-tuning, sampling), see the Top 50 LLM interview questions.
Question list
Show all 50 questions
- What does an AI engineer do?
- How is an AI engineer different from an ML engineer and a data scientist?
- What skills does an AI engineer need?
- What is an AI agent?
- What is the difference between AI agents and agentic AI?
- What is the difference between a workflow and an agent?
- What is MCP (Model Context Protocol)?
- What is the difference between RAG and MCP?
- What is LangChain, and what is LangGraph?
- Why do AI applications need evaluation?
- What is a vector embedding used for in an AI application?
- How do you choose which model to use?
- What does an AI engineer’s development loop look like?
- What are the main risks of putting an LLM application in production?
- What is context engineering?
- Walk through how you would build a RAG pipeline.
- Your RAG system gives wrong answers. How do you debug it?
- How do you handle document permissions in RAG?
- How do you design tools for an agent?
- How do you stop an agent from looping or running up costs?
- How do you evaluate an agent?
- What is LLM observability, and what should you log?
- How do you reduce latency in an AI application?
- What is semantic caching?
- How do you test prompts like code?
- What is hybrid search, and why use it?
- How do you handle conversation memory in a chatbot?
- When should you fine-tune instead of improving prompts or retrieval?
- How do you protect an AI application against prompt injection?
- How would you build a document question-answering system for a company?
- What are structured outputs, and why do they matter in AI pipelines?
- What is a multi-agent system, and when is it useful?
- How do you keep an AI feature’s data pipeline fresh?
- What metrics do you track for an AI product?
- How do you handle a model provider outage or a model change?
- Design an AI customer-support assistant for an online shop.
- How would you build an evaluation pipeline for an LLM product?
- How do you decide between building on an API model and self-hosting an open-weight model?
- How do you keep costs predictable at scale?
- How do you make an agent safe to take real actions?
- How do you measure hallucination in a RAG system?
- What changes when an AI system serves many customers (multi-tenant)?
- How do you improve retrieval quality beyond basic vector search?
- How would you migrate a production AI feature to a new model?
- What would you put in an AI system’s incident runbook?
- How do you work with data privacy rules in AI applications?
- What is cache-augmented generation (CAG), and when would you use it instead of RAG?
- Tell me about an AI project you built end to end.
- What is the difference between a demo and a production AI system?
- What is the difference between RAG and agentic RAG?
Beginner AI engineer interview questions (freshers)
1. What does an AI engineer do?
An AI engineer builds software products and features that use AI models, mostly large language models. They choose models, design prompts, connect the model to data through retrieval, give it tools, build agents and workflows, evaluate quality, and run the system reliably in production with monitoring and cost control. Unlike a research scientist, they rarely train models from scratch; the work is closer to software and data engineering, applied to models.
2. How is an AI engineer different from an ML engineer and a data scientist?
A data scientist analyses data and builds statistical or machine learning models to answer questions. An ML engineer trains, deploys and maintains custom models, with a focus on training pipelines and serving. An AI engineer mostly builds applications on top of existing foundation models: prompting, retrieval, tools, agents and evaluation. The roles overlap, and in small teams one person often does parts of all three.
A quick way to tell them apart in a job description: if it mentions training pipelines, GPUs and model serving, it is closer to ML engineering; if it mentions prompts, RAG, agents and evaluation, it is AI engineering.
3. What skills does an AI engineer need?
Strong software engineering in Python or TypeScript, including APIs, testing and deployment. Knowledge of how LLMs behave: tokens, context windows, sampling and their failure modes. Retrieval skills: embeddings, chunking, vector and hybrid search. Building tools and agents, evaluating outputs systematically, and running systems in production with logging, cost and latency control. Data skills help a lot, because most AI quality problems are data problems.
Product sense matters too: knowing when AI is the right tool, and when a simple rule or search box would work better, saves a lot of wasted effort.
4. What is an AI agent?
An AI agent is a system where a language model decides which steps to take to reach a goal, usually by calling tools in a loop. It reads the task and chooses an action, such as searching a database or calling an API. It looks at the result, then decides the next action, until it can answer or finish. Unlike a fixed workflow, the sequence of steps is chosen by the model at run time, which makes agents flexible but harder to control and test.
Learn more: What Are AI Agents? How the Agent Loop Works, with Code
5. What is the difference between AI agents and agentic AI?
An AI agent is a specific system that uses a model to choose actions and tools to complete a task. Agentic AI is the broader idea, or style of system, where AI acts with some autonomy: planning steps, using tools, working over many steps, and sometimes several agents working together. In practice the terms are often used loosely; in interviews, define what you mean and focus on how much autonomy the system has.
Learn more: What Are AI Agents? How the Agent Loop Works, with Code
6. What is the difference between a workflow and an agent?
In a workflow, your code defines the steps and their order, and the model is called at fixed points, for example classify the email, then extract fields, then draft a reply. In an agent, the model decides the steps and when to stop. Workflows are more predictable, cheaper and easier to test, so use them when the steps are known. Use agents only when the path genuinely depends on what the model finds.
Learn more: Workflows vs AI Agents: When to Use Each
7. What is MCP (Model Context Protocol)?
The Model Context Protocol is an open standard for connecting AI applications to tools and data sources. An MCP server exposes tools, resources and prompts in a standard way, and any MCP-compatible client, such as an AI assistant or agent framework, can discover and use them. Instead of writing a custom integration for every model and every tool, you write one server per system. It standardises the plumbing; you still control permissions and security.
Learn more: MCP (Model Context Protocol) Explained Simply, with Code
8. What is the difference between RAG and MCP?
RAG is a technique: retrieve relevant information and put it in the prompt so the model answers from it. MCP is a protocol: a standard way for AI applications to connect to tools and data sources. They work together: a retrieval system can be exposed as an MCP tool that an agent calls when it needs documents. RAG answers “how do we give the model knowledge?”; MCP answers “how do we connect the model to systems?”
Learn more: What Is RAG? Retrieval-Augmented Generation Explained with Code
9. What is LangChain, and what is LangGraph?
LangChain is an open-source framework with building blocks for LLM applications: model interfaces, prompts, document loaders, retrievers and tool integrations. LangGraph, from the same project, builds agents and multi-step workflows as graphs of nodes with shared state, supporting loops, branching, persistence and human approval steps. Frameworks speed up prototypes, but many teams use plain code for simple systems to keep full control.
A fair interview answer is that you choose the tool for the job: plain code for simple features, frameworks when they save real effort. See LangChain vs LangGraph for a fuller comparison.
10. Why do AI applications need evaluation?
Model outputs vary, and a small change to a prompt, model or retrieval setting can improve some answers while breaking others. Without an evaluation set, you only notice problems when users complain. A good evaluation set of real inputs with expected results lets you measure quality, compare versions and catch regressions before release, just like automated tests for normal software.
Evaluation also settles debates quickly. Instead of arguing whether a new prompt is better, you run both versions on the same 100 examples and compare the scores.
11. What is a vector embedding used for in an AI application?
Embeddings turn text, and sometimes images, into vectors so that similar meanings are close together. AI applications use them to find relevant documents for RAG, to search by meaning, to group similar support tickets, to detect duplicates and to route requests to the right handler. They are created by an embedding model once per document and once per query, and stored in a vector index.
Use the same embedding model for documents and queries, and re-embed everything if you change models, because vectors from different models cannot be compared.
Learn more: What Are Embeddings? Vectors, Similarity and Semantic Search Explained
12. How do you choose which model to use?
Start from the task and constraints: quality needed, latency, cost per request, context length, data privacy and whether you must self-host. Build a small evaluation set and test a few candidate models on it, including cheaper and smaller ones. Pick the cheapest model that meets the quality bar, and consider routing: a small model for easy requests and a larger one for hard ones. Re-evaluate as new models appear.
Do not choose from public benchmarks alone: a model that leads a leaderboard can still be worse on your specific task, format and language.
Learn more: How to Choose an LLM: Quality, Cost, Latency and Privacy
13. What does an AI engineer’s development loop look like?
Define the task and success criteria, collect real example inputs, and build the simplest version, often a single prompt. Run it on the examples, look at failures, and improve the prompt, data, retrieval or tools. Turn the examples into an automated evaluation set, then iterate while tracking scores, cost and latency. Ship behind monitoring, collect real failures, add them to the evaluation set and repeat.
The key habit is to look at real outputs, many of them, by hand. Most improvements come from noticing patterns in failures, not from guessing.
14. What are the main risks of putting an LLM application in production?
Wrong or made-up answers, prompt injection and data leakage, unsafe or off-brand output, high or unpredictable cost, slow responses, dependence on a provider’s availability and model changes, and unclear accountability when the system makes a mistake. Manage them with evaluation, guardrails, least-privilege tools, monitoring, fallbacks, cost limits and human review for high-stakes actions.
In an interview, pair each risk with a mitigation. For example: wrong answers are reduced by retrieval with citations and evaluation; injection by limiting tool permissions; cost by caching, smaller models and budgets; outages by a fallback model; and accountability by logging and human approval for important decisions.
15. What is context engineering?
Context engineering is deciding what information goes into the model’s context window for each call: instructions, retrieved documents, conversation history, tool results and examples, and in what order and form. Prompt engineering focuses on the wording of instructions; context engineering covers the whole input. In agents and long conversations it matters most, because too little context causes mistakes and too much adds cost and confusion.
A practical example: for a support agent, the context might include the system rules, the customer’s last three orders, the relevant policy passage and a short summary of the conversation so far, and nothing else.
Intermediate AI engineer interview questions
16. Walk through how you would build a RAG pipeline.
Ingest: load documents, clean them, split them into chunks on natural boundaries, and keep metadata such as source, date and access rights. Index: embed each chunk and store it in a vector index, often alongside a keyword index. Query: embed the question, retrieve candidates with hybrid search, rerank them, and put the best few in the prompt with instructions to answer only from them and cite sources. Evaluate retrieval and answers separately, and refresh the index when documents change.
Learn more: What Is RAG? Retrieval-Augmented Generation Explained with Code
17. Your RAG system gives wrong answers. How do you debug it?
Find which stage fails. For each bad answer, check whether the right passage was retrieved at all. If not, it is a retrieval problem: fix chunking, add hybrid search or reranking, improve metadata filters, or rewrite queries. If the right passage was retrieved but the answer is still wrong, it is a generation problem: tighten the prompt, reduce irrelevant context, require citations, or try a stronger model. Add the failing cases to the evaluation set.
Learn more: What Is RAG? Retrieval-Augmented Generation Explained with Code
18. How do you handle document permissions in RAG?
Store each chunk’s access information, such as allowed groups, as metadata when indexing, taken from the source system. At query time, filter retrieval by the user’s identity before anything reaches the model, so the model never sees documents the user may not read. Keep permissions in sync when they change, and test with users who have different access. Never rely on the prompt to hide information.
Also apply permissions to everything derived from documents: caches, summaries, logs and evaluation sets can all leak restricted content if they are shared across users.
Learn more: What Is RAG? Retrieval-Augmented Generation Explained with Code
19. How do you design tools for an agent?
Give each tool one clear job, a descriptive name, and a precise description of when to use it and what it returns, because the model chooses tools from these descriptions. Use simple, typed parameters and validate them in code. Return concise, structured results with useful error messages the model can act on. Limit each tool’s permissions, make write actions idempotent, and require confirmation for risky ones such as payments or deleting data.
Learn more: Tool Calling (Function Calling) in LLMs, Explained with Code
20. How do you stop an agent from looping or running up costs?
Set hard limits: maximum steps, maximum tool calls, a time limit and a token or cost budget per task. Detect repeated identical actions and stop or change strategy. Give the agent clear stop conditions and a way to say it cannot finish. Log every step so loops can be analysed, and add the problematic cases to your evaluation set. Often the real fix is replacing part of the agent with a fixed workflow.
Learn more: What Are AI Agents? How the Agent Loop Works, with Code
21. How do you evaluate an agent?
Evaluate the final outcome and the path. Outcome: did it complete the task correctly, measured with checks on the final state, such as the right record updated. Path: did it use sensible tools in a reasonable number of steps, without unsafe actions. Run realistic tasks in a sandbox with fake or test systems, repeat each task several times because results vary, and track success rate, cost and time per task across versions.
Learn more: Evaluating and Securing AI Agents
22. What is LLM observability, and what should you log?
Observability means being able to see what your AI system did and why. For each request, log a trace: inputs, prompt and model versions, retrieved documents, each tool call with arguments and results, outputs, latency, tokens and cost, plus user feedback. Mask personal data. Traces let you debug individual failures, find slow or costly steps, and build evaluation sets from real traffic.
Group traces by session or task so you can follow a whole conversation or agent run, not just single calls. Many teams use dedicated LLM tracing tools, but the same data can be written to your normal logging stack.
23. How do you reduce latency in an AI application?
Measure where the time goes first. Then stream tokens to the user, use a smaller or faster model for simple steps, shorten prompts and retrieved context, run independent calls in parallel, cache embeddings, retrieval results and repeated answers, and avoid unnecessary agent steps. Set a latency budget per step. For self-hosted models, use an optimised inference server and keep models warm.
Users judge speed mostly by time to first token, so streaming and a quick first step often matter more than total time. Show progress, such as “searching documents”, during longer agent tasks.
Learn more: How to Choose an LLM: Quality, Cost, Latency and Privacy
24. What is semantic caching?
Semantic caching stores previous questions and answers, and when a new question has a very similar embedding to a cached one, it returns the cached answer instead of calling the model. It cuts cost and latency for repeated questions, such as common support queries. The risks are returning an answer that is not quite right for a slightly different question, and stale answers, so use a strict similarity threshold, expiry times and exclude personalised answers.
25. How do you test prompts like code?
Keep prompts in version control with the code. Build an evaluation set of inputs with expected results or scoring rules, and run it automatically in CI whenever a prompt, model or retrieval setting changes. Use exact checks where possible and model-graded checks for open answers. Compare scores with the previous version and block releases that get worse on important cases.
Keep a small fast suite of key examples that runs on every change, and a larger suite that runs nightly or before releases, because model calls cost time and money.
Learn more: Prompt Engineering and Structured Output: Getting Reliable Answers from LLMs
26. What is hybrid search, and why use it?
Hybrid search combines keyword search, such as BM25, with semantic vector search and merges the results, often with reciprocal rank fusion, then usually reranks them. Keyword search finds exact terms, product codes and names; vector search finds meaning and synonyms. Together they retrieve the right passages more often than either alone, which is why most production RAG systems use hybrid search.
For example, a question about error code “PAY-317” is best matched by keywords, while “my payment keeps failing” is best matched by meaning. Hybrid search handles both kinds of question in one system.
Learn more: Hybrid Search and Reranking for RAG, Explained with Code
27. How do you handle conversation memory in a chatbot?
Keep recent messages in full, summarise older parts of the conversation, and store important facts, such as the user’s preferences, separately so they can be retrieved when relevant. Respect the context window and cost: do not send the whole history every time. Let users see and delete stored memory where appropriate, and never store sensitive data you do not need.
Test long conversations specifically: many chatbots work well for five messages and then lose track, repeat themselves or contradict earlier answers.
Learn more: Agent Memory and Planning: How AI Agents Remember and Plan
28. When should you fine-tune instead of improving prompts or retrieval?
Fine-tune when prompting and retrieval have been tried and the problem is behaviour, not knowledge: a strict output style, a specialised classification task, or making a smaller model perform as well as a large one to cut cost and latency. You need a good set of high-quality examples, an evaluation set and a plan to retrain when things change. For new facts or private documents, use retrieval instead.
A good rule: fine-tune only when you can show, with your evaluation set, that prompting and retrieval have stopped improving and the remaining errors are about behaviour.
Learn more: Fine-Tuning LLMs and LoRA Explained Simply
29. How do you protect an AI application against prompt injection?
Assume any text the model reads, including documents, web pages, emails and tool results, may contain malicious instructions. Limit the damage rather than hoping to detect everything:
- Give tools the least permissions they need.
- Keep secrets out of prompts.
- Require human confirmation for sensitive actions.
- Mark untrusted content clearly in prompts.
- Validate outputs, and never let the model’s output directly trigger actions on other users’ data.
Test your defences with your own injection attempts, such as documents containing hidden instructions, and add them to your evaluation set so regressions are caught.
Learn more: Evaluating and Securing AI Agents
30. How would you build a document question-answering system for a company?
Clarify users, document sources, access rules and success measures. Build ingestion from the sources with refresh, chunking and permissions metadata; hybrid retrieval with reranking; and a generation step that answers only from retrieved passages with citations and says when it does not know. Add an evaluation set of real questions, logging and feedback buttons, and a review process for failures. Start with one department and expand.
Plan for document quality problems: duplicates, outdated versions and scanned PDFs. Showing the source and date with every answer helps users trust the system and spot outdated information.
31. What are structured outputs, and why do they matter in AI pipelines?
Structured outputs are model responses that follow a defined schema, such as JSON with specific fields, enforced by the API or by constrained decoding. They matter because code needs to use the result: save fields to a database, decide the next step or call a tool. Combine them with validation in code and clear field descriptions; when validation fails, retry with the error or fall back to a safe default.
Learn more: Prompt Engineering and Structured Output: Getting Reliable Answers from LLMs
32. What is a multi-agent system, and when is it useful?
A multi-agent system splits work between several agents with different roles, tools or instructions, such as a planner, a researcher and a writer, coordinated by an orchestrator or by passing messages. It can help when tasks are large and separable, or when different steps need very different tools and context. It also adds cost, latency and new failure modes, so start with one agent or a workflow and split only when there is a clear benefit.
Learn more: Multi-Agent Systems: When and How to Use Several Agents
33. How do you keep an AI feature’s data pipeline fresh?
Treat the knowledge base like any data pipeline: load new and changed documents incrementally, remove deleted ones, re-embed changed chunks, and track freshness with monitoring and alerts. Record when each document was last indexed and show it in citations where useful. Re-index everything when you change the chunking or embedding model. See modern data pipelines.
A useful check is a freshness test: ask the system about something that changed yesterday and confirm it answers with the new information. If it does not, the pipeline or the index is behind.
34. What metrics do you track for an AI product?
Quality: task success, correctness on evaluation sets, groundedness of answers, user ratings. Usage: active users, repeat use, tasks completed. Reliability: error and timeout rates, fallback rate. Speed: time to first token and total response time. Cost: tokens and money per request and per successful task. Safety: blocked or flagged outputs and incidents. Tie at least one metric to the business outcome the feature was built for.
Review these metrics together: a rise in success rate is not good news if cost per task doubled or users stopped coming back.
35. How do you handle a model provider outage or a model change?
Design for it: put model calls behind your own interface, keep a fallback model or provider configured and tested, use retries with backoff and timeouts, and degrade gracefully, such as showing search results without a generated answer. Pin model versions where the provider allows, and re-run your evaluation set before switching versions, because a newer model can behave differently on your tasks.
Test the fallback regularly, for example by sending a small share of traffic to it, so you know it still works and how its quality compares.
Advanced AI engineer interview questions (experienced)
36. Design an AI customer-support assistant for an online shop.
Clarify scope: answer questions, check orders, start returns, and when to hand over to a human. Retrieval over help articles and policies; tools for order lookup and return creation with the customer’s identity enforced in code; a classifier or router for intent and urgency; guardrails for refunds above a limit, which go to a human. Evaluate on real past tickets, monitor resolution rate, escalation rate and customer satisfaction, and review failures weekly.
37. How would you build an evaluation pipeline for an LLM product?
Collect a versioned dataset from real usage and expert-written edge cases, labelled with expected answers or rubrics. Define scorers: exact and code-based checks first, then model-graded rubrics checked against human labels. Run the suite automatically on every change and nightly on production samples, store results per version, and show trends on a dashboard. Use online signals such as user feedback and A/B tests for final decisions.
Keep the dataset fresh: add new failure cases from production every week, and remove or update cases whose expected answers have changed.
38. How do you decide between building on an API model and self-hosting an open-weight model?
API models are fastest to start, usually the most capable, and need no infrastructure, but you pay per token and depend on the provider. Self-hosting gives control over data location, customisation and cost at high volume, but needs GPUs, an inference stack and operations skills. Compare quality on your evaluation set, total cost at expected volume, latency, privacy rules and team skills. Many teams use APIs first and self-host high-volume narrow tasks later.
39. How do you keep costs predictable at scale?
Measure cost per request and per successful task, by feature and customer. Use the smallest model that meets quality, route hard cases to bigger models, cap output length and context size, cache aggressively, and batch offline jobs. Set budgets and alerts, rate-limit heavy users, and watch agent step counts. Re-check prices and models regularly, since cheaper models of similar quality appear often.
Watch for hidden costs too: retries, long conversation histories, large retrieved contexts and agent loops often cost more than the main model call.
40. How do you make an agent safe to take real actions?
Enforce safety in code, not in prompts. Give the agent only the tools and permissions the task needs, scoped to the current user. Validate every argument, make actions reversible or idempotent where possible, and require human approval for high-impact actions. Log every action with its reasoning and inputs. Test in a sandbox with adversarial inputs, and limit steps, spending and the systems it can reach.
A useful pattern is “propose, then execute”: the agent prepares an action, such as a refund, and a person or a strict rule in code approves it before it runs.
Learn more: Evaluating and Securing AI Agents
41. How do you measure hallucination in a RAG system?
Measure groundedness: whether each claim in the answer is supported by the retrieved passages. Break answers into claims and check each against the sources, using human reviewers on a sample and a judge model at scale, validated against the human labels. Also measure citation accuracy and whether the system correctly says “I don’t know” when the answer is not in the sources. Track these per release.
Separate two failures: answers that contradict the sources, and answers that add information not in the sources. Both are hallucinations, but they often need different fixes, such as better instructions or better retrieval.
Learn more: How to Evaluate a RAG System: Retrieval and Answer Quality
42. What changes when an AI system serves many customers (multi-tenant)?
Each customer’s data, prompts, indexes and logs must be isolated, with tenant identity enforced in retrieval and tools. Costs and rate limits need per-tenant tracking, so one customer cannot use all capacity. Customers may need different models, settings or data locations. Evaluation should include tenant-specific test sets for important customers, and deletions must remove the tenant’s data from indexes and caches.
Test isolation explicitly: create two test tenants with different data and confirm that neither can retrieve, see in logs or influence the other’s results.
43. How do you improve retrieval quality beyond basic vector search?
Use hybrid search and a reranker. Improve chunks: split by structure, add titles and context to each chunk, and remove boilerplate. Use metadata filters such as product or date. Rewrite or expand queries, including splitting multi-part questions. For connected information, consider a knowledge graph or links between documents. Measure every change against a retrieval evaluation set, since improvements are data-specific.
Start by looking at failed questions: often the right passage is missing, badly chunked or buried in boilerplate, and fixing the data helps more than a new search method.
Learn more: What Is RAG? Retrieval-Augmented Generation Explained with Code
44. How would you migrate a production AI feature to a new model?
Run the new model on your full evaluation set and compare quality, cost and latency. Check prompts, because a new model may need adjustments. Then shadow-test on real traffic without showing results, compare outputs, and roll out gradually to a share of users with monitoring and a quick rollback. Keep the old version available until metrics confirm the new one is at least as good.
Compare cost and latency on real traffic too, since a newer model may use more tokens for the same task, and watch user feedback closely during the first days after rollout.
45. What would you put in an AI system’s incident runbook?
A useful runbook covers:
- Detection: alerts on errors, cost, latency, quality and safety flags.
- Switching off: feature flags to turn the AI feature off quickly.
- Fallbacks: a backup model, or a path without AI.
- Contacts: who to call at the model provider.
- Investigation: how to find affected requests from traces.
- Communication: how to tell affected users.
- Follow-up: adding the failure to the evaluation set after the fix.
Practise it before you need it.
46. How do you work with data privacy rules in AI applications?
Know what personal data flows into prompts, logs, caches and indexes, and minimise it: mask or remove what the task does not need. Check where the model provider processes and stores data, and their retention and training terms. Respect deletion requests across indexes and logs. Restrict access to traces, and document the processing for privacy reviews. When rules are strict, use models hosted in the required region or self-host.
Involve your privacy or legal team early, and keep a simple record of which data goes to which model and why.
47. What is cache-augmented generation (CAG), and when would you use it instead of RAG?
CAG skips retrieval. The whole knowledge base is placed in the model’s context once, the model’s processed version of it (its key-value cache, or the provider’s prompt cache) is saved, and every question is answered against that cached context. RAG instead searches for a few relevant chunks for each question and puts only those in the prompt.
CAG fits when all of these are true:
- the knowledge is small enough to fit comfortably in the context window;
- it changes rarely, such as a product manual or one set of policies;
- every user is allowed to see all of it;
- many questions are asked against the same content, so the cache is reused.
It removes retrieval errors and the moving parts of an index, and prompt caching keeps repeated requests cheaper and faster.
RAG remains the better choice for large or frequently changing collections, for content with per-user permissions, and when answers must cite specific passages. Models can also miss details buried in the middle of a very long context, so measure answer quality before relying on CAG. Many systems combine the two: a cached core of stable facts plus retrieval for everything else.
Learn more: RAG vs CAG: Retrieval vs Cache-Augmented Generation
48. Tell me about an AI project you built end to end.
Describe the problem and users, why AI was the right tool, the design (model, retrieval, tools), how you evaluated it with numbers, what failed and how you fixed it, and the result in production, such as time saved or tickets resolved. Mention cost, latency and safety decisions. Interviewers look for evaluation discipline and production thinking more than the choice of framework.
Be ready for follow-up questions on your evaluation: how many test cases, how you labelled them, and what score you reached before launch.
49. What is the difference between a demo and a production AI system?
A demo works on a few chosen examples. A production system works reliably on real, messy inputs at scale: it has an evaluation set and regression tests, handles errors and timeouts, enforces permissions and security, controls cost and latency, logs and monitors behaviour, has fallbacks, and is improved continuously from real failures. Most of an AI engineer’s work is the gap between the two.
In interviews, giving an example of something that worked in the demo but failed with real users, and how you fixed it, shows you understand this gap from experience.
50. What is the difference between RAG and agentic RAG?
Classic RAG is a fixed pipeline: take the question, search the knowledge base once, add the best chunks to the prompt and answer. Agentic RAG lets the model control retrieval. It decides whether to search, rewrites vague questions, chooses between sources such as document search, a SQL tool or the web, checks whether the results are enough, and searches again before answering.
Agentic RAG handles questions that one search cannot answer. “Did our refund policy for business customers change since last year, and how many refunds did we issue last month?” needs two document searches and a database query. The price is higher and less predictable latency and cost, and harder testing, because the steps differ between runs.
How to approach it in practice:
- Start with classic RAG and fix chunking and search quality first; an agent that searches badly just searches badly more often.
- Add agentic steps, such as query rewriting or one extra search when results score poorly, where evaluation shows a single search is not enough.
- Limit the number of tool calls and the time per request.
- Enforce the user’s permissions in every retrieval tool.
- Evaluate the path the agent took, not only the final answer.
Learn more: RAG vs Agentic RAG: What Is the Difference?
How to prepare
- Build two projects and measure them: a RAG application over a real document set, and an agent with two or three tools. Give each an evaluation set and know your numbers.
- Practise AI system design: support assistants, document Q&A, and agents that take actions. Cover data, retrieval, tools, guardrails, evaluation, monitoring and cost.
- Know the trade-offs: workflow vs agent, prompting vs RAG vs fine-tuning, API vs self-hosted model, bigger vs cheaper model.
- Revise LLM fundamentals with the LLM interview questions.
- For customer-facing AI roles, read the forward deployed engineer interview questions.