CareerGuide 21 of 36
Career guide · Guide 21 of 36
Top 50 LLM Interview Questions and Answers
Top 50 LLM interview questions with simple answers: tokens, embeddings, transformers and attention, RAG, fine-tuning and LoRA, evaluation, safety and serving.
On this page
These 50 questions cover what interviews for AI engineer, LLM engineer, ML engineer and generative AI roles ask about large language models: how they work, how to adapt them (prompting, RAG, fine-tuning), how to evaluate them, and how to run them safely in production. Each answer is written in plain words and gets straight to the point. Questions are grouped from fresher level to experienced.
Question list
Show all 50 questions
- What is a large language model (LLM)?
- What is a token, and why does tokenization matter?
- What is a context window?
- What is the difference between pre-training and fine-tuning?
- What are embeddings?
- What is temperature in an LLM?
- What is a hallucination?
- What is prompt engineering?
- What is zero-shot vs few-shot prompting?
- What is the transformer architecture?
- What is the difference between an open-weight and a closed model?
- What is a system prompt?
- What is retrieval-augmented generation (RAG)?
- What are parameters in an LLM?
- How do LLMs generate text one token at a time?
- How does self-attention work?
- Why do transformers need positional information?
- What is the difference between encoder-only, decoder-only and encoder-decoder models?
- What are top-k and top-p sampling?
- What is instruction tuning?
- What is the difference between a base model and an instruct (chat) model?
- What is RLHF?
- When should you use RAG and when should you fine-tune?
- What is LoRA?
- What is quantization?
- What is the KV cache?
- How do you get reliable structured output, such as JSON, from an LLM?
- What is function calling or tool calling?
- What is chain-of-thought prompting?
- How do you evaluate an LLM application?
- What is LLM-as-a-judge, and what are its risks?
- What is prompt injection?
- How do you reduce LLM cost and latency?
- What is a vector database?
- What is semantic search, and how is it different from keyword search?
- What is chunking in RAG, and how do you choose a chunk size?
- What is reranking in retrieval?
- What is a mixture-of-experts (MoE) model?
- Why does attention cost grow with context length, and how is it reduced?
- How would you design evaluation for a production RAG system?
- How do you serve an open-weight LLM at scale?
- What is speculative decoding?
- What are the main ways to align a model with human preferences, and how do they differ?
- How do you detect and handle data leakage in LLM evaluation?
- How do you fine-tune a model safely for a specific task?
- What is the difference between an LLM and an SLM?
- How would you build guardrails around an LLM application?
- What are scaling laws?
- Why do LLMs struggle with long contexts, and how do you design around it?
- How do you monitor an LLM application in production?
Beginner LLM interview questions (freshers)
1. What is a large language model (LLM)?
A large language model is a neural network trained on a very large amount of text to predict the next token (a word or part of a word) given the text before it. By repeating that prediction, it can write answers, summaries, code and translations. Modern LLMs use the transformer architecture and have billions of parameters. After the first training stage, most are further trained to follow instructions and to be helpful and safe, which turns a text predictor into a useful assistant.
2. What is a token, and why does tokenization matter?
A token is the unit of text a model reads and writes: often a whole common word, part of a rarer word, a punctuation mark or a space. A tokenizer splits text into tokens and maps each to a number. It matters because models have limits measured in tokens (the context window), most APIs charge per token, and tokenization affects behaviour: numbers, rare words and some languages split into many tokens, which can make counting, spelling and arithmetic harder for the model.
Learn more: Tokens and Tokenization in LLMs, Explained Simply
3. What is a context window?
The context window is the maximum number of tokens a model can consider at once: the prompt, any documents or conversation history you include, and the answer it writes. Anything outside the window is invisible to the model. Larger windows let you include more documents, but cost more and can make it harder for the model to use information buried in the middle. Applications manage the window by summarising history or retrieving only the relevant parts.
Learn more: Context Windows and Hallucinations: Why LLMs Make Things Up
4. What is the difference between pre-training and fine-tuning?
Pre-training teaches a model general language and knowledge by predicting the next token on a huge, broad dataset; it is very expensive and done once by the model builder. Fine-tuning continues training a pre-trained model on a much smaller, targeted dataset to change its behaviour: following instructions, using a format, or learning a domain’s style. Fine-tuning is far cheaper and needs examples of the behaviour you want.
Between the two, most chat models also go through preference training, such as RLHF, which makes their answers more helpful and safer.
Learn more: Fine-Tuning LLMs and LoRA Explained Simply
5. What are embeddings?
An embedding is a list of numbers (a vector) that represents the meaning of a piece of text, so that texts with similar meaning have vectors close to each other. Embedding models turn sentences or documents into these vectors. They power semantic search, recommendations, clustering and retrieval-augmented generation: you embed your documents once, embed each question, and find the documents whose vectors are closest to the question’s.
For example, “How do I reset my password?” and “I forgot my login details” share few words, but their embeddings are close, so a search finds the same help article for both.
Learn more: What Are Embeddings? Vectors, Similarity and Semantic Search Explained
6. What is temperature in an LLM?
Temperature controls how random the model’s word choices are. The model gives a probability for every possible next token; a low temperature makes it pick the most likely tokens almost every time, giving focused, repeatable answers, while a higher temperature spreads the choice to less likely tokens, giving more varied and creative text. For extraction, classification and code, use a low temperature; for brainstorming, a higher one.
Temperature does not make a model more or less knowledgeable: a low temperature gives the same answer more consistently, but that answer can still be wrong.
Learn more: How LLMs Generate Text: Temperature, Top-k and Top-p Explained
7. What is a hallucination?
A hallucination is a confident answer that is false or not supported by the given information, such as an invented fact, citation or function name. It happens because the model generates plausible text, not checked facts. You reduce it by giving the model the relevant information (retrieval), asking it to answer only from that information and to say when it does not know, lowering temperature, requiring citations, and checking outputs automatically or with people for important uses.
Learn more: Context Windows and Hallucinations: Why LLMs Make Things Up
8. What is prompt engineering?
Prompt engineering is designing the instructions and context you give a model to get reliable, useful output. Good prompts state the task and audience, give the needed context, show the exact output format, include a few examples when the task is unusual, and say what to do when information is missing. It is iterative: you test prompts on real examples, measure the results and adjust, just like code.
Small details matter: putting instructions before long documents, using clear section labels, and asking for a fixed output structure often improve results more than clever wording.
Learn more: Prompt Engineering and Structured Output: Getting Reliable Answers from LLMs
9. What is zero-shot vs few-shot prompting?
Zero-shot prompting asks the model to do a task with instructions only, no examples. Few-shot prompting includes a few worked examples of inputs and correct outputs in the prompt, so the model copies the pattern. Few-shot helps when the format or judgement is unusual, such as a specific labelling scheme. It costs more tokens, and the examples should be varied and correct, because the model will copy their mistakes too.
Learn more: Prompt Engineering and Structured Output: Getting Reliable Answers from LLMs
10. What is the transformer architecture?
The transformer is the neural network design behind almost all modern LLMs. Its key idea is attention: each token looks at the other tokens in the context and decides how much each one matters for understanding it. Transformers process all tokens of the input in parallel during training, which made training on huge datasets practical. A transformer is a stack of layers, each with an attention block and a feed-forward block.
Learn more: Attention and Transformers Explained Simply
11. What is the difference between an open-weight and a closed model?
An open-weight model’s trained parameters are published, so you can download it, run it on your own hardware and fine-tune it, subject to its licence. A closed model is only available through the provider’s API or apps. Open-weight models give control over data, cost and customisation, but you must host and operate them; closed models are easier to start with and are often the most capable, but you depend on the provider.
12. What is a system prompt?
A system prompt is a set of instructions given to the model separately from the user’s messages, usually by the application developer. It sets the role, rules, tone, output format and limits for the whole conversation, for example “You are a support assistant for our billing product; answer only from the provided documents”. Users normally do not see it. It guides behaviour strongly but is not a security boundary, so sensitive rules still need checks in code.
Learn more: Prompt Engineering and Structured Output: Getting Reliable Answers from LLMs
13. What is retrieval-augmented generation (RAG)?
RAG gives a model relevant information at question time instead of relying only on what it learned in training. The application searches a knowledge source, such as your documents, for passages related to the question, puts them in the prompt, and asks the model to answer from them, ideally with citations. RAG keeps answers current and specific to your data and reduces hallucinations, without retraining the model.
Its quality depends mostly on retrieval: if the right passage is not found, the model cannot answer correctly. See RAG vs fine-tuning for when to use each.
Learn more: What Is RAG? Retrieval-Augmented Generation Explained with Code
14. What are parameters in an LLM?
Parameters are the numbers inside the neural network, the weights, that are adjusted during training. They store what the model has learned about language and the world. Model size is usually given in parameters, such as 7 billion or 70 billion. More parameters generally means more capability, but also more memory and compute to run, which is why smaller, well-trained models are often preferred when they are good enough.
As a rough guide, each parameter takes 2 bytes in 16-bit precision, so a 7-billion-parameter model needs about 14 GB of memory just for its weights, before the memory needed to process requests.
15. How do LLMs generate text one token at a time?
Generation is a loop. The model reads the prompt and produces a probability for every possible next token. A decoding method picks one token, for example the most likely one or a random choice shaped by temperature. That token is added to the text, and the model predicts the next one, until it produces an end token or reaches a length limit. Because each step depends on earlier choices, small changes early can change the whole answer.
Learn more: Tokens and Tokenization in LLMs, Explained Simply
Intermediate LLM interview questions
16. How does self-attention work?
For each token, the model creates three vectors: a query, a key and a value. It compares the token’s query with every token’s key to get attention scores, turns the scores into weights with softmax, and takes a weighted sum of the values. So each token’s new representation mixes in information from the tokens most relevant to it. Multi-head attention does this several times in parallel, so different heads can focus on different relationships, such as grammar or meaning.
Learn more: Attention and Transformers Explained Simply
17. Why do transformers need positional information?
Attention on its own treats the input as a set: it does not know which token came first. Word order matters for meaning, so transformers add position information to each token. Early models added fixed or learned position vectors; many modern models use rotary position embeddings, which encode relative positions inside the attention calculation and handle long contexts better.
For example, “the dog bit the man” and “the man bit the dog” contain the same words; without position information, attention alone could not tell them apart. How positions are encoded also affects how well a model handles prompts longer than those it was trained on.
Learn more: Attention and Transformers Explained Simply
18. What is the difference between encoder-only, decoder-only and encoder-decoder models?
Encoder-only models read the whole input in both directions and produce representations, which suits classification and embeddings. Decoder-only models generate text left to right, each token seeing only earlier tokens; most chat LLMs are decoder-only. Encoder-decoder models encode an input and then decode an output, which suits translation and summarisation. Decoder-only models became dominant because one design handles almost any text task.
Encoder-only models, in the BERT family, are still widely used for embeddings, search and classification, because they are small and fast.
19. What are top-k and top-p sampling?
Both limit which tokens the model can choose from before sampling. Top-k keeps only the k most likely tokens. Top-p, or nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p, such as 0.9, so the set grows when the model is unsure and shrinks when it is confident. They stop the model from picking very unlikely tokens while keeping some variety. They are usually combined with temperature.
Learn more: How LLMs Generate Text: Temperature, Top-k and Top-p Explained
20. What is instruction tuning?
Instruction tuning fine-tunes a pre-trained model on many examples of instructions paired with good responses, such as “summarise this email” with a good summary. It teaches the model to follow requests instead of just continuing text. It is usually the first step after pre-training to make a chat assistant, followed by preference training to make answers more helpful and safe.
The quality and variety of the instruction examples matter more than their number: a few thousand excellent, diverse examples can work better than many thousands of mediocre ones. Instruction tuning is a form of supervised fine-tuning.
Learn more: Fine-Tuning LLMs and LoRA Explained Simply
21. What is the difference between a base model and an instruct (chat) model?
A base model is the result of pre-training only: it continues text in the style of its training data, so a question may be answered with more questions. An instruct or chat model has been further trained with instruction tuning and preference training to follow requests, hold a conversation, use a chat format and refuse harmful requests. Applications almost always use instruct models; base models are mainly used as a starting point for custom fine-tuning.
22. What is RLHF?
Reinforcement learning from human feedback trains a model using people’s preferences. People compare pairs of model answers and pick the better one; a reward model learns to predict those preferences; then the LLM is trained to produce answers the reward model scores highly, while staying close to its original behaviour. It improves helpfulness and safety. Simpler methods such as direct preference optimisation learn from the same comparison data without a separate reward model.
Learn more: Fine-Tuning LLMs and LoRA Explained Simply
23. When should you use RAG and when should you fine-tune?
Use RAG when the model needs knowledge: facts that change, private documents, or answers that must cite sources. Use fine-tuning when you need behaviour: a consistent format, tone, a specialised task, or a smaller model that performs a narrow job well. Often you combine them: fine-tune for behaviour and use RAG for knowledge. Try prompting and RAG first, because they are cheaper and easier to update.
See RAG vs fine-tuning for a full comparison with a decision table.
Learn more: Fine-Tuning LLMs and LoRA Explained Simply
24. What is LoRA?
LoRA (low-rank adaptation) is a cheap way to fine-tune a large model. Instead of updating all the weights, it freezes the original model and trains small extra matrices that add a low-rank change to selected layers. This cuts memory and storage needs dramatically, and you can keep many small adapters for different tasks on one base model. QLoRA goes further by loading the frozen base model in 4-bit precision.
A LoRA adapter for a large model can be only tens of megabytes, compared with many gigabytes for the full model, which makes it practical to train and store many task-specific versions.
Learn more: Fine-Tuning LLMs and LoRA Explained Simply
25. What is quantization?
Quantization stores a model’s weights, and sometimes its activations, with fewer bits, for example 8 or 4 bits instead of 16. The model uses less memory and often runs faster, so large models fit on smaller GPUs or even laptops. The cost is a small loss of accuracy, which depends on the method and the bit width. It is the most common way to make open-weight models cheaper to serve.
26. What is the KV cache?
During generation, each new token needs attention over all earlier tokens. The key and value vectors of earlier tokens do not change, so the model stores them in a key-value cache instead of recomputing them at every step. This makes generation much faster, but the cache uses memory that grows with context length and the number of parallel requests, which is a main limit on how many users a GPU can serve.
27. How do you get reliable structured output, such as JSON, from an LLM?
Describe the exact schema in the prompt, with an example, and keep the temperature low. Better, use the provider’s structured output or JSON mode, or constrained decoding in open-source servers, which only allows tokens that fit a schema. Always validate the result in code, and on failure retry with the error message or fall back. Keep schemas simple and give fields clear names and descriptions.
Also handle the case where the input simply does not contain the information: allow null values or an explicit “not found” field, so the model is not pushed to invent a value.
Learn more: Prompt Engineering and Structured Output: Getting Reliable Answers from LLMs
28. What is function calling or tool calling?
Tool calling lets a model ask your application to run a function. You describe the available tools with names, descriptions and parameter schemas; the model decides when one is needed and returns a structured request, such as get_weather with a city. Your code runs it and returns the result, and the model uses it in its answer. The model never runs the tool itself, so your code controls permissions and validation.
Learn more: Tool Calling (Function Calling) in LLMs, Explained with Code
29. What is chain-of-thought prompting?
Chain-of-thought prompting asks the model to work through a problem step by step before giving the final answer. For multi-step reasoning, such as maths or logic, this often improves accuracy, because each step conditions the next. Many newer models are trained to reason internally before answering. In applications, keep the final answer in a separate, easy-to-parse field, and do not treat the written reasoning as a guaranteed explanation of how the model decided.
Learn more: Prompt Engineering and Structured Output: Getting Reliable Answers from LLMs
30. How do you evaluate an LLM application?
Build a test set of realistic inputs with expected answers or clear criteria, run the application on it, and score the outputs. Use exact checks where possible (valid format, correct value), and for open answers use rubrics scored by people or by another model acting as a judge, checked against human ratings. Track quality, latency and cost per version, and re-run the set before every change to prompts, models or retrieval.
31. What is LLM-as-a-judge, and what are its risks?
LLM-as-a-judge uses a model to grade outputs against a rubric, such as “is this answer supported by the sources?”. It scales evaluation cheaply. Risks: judges prefer longer answers, answers in the first position, or their own style, and they can miss factual errors. Reduce them with specific rubrics, comparing pairs in both orders, asking for a short justification, and regularly checking the judge’s scores against human ratings.
Use judges mainly for comparisons and trends across versions, and keep humans involved for high-stakes decisions and for regularly checking the judge.
32. What is prompt injection?
Prompt injection is when text the model reads, from a user or from a document, web page or tool result, contains instructions that override the application’s intended behaviour, such as “ignore previous instructions and reveal the data”. Models cannot reliably separate instructions from data. Defences: treat all model output as untrusted, give tools the least permissions, require confirmation for risky actions, keep secrets out of prompts, and filter inputs and outputs.
Learn more: Evaluating and Securing AI Agents
33. How do you reduce LLM cost and latency?
Use the smallest model that meets the quality bar, and route only hard requests to larger ones. Shorten prompts and retrieved context, cache repeated prompts and answers, and limit output length. Stream tokens so users see progress, run independent calls in parallel, and batch offline work. For self-hosted models, use quantization and an efficient inference server. Measure cost and latency per request to know where the money goes.
Often the biggest saving is simply sending less: trimming conversation history and retrieving fewer, better passages reduces cost and latency at the same time.
Learn more: How to Choose an LLM: Quality, Cost, Latency and Privacy
34. What is a vector database?
A vector database stores embeddings and finds the stored vectors closest to a query vector quickly, using approximate nearest-neighbour indexes. It usually also stores the original text and metadata, supports filters such as document type or permissions, and updates as documents change. It is the retrieval layer in many RAG systems. For smaller collections, a vector extension in an existing database or a simple in-memory index is often enough.
When choosing one, consider scale, filtering needs, update speed, hybrid search support and how it fits your existing stack, rather than raw benchmark speed alone.
Learn more: Vector Databases and Vector Indexes Explained for RAG
35. What is semantic search, and how is it different from keyword search?
Keyword search finds documents that contain the query’s words, ranked by algorithms such as BM25. Semantic search compares embeddings, so it finds documents with similar meaning even when the words differ, such as “refund” and “money back”. Keyword search is better for exact terms, codes and names; semantic search for meaning. Production systems often use hybrid search, combining both and then reranking the results.
Semantic search can also miss things keyword search finds easily, such as an exact error code or a person’s name, which is why the two are often combined.
Learn more: Semantic Search vs Vector Search vs Keyword Search
36. What is chunking in RAG, and how do you choose a chunk size?
Chunking splits documents into smaller passages before embedding them, because whole documents are too long and too broad to retrieve precisely. Chunks that are too small lose context; chunks that are too large add noise and cost. Split on natural boundaries such as headings and paragraphs, add a little overlap, and keep metadata like title and section. Choose the size by testing retrieval quality on real questions.
A common starting point is a few hundred tokens per chunk with a small overlap, then adjusting based on your evaluation results.
Learn more: Chunking Strategies for RAG: How to Split Documents
37. What is reranking in retrieval?
Reranking is a second step after the first, fast search. The first search returns, say, 50 candidates using embeddings or keywords. A reranker, usually a cross-encoder model that reads the question and each passage together, scores them more accurately, and only the top few go into the prompt. It improves answer quality noticeably for a small extra cost and delay.
Rerankers are slower per document than embedding search, so they only score the shortlist, not the whole collection. Measuring retrieval before and after adding one shows whether it is worth the extra latency for your data.
Learn more: Hybrid Search and Reranking for RAG, Explained with Code
38. What is a mixture-of-experts (MoE) model?
A mixture-of-experts model replaces some feed-forward layers with many smaller expert networks and a router that sends each token to only a few of them. The model has many parameters in total, but uses only a fraction for each token, so it can be more capable for the same compute per token. The trade-offs are higher total memory, more complex training and serving, and the need to balance load across experts.
Advanced LLM interview questions (experienced)
39. Why does attention cost grow with context length, and how is it reduced?
Standard attention compares every token with every other token, so compute and memory grow with the square of the sequence length. Doubling the context makes attention roughly four times as expensive. Reductions include memory-efficient kernels such as FlashAttention (same result, far less memory traffic), grouped-query attention (smaller KV cache), sliding-window or sparse attention patterns, and KV cache compression. Applications also avoid sending context they do not need.
This is also why very long prompts cost more and respond more slowly, even when the provider supports a large context window.
Learn more: Attention and Transformers Explained Simply
40. How would you design evaluation for a production RAG system?
Evaluate retrieval and generation separately. For retrieval, build questions with known relevant passages and measure recall and ranking quality. For generation, score faithfulness to the retrieved passages, answer correctness and completeness, citation accuracy, and refusals when the answer is not in the sources. Combine automatic scorers with human review of a sample, track results per release, and add failed production questions to the test set.
Include questions that should not be answered from the documents, to check that the system declines instead of making something up.
Learn more: How to Evaluate a RAG System: Retrieval and Answer Quality
41. How do you serve an open-weight LLM at scale?
Use an inference server built for LLMs, with continuous batching (adding new requests to running batches), efficient KV cache memory management and optimised attention kernels. Choose precision and quantization to fit the GPU memory, use tensor parallelism when a model does not fit on one GPU, and autoscale replicas on queue length. Monitor time to first token, tokens per second, errors and GPU utilisation.
Plan capacity from real traffic: average prompt and answer lengths, peak requests per second and latency targets decide how many GPUs you need, more than the model size alone.
42. What is speculative decoding?
Speculative decoding speeds up generation with a small, fast draft model. The draft model proposes several tokens ahead; the large model then checks them all in one parallel pass and accepts the ones it agrees with, correcting the first one it rejects. When the draft is often right, several tokens are produced for the cost of one large-model step, and the output distribution matches the large model’s.
It works best when the text is predictable, such as code or structured output, because the draft model is right more often there.
43. What are the main ways to align a model with human preferences, and how do they differ?
After instruction tuning, models learn from comparisons between better and worse answers. RLHF trains a reward model on human comparisons and optimises the LLM against it with reinforcement learning. Direct preference optimisation and similar methods skip the reward model and train directly on the comparison pairs, which is simpler and more stable. Some approaches use AI-generated feedback guided by written principles to scale the comparisons.
All of these methods depend heavily on the quality of the comparison data: unclear or inconsistent preferences produce a model that is confidently unhelpful in new ways.
44. How do you detect and handle data leakage in LLM evaluation?
Leakage happens when test questions or very similar ones were in the training data or prompt examples, so scores look better than real performance. Keep test sets private and versioned, create new questions from recent or internal data, check for near-duplicates between examples and tests, and compare scores on public benchmarks with scores on your own fresh tasks. Treat public leaderboard results as a rough guide only.
A simple check is to change the wording or numbers in a test question: if accuracy drops sharply, the model may have memorised the original rather than solved it.
45. How do you fine-tune a model safely for a specific task?
Start with a clear task and a baseline from prompting. Collect high-quality, varied examples, and remove personal or confidential data you are not allowed to use. Hold out a test set. Use parameter-efficient fine-tuning such as LoRA, with a small learning rate, and watch validation results to avoid overfitting. Check that general abilities and safety behaviour did not degrade, and compare cost and quality with the prompting baseline before shipping.
Learn more: Fine-Tuning LLMs and LoRA Explained Simply
46. What is the difference between an LLM and an SLM?
An LLM (large language model) is a general-purpose model, usually with tens to hundreds of billions of parameters, served from data-centre GPUs, often through a provider’s API. An SLM (small language model) is the same kind of transformer made much smaller, typically from under 1 billion to around 10 billion parameters, so it can run cheaply on one GPU, a laptop or even a phone. There is no official size cut-off.
The trade-off is capability against cost:
- Large models know more and handle open-ended questions and multi-step reasoning better.
- Small models are faster, cheaper and easier to run privately, and on narrow tasks such as classification, extraction or routing they can match a large model, especially after fine-tuning.
Many small models are built with distillation: a small “student” model is trained to reproduce the outputs of a large “teacher”, keeping much of its quality at a fraction of the size and cost.
A common production design is a router: a small model handles the easy majority of requests and only hard or uncertain ones go to a large model. To choose, build a test set of real examples and pick the cheapest model that meets the quality bar.
For example, a small model fine-tuned on a few thousand labelled support tickets can sort new tickets into billing, delivery or login almost as accurately as a large model, at a much lower cost per request.
Learn more: LLM vs SLM: Large vs Small Language Models Compared
47. How would you build guardrails around an LLM application?
Use layers. Before the model: validate and filter inputs, detect prompt injection and personal data, and limit what context is included. Around the model: clear system rules and tool permissions. After the model: validate format, check claims against sources, filter unsafe content, and require human approval for risky actions such as payments or emails. Log decisions, monitor for abuse, and test the guardrails with adversarial examples.
Measure guardrails like any feature: track how often they block real misuse and how often they wrongly block normal users, and tune them to keep both low.
Learn more: Evaluating and Securing AI Agents
48. What are scaling laws?
Scaling laws are measured relationships showing that a model’s loss falls predictably as you increase parameters, training data and compute. They guide how to spend a training budget: later work showed many earlier models were too large for their data, and that more training tokens per parameter gives better results for the same compute. In practice, models are now often trained on far more data than that optimum, because smaller models are cheaper to run.
49. Why do LLMs struggle with long contexts, and how do you design around it?
Models trained mostly on shorter sequences can use information less reliably when it is far away or in the middle of a very long prompt, and long prompts are slow and expensive. Design around it: retrieve only the relevant parts, put the most important instructions and evidence near the start or end, summarise long history, split large tasks into steps, and test with your real document lengths rather than trusting the advertised limit.
Learn more: Context Windows and Hallucinations: Why LLMs Make Things Up
50. How do you monitor an LLM application in production?
Log each request with its prompt version, model, retrieved documents, tool calls, output, latency, token counts and cost, with personal data masked. Track quality signals: user feedback, automatic checks on samples, refusal and error rates. Alert on cost spikes, latency, and drops in quality scores. Review real failures regularly and add them to the evaluation set, so each fix is tested in future releases.
Track behaviour by prompt and model version, so that when quality drops you can see which change caused it.
How to prepare
- Know the basics cold: tokens, embeddings, attention, context windows and sampling come up in almost every interview, at every level.
- Build one small RAG application end to end: load documents, chunk, embed, retrieve, answer with citations, and evaluate on 20 real questions. Most practical questions refer to this kind of system.
- Be ready to compare options with their trade-offs: prompting vs RAG vs fine-tuning, a large vs a small model, keyword vs semantic search.
- Show production thinking: evaluation sets, monitoring, cost, latency, prompt injection and guardrails separate experienced candidates from beginners.
- For questions about building agents, RAG systems and production AI, see the Top 50 AI engineer interview questions; for customer-facing AI roles, the forward deployed engineer interview questions.