How to Choose an LLM: Quality, Cost, Latency and Privacy

There is no single best language model. The right choice depends on your task, your budget, how fast the answer must arrive, and where your data is allowed to go. Teams that choose well do the same thing every time: test a few candidates on their own examples, then pick the cheapest model that is good enough.

The factors that matter

Factor	Question to ask
Quality	How often is it right on our task, measured on our own test set?
Cost	What will it cost per request and per month at our expected traffic?
Latency	How fast is the first token, and the whole answer? Does the user wait for it?
Context window	Can it fit our instructions, documents and answer?
Features	Does it support tool calling, structured output, images or the languages we need?
Privacy and location	Where is data processed and stored? Is it used for training? Do we need to self-host?
Reliability	What are the rate limits and uptime? Is there a fallback?

Test on your own data

Public leaderboards are a starting point, not an answer. A model that ranks first in general can be worse on your format, your language or your domain. Build a test set of 50 to 200 real examples with expected results, run each candidate model on it, and record quality, cost and latency together. This is the same evaluation set you will use later to check every prompt or model change.

Estimate the cost

Cost is mostly tokens: input tokens (your prompt and documents) and output tokens (the answer), each with its own price per million. This calculator uses example prices to show the method:

# Example prices only, to show the method. Check your provider's current price list before using real numbers.
MODELS = {
    "small":  {"in": 0.15, "out": 0.60,  "accuracy": 0.86},   # dollars per million tokens; accuracy on YOUR test set
    "medium": {"in": 1.00, "out": 4.00,  "accuracy": 0.92},
    "large":  {"in": 5.00, "out": 20.00, "accuracy": 0.95},
}
requests_per_day, tokens_in, tokens_out = 50_000, 1_200, 300

def monthly_cost(m, share=1.0):
    per_request = (tokens_in * m["in"] + tokens_out * m["out"]) / 1_000_000
    return per_request * requests_per_day * 30 * share

for name, m in MODELS.items():
    print(f"{name:6}  accuracy {m['accuracy']:.0%}   ${monthly_cost(m):>9,.0f} per month")

# Routing: the small model handles easy requests; hard ones (say 20%) go to the large model.
routed = monthly_cost(MODELS["small"], 0.8) + monthly_cost(MODELS["large"], 0.2)
print(f"routed  small 80% + large 20%   ${routed:>9,.0f} per month")

small   accuracy 86%   $      540 per month
medium  accuracy 92%   $    3,600 per month
large   accuracy 95%   $   18,000 per month
routed  small 80% + large 20%   $    4,032 per month

The large model is more accurate, but 33 times the price of the small one at this traffic. Whether 9 more points of accuracy are worth it depends on the cost of a wrong answer.

Routing: use more than one model

Many requests are easy, and a few are hard. Routing sends easy requests to a small, cheap model and hard ones to a larger model. In the example, sending 80% of traffic to the small model and 20% to the large one costs a fraction of using the large model for everything, while keeping the large model for the cases that need it. The router can be a simple rule (such as question length or type), a classifier, or a check on the small model’s confidence.

API or self-hosted?

• Hosted APIs are the quickest way to start, usually offer the most capable models, and need no infrastructure. You pay per token and depend on the provider.
• Self-hosted open-weight models give control over data location and customisation, and can be cheaper at very high volume. You need GPUs, an inference server and people to run them.

Many teams start with APIs and move a narrow, high-volume task to a self-hosted model later.

Plan for change

New models appear often, and prices change. Put model calls behind one interface in your code, keep your test set ready, and re-evaluate a few times a year. Switching should be a measured decision you can make in days, not a rewrite.

Common mistakes

1. Choosing from a leaderboard without testing on your task.
2. Ignoring output tokens, which often cost several times more than input tokens.
3. Using the largest model everywhere when most requests are easy.
4. Forgetting data rules until a privacy review blocks the launch.

Key takeaway

Choose a model by measuring quality, cost and latency on your own examples, checking context, features and privacy needs, and picking the cheapest model that meets the bar. Use routing to combine models, and keep your code ready to switch. Next: adapting a model to your task with fine-tuning and LoRA.