Vector Databases and Vector Indexes Explained for RAG

Once documents are turned into embeddings, retrieval means finding the stored vectors closest to the question’s vector. With a few thousand chunks, you can simply compare the question with every vector. With millions, that becomes too slow. Vector indexes solve this by searching only a small, likely part of the data, and vector databases wrap those indexes with storage, filtering and updates.

What a vector database stores

For each chunk, a vector store keeps:

• the vector (embedding),
• the text or a reference to it,
• metadata: source, date, language, product, and access permissions,
• an ID, so the chunk can be updated or deleted.

It answers queries such as “the 10 nearest vectors to this one, where language is English and the user is allowed to see them”.

Exact search vs approximate search

Exact (brute-force) search compares the query with every vector. It always finds the true nearest neighbours, and it is fine for small collections. Its cost grows with the number of vectors.

Approximate nearest-neighbour (ANN) search builds an index that skips most vectors. It is much faster, and it usually finds most but not always all of the true nearest neighbours. The fraction it finds is called recall, and you trade a little recall for a lot of speed.

How IVF works

An inverted file index (IVF) groups vectors into clusters, each with a centre. At query time, it finds the clusters whose centres are closest to the query and searches only inside them. Searching more clusters gives better recall but takes more time.

This example builds a simple IVF index over 50,000 synthetic vectors and compares it with exact search:

import numpy as np

rng = np.random.default_rng(1)
n, dim = 50_000, 64
# Fake document embeddings in 100 topic groups, normalised so a dot product is cosine similarity
centers = rng.normal(size=(100, dim))
docs = centers[rng.integers(0, 100, n)] + 0.3 * rng.normal(size=(n, dim))
docs /= np.linalg.norm(docs, axis=1, keepdims=True)
queries = docs[rng.choice(n, 200, replace=False)] + 0.05 * rng.normal(size=(200, dim))
queries /= np.linalg.norm(queries, axis=1, keepdims=True)

def exact_top10(q):
    return np.argsort(-(docs @ q))[:10]                    # compares the query with every vector

# A simple approximate index (IVF): group vectors into clusters, then search only the closest clusters
n_lists = 100
centroids = docs[rng.choice(n, n_lists, replace=False)]
for _ in range(5):                                         # a few rounds of k-means
    assign = np.argmax(docs @ centroids.T, axis=1)
    centroids = np.array([docs[assign == c].mean(axis=0) if (assign == c).any() else centroids[c]
                          for c in range(n_lists)])          # keep a cluster's old centre if it is empty
    centroids /= np.linalg.norm(centroids, axis=1, keepdims=True)
assign = np.argmax(docs @ centroids.T, axis=1)
lists = [np.where(assign == c)[0] for c in range(n_lists)]

def ivf_top10(q, n_probe):
    probe = np.argsort(-(centroids @ q))[:n_probe]          # closest clusters only
    candidates = np.concatenate([lists[c] for c in probe])
    return candidates[np.argsort(-(docs[candidates] @ q))[:10]], len(candidates)

truth = [set(exact_top10(q)) for q in queries]
for n_probe in (1, 4, 16):
    found, compared = 0, 0
    for q, t in zip(queries, truth):
        res, c = ivf_top10(q, n_probe)
        found += len(t & set(res)); compared += c
    print(f"IVF, {n_probe:2} clusters searched: recall@10 = {found / (10 * len(queries)):.2f}, "
          f"vectors compared per query = {compared // len(queries):,} of {n:,}")

IVF,  1 clusters searched: recall@10 = 0.94, vectors compared per query = 621 of 50,000
IVF,  4 clusters searched: recall@10 = 1.00, vectors compared per query = 2,101 of 50,000
IVF, 16 clusters searched: recall@10 = 1.00, vectors compared per query = 8,031 of 50,000

Searching one cluster compared about 1% of the vectors and still found 94% of the true top 10. Searching four clusters found all of them while comparing about 4%. Real data is messier than these clean synthetic clusters, so always measure recall on your own vectors.

Other index types

• HNSW (hierarchical navigable small world) builds a graph where each vector links to its near neighbours, with a few long-range links on upper layers. A search walks the graph towards the query. It is fast and accurate, and the most common default, but uses extra memory for the links.
• Quantisation, such as product quantisation, compresses vectors so more fit in memory, at some cost in accuracy. It is often combined with IVF, with exact re-scoring of the best candidates.
• Flat indexes are exact search, still the right choice for small collections.

Choosing a vector store

Option	Fits when
In-memory library (a flat or ANN index inside your app)	Small to medium collections, prototypes, batch jobs
Vector extension of an existing database, such as PostgreSQL with pgvector	You already run that database and want SQL, joins and transactions with your vectors
Search engine with vector support	You want keyword and vector search together (hybrid)
Dedicated vector database	Very large collections, heavy query load, many tenants

Check filtering (can it filter by metadata during the search, not only after?), update speed, hybrid search support, and how it fits your existing stack. Start simple; many RAG systems never need more than a vector extension of a database they already run.

Common mistakes

1. Choosing by benchmark speed without measuring recall on your data.
2. Filtering after the search, so a strict filter leaves too few results; prefer stores that filter during the search.
3. Mixing vectors from different embedding models in one index.
4. Forgetting memory: HNSW graphs and full-precision vectors can need a lot of RAM at scale.

Key takeaway

Vector databases store embeddings with text and metadata, and use approximate nearest-neighbour indexes such as IVF and HNSW to search millions of vectors quickly by skipping most of them. Measure the recall you get for the speed you gain, and choose the simplest store that meets your scale and filtering needs. Next: RAG in production.