Why I dropped vector search for my wiki

When I built my LLM wiki à la Karpathy, the reflexive first step was: embeddings in, vector DB alongside, RAG done. A few weeks later I ripped the vector search back out. Not on principle – but because for this knowledge base it was the worse choice. Here is why.

What vector search actually does

Vector search embeds text into high-dimensional vectors and finds the “semantically nearest” chunks by cosine similarity. That is powerful for unstructured, fuzzy queries over huge, inconsistent corpora. A well-maintained personal wiki, though, is exactly the opposite: structured, curated, with metadata.

Where it concretely failed for me

1. Chunking destroys structure. A wiki entry has a title, tags, date, relationships. Hacked into 512-token windows, what’s left is text mush. The boundary between two topics lands in the middle of a chunk.

2. Cosine proximity ≠ relevance. “How do I deploy X?” eagerly pulls every paragraph that says “deploy” a lot – including the stale one from a year ago. Semantic proximity knows no “current” and no “authoritative”.

3. Exact lookups get lost. “Show me entry cbks-arch from March.” That is not a similarity problem, it is a WHERE clause. Embeddings turn it into a guessing game.

4. No clean filtering. “Only tag=infra, only last quarter” – awkward in pure vector search, a one-liner in SQL.

What I do instead: structured retrieval

My wiki lives in PostgreSQL anyway. So I use what’s already there – metadata filters plus full text (BM25-style) instead of embeddings:

-- Candidates via structure + full text, not cosine guessing
SELECT id, title, tags, updated_at,
       ts_rank(search_vector, query) AS rank
FROM wiki_entries,
     plainto_tsquery('english', 'deployment fly.io') AS query
WHERE search_vector @@ query
  AND 'infra' = ANY(tags)         -- metadata filter
  AND updated_at > now() - interval '180 days'  -- current knowledge only
ORDER BY rank DESC
LIMIT 5;

That yields precise, filterable, fresh hits – and, incidentally, the structured metadata right along with them. Exactly the raw material I then hand to the model as compact context (the principle from the context-engineering post):

// Build a compact, budgeted context from the hits
const context = rows.map((r) => `## ${r.title}  [${r.tags.join(', ')}] (${r.updated_at})\n${r.snippet}`).join('\n\n');

const prompt = `<instructions>
Answer only from <context>. Cite the entry title as the source.
If the info is missing, say so.
</instructions>

<context>
${context}
</context>

<question>${userQuery}</question>`;

When vector search still wins

This is not “embeddings are bad”. They are the right choice when:

• the corpus is large and unstructured (thousands of PDFs without metadata),
• queries are purely semantic (“find everything thematically similar”),
• there is no reliable metadata to filter on.

For a curated wiki, none of that holds. The pragmatic middle is often hybrid: pre-filter structurally, then rank semantically within the hits. But the expensive vector layer as the first step was simply the wrong order for me.

Conclusion

Vector search is a default, not a law of nature. Before you set up embeddings, a vector DB and chunking pipelines, it’s worth asking: is my knowledge already structured? If so, the boring answer – WHERE, ts_rank, metadata – is often more precise, cheaper and easier to debug than any embedding stack.

[Top]