Vector Search: What It Is and What It Costs to Run
Vector search finds things by meaning rather than by keyword. The infrastructure question is where the index lives and what it costs.
A vector search system turns text or images into numeric vectors, stores them, and finds the nearest ones to a query vector. It is what makes semantic search and retrieval-augmented generation work.
The interesting infrastructure question is not the algorithm. It is memory.
The pipeline
- Chunk the source documents into passages of a few hundred tokens.
- Embed each chunk into a vector using an embedding model.
- Store the vectors with metadata and a reference to the source.
- Index them so nearest-neighbour search is fast.
- Query by embedding the question and retrieving the nearest chunks.
- Use the retrieved text — displayed as results, or passed to a language model as context.
Chunking strategy affects quality more than most people expect, and it is far cheaper to iterate on than anything else in the pipeline.
Where to put the index
| Option | Suits | Consideration |
|---|---|---|
| PostgreSQL with pgvector | Existing Postgres users, moderate scale | One database to operate; excellent default |
| Managed vector service | Teams avoiding operations | Priced per vector or per hour |
| Self-hosted vector database | Large scale, specific requirements | Another system to run |
| Search service with vector support | Existing search infrastructure | Hybrid keyword and vector in one place |
| In-memory library | Small, static corpora | Simple; rebuild on change |
For most applications, pgvector in a managed PostgreSQL instance is the right answer. It handles millions of vectors, needs no new infrastructure, and keeps metadata filtering in SQL where it is easy.
Memory is the cost driver
Vector indexes are typically held in memory for fast search. Estimate roughly: number of vectors multiplied by dimensions multiplied by four bytes, plus index overhead that can be substantial.
A million vectors at 1,536 dimensions is around 6 GB of raw data before indexing. That figure — not query volume — is what determines the instance you need.
- Use smaller embedding dimensions where quality permits; several models offer configurable dimensions.
- Quantise vectors, which reduces memory substantially with modest recall loss.
- Filter by metadata before vector search where possible, reducing the search space.
- Archive or delete stale vectors; indexes grow indefinitely otherwise.
Hybrid search is usually better
Pure vector search is weak at exact matches — product codes, names, specific identifiers. Pure keyword search is weak at paraphrase. Combining them, and re-ranking the merged results, outperforms either alone in most real applications.
Do not treat vector search as a replacement for keyword search. Treat it as an addition.
Operational details that matter
- Re-embedding is expensive. Changing embedding model means regenerating every vector, so choose deliberately and record which model produced each index.
- Keep the source text. You will need it to display results and to re-embed later.
- Handle updates. Decide whether changed documents are re-embedded immediately or in batches.
- Version the index, so a bad re-index can be rolled back.
- Measure recall, not just latency. A fast index returning poor results is a worse outcome than a slow accurate one.
- Back up the source data, not only the index — the index can always be rebuilt from source.
Common questions
Do I need a dedicated vector database?
Below a few million vectors, usually not. PostgreSQL with pgvector is simpler to operate and performs well. Dedicated systems earn their place at larger scale or with demanding latency requirements.
How large should chunks be?
A few hundred tokens is a common starting point, with overlap between chunks so context is not cut mid-thought. It is worth testing on your own content; the effect on quality is large.
Can I run this without a GPU?
Yes. Embedding via a hosted API needs no local GPU, and search itself is CPU work. A GPU only becomes relevant if you self-host the embedding model at volume.
How much does it cost to run?
Dominated by memory for the index and by embedding generation. A moderate corpus in pgvector on an existing managed database often adds very little.
Where to go next
See AI services versus self-hosted models and cloud storage types explained.
Read next
Training vs Inference: Two Very Different Cost Problems
Training is a large, bounded cost you can plan. Inference is a small, unbounded cost that grows with every user.
Cloud AI Services or Your Own Models
Every hyperscaler sells ready-made AI services. Whether to use them or run your own comes down to four questions.
Infrastructure Checklist for an AI Workload
Machine learning projects fail on infrastructure more often than on modelling. Most of it is foreseeable.