Somewhere along the way, vector search optimization turned into one line of advice: dump everything into a vector database and let HNSW sort it out. I have watched production bills explode behind that advice, because teams budget vector RAM as if it were free. Ingest millions of documents with no plan for how they are stored and the feature ships with a cost problem attached.
I recently audited a system where the vector infrastructure cost more than the LLM API calls did. Nothing was wrong with the logic. The problem was the raw memory footprint of 1024-dimensional float32 vectors, which is where contextual retrieval performance usually dies. Pairing Matryoshka Representation Learning (MRL) with sensible quantization cut those costs by up to 80%.
The precision trap: why your index is expensive
Standard embedding models emit 32-bit floats, so a 1024-dimensional vector is 4KB. Replicate three times for high availability and 100 million vectors need over 1.2TB of RAM, around $6,000 a month at cloud pricing. That is storage alone, before you count the graph connections that a FAISS or HNSW index adds on top.
Scalar quantization: start here
Scalar quantization to int8 is the vector search optimization I reach for first in production. Precision drops from 4 bytes per value to 1, so storage falls 4x, and retrieval quality holds at roughly 98%. Recall@10 barely moves. Binary quantization goes further and often falls off a performance cliff instead.
// Example: Conceptual HNSW configuration for Scalar Quantization
{
"index_type": "HNSW",
"quantization": "int8",
"dimensions": 384,
"m": 16,
"ef_construction": 200
}Matryoshka embeddings: the nesting doll trick
MRL attacks the other axis. Instead of cutting precision, it cuts dimensions. Models trained this way, such as mixedbread-ai’s v1, pack the strongest semantic signal into the leading dimensions, the way a nesting doll holds the smaller dolls inside. That lets you truncate a 1024-dimensional vector to 256 or even 128 and lose very little accuracy. The MRL research paper works through the details.
Stack the two and the numbers get interesting. Truncating to 128 dimensions and quantizing to int8 gives a 77.9% cut in storage. If you are running a cost-sensitive search feature or a performance-critical WordPress backend, that is what makes the thing affordable at scale.
Binary quantization and the performance cliff
Binary quantization is the far end of the scale: one bit per float, a 32x reduction, and a trap more often than a win. In my experience retrieval quality collapses as the dimension count comes down, and by the time you are near 64 dimensions binary recall can sit under 10%. Only run it if a solid cross-encoder re-ranker sits in the second stage of your pipeline to clean up the results.
If this vector search work is eating up your dev hours, I can take it on. I have been wrestling with WordPress and backend infrastructure since the 4.x days.
What I would ship
For a production RAG app today: drop float32, use MRL to find the dimensionality that still holds up for your data, usually around 256, and quantize to int8. That combination buys the most infrastructure savings per unit of accuracy you give up, and the search still returns something better than a legacy SQL LIKE query.
