Thinletter

Keep the index, shrink the query encoder.

Your document index stays exactly as it is. Only the query side of the embedding model is replaced by a 235–340 MiB file that runs in llama.cpp or in a browser tab and holds 96–101 % of the full model's retrieval quality, measured against your own index, with confidence intervals.

Try it in your browser Read the report Checkpoints on Hugging Face Code & recipe

The demo downloads a model once (235–340 MiB, cached in your browser) and then searches 25 656 scientific abstracts locally; no query leaves your machine.

What holds

96–101 %of fp32 nDCG@10 kept by the 3.4-bit harrier-0.6b client on four English corpora and by the jina client on a Czech legal index of 55 071 segments, index unchanged.
+0.09nDCG@10 from calibrating on Czech instead of English text at 2.6 bits. The language of the calibration text matters more than its domain (+0.01–0.02).
103 msper query for the 119 MiB vector-quantised client on an integrated GPU, against 444 ms for llama.cpp's WebGPU path with a 235 MiB file. Runtime not public yet; numbers are.
= llama-quantizeAt ~200 MiB our GPTQ export is not a better file than llama.cpp's own quantizer with the same text. We say so, and the pipeline uses it as the first arm.

Released query clients

filebase model / licenceMiBquality vs the model's own fp32 index
harrier-0.6b Q3_K, generic English textmicrosoft/harrier-oss-v1-0.6b · MIT235SciFact 98.9 % · NFCorpus 99.3 % · ArguAna 101 % · SciDocs 99.2 %
harrier-0.6b Q2_K, SciDocs synthetic queriesMIT192SciDocs 94.5 % (2.6-bit files do not generalise: 98 % on SciFact, 93 % on SciDocs)
Qwen3-Embedding-0.6B Q4_K_M + q4_0 table, English imatrixQwen/Qwen3-Embedding-0.6B · Apache-2.0340SciFact 100.0 % · NFCorpus 99.4 % · ArguAna 100.1 % · SciDocs 99.3 %
Qwen3-Embedding-0.6B Q4_K_M + q4_0 table, Czech imatrixApache-2.0340Czech legal index 98.7 % (every ≤ 3-bit file of this model fails: 80–93 %)

Every client is a query encoder for exactly one document encoder and its settings; same dimension does not mean same space. Model cards list the compatibility, the sha256 and the recipe. harrier · Qwen3-Embedding

Verify it on your own index

  1. Import your corpus and your existing document vectors; nothing is re-encoded.
  2. Generate synthetic queries for held-out documents (no real queries needed).
  3. Quantize with llama-quantize --imatrix using text in your corpus language, or with our GPTQ exporter.
  4. Verify: nDCG@10, cosine to the fp32 query vector, top-10 overlap, paired bootstrap; keep the smallest file that passes.

Five commands, in the README. If you run it, send the table; that is the most useful thing you can send.

Vector-quantised runtime preview

Below llama.cpp's smallest format we run a 1.8–2.1-bit container with its own WebGPU runtime: 119.5 MiB at 97.6 % of fp32 on SciFact, 103 ms p50 / 113 ms p95 per query on an integrated GPU, 77 % of the process memory of the llama.cpp path. Format and runtime carry over to jina and Qwen3-Embedding; quality below 2.2 bits on Czech does not yet (83 % / 64 %).

Numbers, caveats and the paired comparison with BitNet-270m: vq_results.md. A live demo with the VQ clients runs at lab.thinletter.io by invitation. Pilots: we measure your index first; the numbers are yours either way.

How it is measured

Every claim has a pre-registered prediction and a kill rule, every comparison changes one thing at a time at the same calibration budget, and differences are paired bootstrap intervals over queries (10 000 draws). Negative results are kept: the 2.6-bit reference number did not generalise beyond SciFact, two-sided rotation has no stable effect on the scalar grid, and "generic text is enough at 3.4 bits" turned out to hold for one model, not the recipe. A paired interval does not contain the variance of the calibration draw (~0.01 nDCG) nor between-machine variation (±0.006); read differences under 0.01 as ties.