Smaller models.
More possibilities.
thinletter is a Czech AI startup. We compress language models and represent them efficiently, and we tune them to a specific language or domain — measured against the unchanged system they have to fit into.
What we do
Three things, kept deliberately narrow.
Compression
Language models at 2–5 bits per weight that keep the quality of the original where it matters: in the retrieval results, not in a benchmark of the file.
How
We quantise the query encoder of an embedding model and leave the document index untouched: 235–385 MiB files instead of 1.1 GiB, running in llama.cpp and in the browser (WebAssembly or WebGPU). Below the smallest llama.cpp format we run a vector-quantised container at 1.8–2.1 bits per weight with our own WebGPU runtime.
Every file is verified on the index it has to serve: nDCG@10, cosine to the full-precision query vector, top-10 overlap, paired bootstrap over queries.
Language and domain
Quantisation calibrated on the language of your corpus — Czech first — and, where it pays, on your domain.
What we found
At low bit widths the language of the calibration text matters more than its domain: on a Czech legal index, calibrating on Czech instead of English text was worth +0.09 nDCG@10 at 2.6 bits, domain another +0.01–0.02. We release clients calibrated on Czech text for three model families; a Czech calibration does not cost English quality.
Tolerance to compression is a property of the checkpoint, not of the recipe: one 0.6B model holds 99 % at 3.4 bits, another of the same architecture fails below 4.5.
Evidence
Pre-registered predictions, paired comparisons, published negative results. Numbers you can reproduce on your own index.
How we measure
Each run has a prediction and a kill rule written down before it starts. Comparisons change one thing at a time at the same calibration budget. Differences are paired bootstrap intervals over queries (10 000 draws), read against calibration-draw variance (~0.01) and between-machine variation (±0.006): under 0.01 nDCG@10 is a tie.
Negative results stay in the report: a 2.6-bit headline number that did not generalise beyond one corpus, a rotation with no stable effect, a claim that held for one model and not for the recipe.
Results
Measured on the model's own full-precision index, test split. Click a number for the context.
Where the smallest files stand
| client (SciFact, harrier-0.6b) | MiB | % of fp32 nDCG@10 | what it is |
|---|---|---|---|
| harrier-0.6b fp16, the server model | 1 143 | 100 % | reference |
| llama.cpp Q3_K with imatrix | 235 | 98.9 % | the free baseline; our GPTQ export ties it, we say so |
| llama.cpp IQ2_XS, its smallest format | 178 | 93.9 % | below this size llama.cpp has nothing |
| BitNet-270m, ternary, trained (Microsoft) | 140 | 97.0 % | the strongest small competitor; needs training |
| thinletter VQ 2.1 bits per weight | 120 | 98.2 % | ours: post-training, no retraining, WebGPU runtime |
| thinletter VQ 1.8 bits per weight | 105 | 96.5 % | ours |
| thinletter VQ 1.6 bits per weight | 91 | 93.6 % | ours, the smallest working client |
nDCG@10 against the model's own fp32 index, SciFact test split (300 queries, fp32 0.7559). Our rows are torch simulations of the container with synthetic-query calibration; the released SciDocs files of the same recipe (127 / 113 MiB) hold 94.7 / 91.3 % and were verified in the browser to ±0.001. BitNet-270m and the llama.cpp files are measured as files.
98.2 %
SciFact test split, nDCG@10 against the model's own fp32 index (0.7559): 120 MiB at 2.1 bits per weight 98.2 %, 105 MiB at 1.8 bits 96.5 %, 91 MiB at 1.6 bits 93.6 % (torch simulation of the container, synthetic-query calibration, 4-d codebooks per 256-column block, input-side structured rotation). The released SciDocs containers of the same recipe (127 / 113 MiB) hold 94.7 / 91.3 % and were verified in the browser to ±0.001. Below ~180 MiB llama.cpp has no format at all; BitNet-270m gets to 140 MiB only by training the model ternary.
99.2 %
bge-m3 Q4_K_M with a 4-bit token table: cosine to the fp32 query vector 0.990, top-10 overlap 0.864; the 321 MiB Q3_K file keeps 98.3 %. Qwen3-Embedding-0.6B: 98.7 % at 340 MiB, 99.5 % at 385 MiB. The same Czech-calibrated files keep 99.6–100 % on SciFact. The Czech evaluation uses synthetic queries; its full-precision level is the same for every base model we tried, so it measures the compression, not the base.
103 ms
harrier-0.6b at 2.10 bits per weight in our WebGPU runtime: p50 103 ms, p95 113 ms, load 2.4 s, 77 % of the peak process memory of the llama.cpp path, at −0.011 nDCG@10 against the 3.4-bit file (97.6 % of fp32 on SciFact). One laptop, one integrated GPU, 300 queries; no discrete-GPU or mobile numbers yet. The runtime is not public; the containers are.
What we learned along the way
Measured the same way, kept on the page because the next reader will ask: ties, negative results and the baseline everyone gets for free.
96–101 %
SciFact 98.9 %, NFCorpus 99.3 %, ArguAna 101 %, SciDocs 99.2 % (microsoft/harrier-oss-v1-0.6b, GPTQ on the llama.cpp Q3_K grid, generic English calibration). At this bit width llama.cpp's own quantiser reaches the same quality with the same text; the value is in the verification, and we say so. The file stays in the table below because a verified client is useful even when it is not novel.
−0.008
The idea: give the neurons the corpus uses most a larger codebook and the rest a smaller one at the same average rate. Three rotation seeds, matched random control: importance minus random −0.0085 cosine to fp32, nDCG@10 within noise; using the importance only to weight the codebook fit: −0.008. Side finding: two quantisations that differ only in which rows share a codebook differ by up to 0.03 cosine, so on this grid an effect under 0.01 needs three seeds or a paired construction. Pre-registered and closed 2026-09-10.
+0.010
The prompt in front of every query is the same 19 tokens, so the full-precision model computes its keys and values once (2 MB) and the compressed model only reads the user's words. The naive version, swapping the exact prompt into an existing client, looked huge (+0.037 at 1.6 bits) but only repaired files calibrated on documents; a client calibrated on synthetic queries reproduces the prompt itself and gains nothing (−0.003 / +0.000 on the released files). Calibrating every block with the exact prompt in place and shipping it is the real thing: +0.013 / +0.009 / +0.008 nDCG@10 over the best query-calibrated client of the same seed at 1.8 bits on SciFact, every paired interval above zero, cosine to fp32 0.90 → 0.92. On SciDocs (short title-like queries) +0.002 / +0.004 / +0.000, at 2.1 bits +0.002, at 1.6 bits +0.001: nothing. Such a client needs the prompt at inference (75–84 % without it). Pre-registered, 2026-09-10/11; the recipe is in the runtime, the published containers are unchanged.
Released query clients
| client | base model · licence | MiB | quality vs the model's own fp32 index |
|---|---|---|---|
| harrier-0.6b Q3_K, generic English calibration | microsoft/harrier-oss-v1-0.6b · MIT | 235 | SciFact 98.9 % · NFCorpus 99.3 % · ArguAna 101 % · SciDocs 99.2 % |
| harrier-0.6b Q2_K, SciDocs synthetic queries | MIT | 192 | SciDocs 94.5 % |
| harrier-0.6b vector-quantised 2.10 / 1.83 bpw | MIT | 127 / 113 | SciDocs 94.7 % / 91.3 % (needs the thinletter runtime) |
| Qwen3-Embedding-0.6B Q4_K_M, English calibration | Qwen/Qwen3-Embedding-0.6B · Apache-2.0 | 340 | SciFact 100.0 % · NFCorpus 99.4 % · ArguAna 100.1 % · SciDocs 99.3 % |
| Qwen3-Embedding-0.6B Q4_K_M / Q5_K_M, Czech calibration | Apache-2.0 | 340 / 385 | Czech legal index 98.7 % / 99.5 % |
| bge-m3 Q4_K_M / Q3_K, Czech calibration | BAAI/bge-m3 · MIT | 355 / 321 | Czech legal index 99.2 % / 98.3 % · SciFact 99.6 % / 100.0 % |
Every client is a query encoder for exactly one document encoder and its settings. Each row links to its Hugging Face repository; the model card carries compatibility, recipe and sha256.
Open releases
Report, verification recipe, checkpoints and a browser demo. Run the recipe on your own index; the numbers are yours either way.
Work with us
Pilots start with a measurement: we take your embedding index, quantise the query side, and hand you the paired numbers before anything is decided. The vector-quantised runtime is available to pilot partners by invitation.
Jan Rosecký, founder · a contact address is being set up