Measurement log
The source of every number on this site
This page does two things: it gives the product measurement with its method, and it says where every number on the site comes from. The date changes when the measurement is renewed.
Method
Answer quality measured: 15 June 2026 · permission fuzz test: 12 October 2026
- 01
Test set: 39 Turkish questions; 30 are answered by the given documents, 9 are not. Three domains in equal parts: law, finance, insurance.
- 02
Each question comes with its own evidence passages; this measures answer generation and verification, not retrieval. Retrieval is tracked by a separate measurement.
- 03
The answering model and the judge model are different; temperature 0. The judge evaluates each answer 3 times independently and the majority counts.
- 04
Three separate runs were taken; the table is the last run, the differences between runs are in the findings.
Result
| Metric | Measured | Target | Meaning |
|---|---|---|---|
| Citation precision | 1.000 | ≥ 0.90 | Every citation in the answers is in the evidence list; no ghost citations |
| Citation coverage | 1.000 | — | All 29 answers given are cited |
| Abstain precision | 0.900 | ≥ 0.80 | 9 of 10 abstentions were warranted, 1 was needless |
| Abstain recall | 1.000 | — | Stopped on 9 of the 9 questions with no answer |
| Unsupported answer rate | 0.034 | ≤ 0.05 | 1 of 29 answers was flagged by the judge |
Findings, weak spots included
01A needless stop on one question
On a question about the no-claims discount in motor insurance the evidence was complete; the model abstained anyway. It did the same in all three runs. An error in the safe direction, but an error. We did not remove it from the test set.
02The judge is undecided on one borderline answer
For identical answers, the judge flagged 0 answers in one run and 1 in two runs. The quality gate was therefore set so that a single flag does not fail it (0.05); 2 flags do.
03A quoted abstention was misclassified
When the model returned the abstention sentence wrapped in quotes, the production code took it for an answer without citations. Fixed, with a test. It is here because the measurement found it.
Limits
- The set is small: 39 questions. One answer moves the judge rate by 0.034. The set is being grown.
- Our set, our documents. In the pilot the same measurement is repeated on your documents; the acceptance gate is that measurement, and it includes a manual review of 100 samples.
- This page is an automated measurement; it does not replace human review.
The numbers on this site and their sources
If a number's source is not here, the number is not on the site.
- 19% (share of time spent searching)
- McKinsey Global Institute, The social economy: Unlocking value and productivity through social technologies, July 2012, Exhibit 2: 19% of knowledge workers' week goes to searching for and gathering information. ↗
- 180,500 hours / year
- Arithmetic: 500 people × 1,900 hours × 0.19. Put in your own numbers for your organization.
- 29/29 · 9/9 · 1/29 · 1/30 · 1.000 · 0.900 · 0.034
- The measurement above, 15 June 2026.
- 0.20 · 0.30 · 0.20 (thresholds)
- Production defaults: mean verification 0.20, weakest sentence 0.30, reranked retrieval confidence 0.20 (0.015 without a reranker). A tenant setting can only tighten them.
- 516 answers, 477 accepted
- Moving the weakest-sentence threshold to 0.30: re-evaluation on 516 recorded answers, October 2026 (engineering notes).
- 2,010 queries, 0 leaks
- The last full run of the permission fuzz test, 12 October 2026. Repeated on every release.
- 13 threats
- The written threat model, T-1 to T-13; a living document.
- 5% to 15% (abstain rate)
- Target band for the pilot; tracked per tenant with a counter.
- 2 weeks
- Pilot plan: week 1 connecting and indexing, week 2 measuring. Hardware preparation excluded.
- 0.97 rank agreement · 30 → 8 · ~7 s
- The int8 build of the local reranker against the full build on a Turkish gold set; 30 passages on a laptop CPU (engineering notes).
Measure it on your own documents.
A two-week pilot: your sources are connected, the result is measured with your questions, and the report is yours.