How the "insufficient information" decision is made

Writing a fluent answer to a question the documents do not cover is what a language model does best. In Brainer, three gates stop that behavior.

The first gate is retrieval confidence. If the best result's score is below the threshold, generation does not start at all. There are two scales: 0.20 when the reranker ran, 0.015 for the fused ranking score when it did not. When we tried to manage both with a single threshold, the gate never closed while the reranker was on; the measurement caught that.

The second gate is sentence-level verification. The generated answer is split into sentences; each sentence is given to a multilingual NLI model together with the sentences of the cited passage that carry the claim. Every number in the sentence is looked up verbatim in the source; a missing number fails the sentence. If the mean is below 0.20 or the weakest sentence below 0.30, the answer is rejected.

The thresholds were not taken from English scales. We saw that the model scores correct sentences low on Turkish input too; the 0.70 threshold recommended for English rejected almost every answer in Turkish. When we moved the weakest-sentence threshold to 0.30 on 516 recorded answers, 477 were accepted, and the share of answers carrying one unrelated sentence that still passed fell from 44% to 15%.

The third gate is the citation check. Every citation must be in the list of passages retrieval brought back. A citation not in the list is an invented source; the answer is rejected. A rejected answer is tried once more with a stricter prompt; if it fails again, the user sees "insufficient information" and the reason is kept in a counter.

The cost: the model sometimes stops on a question that has an answer. In the June measurement this happened on 1 of 30 questions. We count it as an error in the safe direction and keep it in the test set. The target in the pilot is an abstain rate between 5% and 15%; above that, the thresholds are tuned together for the tenant.