Verify LLM Answers Against Their Source: Jev API Grounding Check (Three Nouls, One Request)
A retrieval pipeline has one failure mode that no amount of prompt engineering removes: the model writes something the retrieved paragraphs do not support, and the output looks exactly like an answer that is fully grounded. There is a second one right beside it — the model says the paragraphs do not answer the question when they plainly do, and again nothing in the text marks the difference.
This page is a build guide for a check that catches both, using the Jev API. It is a companion to the official verify_llm_outputs cookbook: every number below is copied from that cookbook’s published run, made with jev-1.14.0 on 2026-11-07 against answers that gpt-6-luna wrote the same day. Nothing here is an estimate of what the check “should” do.
The two mistakes that look like answers
The setup is the ordinary one. You have questions, you retrieve paragraphs for them, you ask an LLM to answer from those paragraphs only.
The first mistake fills a gap. The chain of facts needed to reach the answer is missing one step, and the model supplies that step from memory instead of from the text. The sentence that comes back reads like every other answer in the batch.
The second mistake refuses too early. The paragraphs do contain the answer, but the model reports that they do not — often because the fact is two hops away and stated in a paragraph that looks like background.
Both are invisible at the output layer, and both are cheap to make at scale. A check therefore cannot be a second opinion on the same prompt; it has to ask a different kind of question, about the paragraphs rather than about the answer.
One request holds the whole check
The check’s state is built once per answer, with four named fields:
request— the original questionresponse_requirement— the instruction the answering LLM was given, so the checker knows what it was asked to doparagraphs— the retrieved text, each paragraph under its own titlecandidate_response— the answer under suspicion
All three questions go into that single request. This matters for cost and latency rather than elegance: Jev ingests the state once and evaluates every question against it in parallel, so adding questions to a request does not multiply the state you pay to send. The official models page puts the two context limits at 64k tokens per request and 32k per question, which is the constraint to respect when a retrieval window is generous — the state plus all questions combined has to fit the first limit, the state plus any one question the second.
The answerer’s instruction matters enough to be worth reproducing verbatim in the check, because the published run’s numbers depend on it:
Answer the question using only the paragraphs. Return only the shortest answer that fully answers the question. Do not explain your reasoning. If the paragraphs do not answer the question, say so.
That last sentence is what produces the refusals the second question then examines. Swap the instruction for one that forbids refusals and you have removed half of what the check verifies — you will still catch unsupported answers, and you will never see an unnecessary refusal.
Three Noul questions, written out
A Noul is the primitive for a yes/no question whose answer comes back as a probability. Each one takes an instruction and two criteria — what a true verdict means, what a false verdict means — and writing those criteria carefully is most of the work of getting a check that behaves.
answerable — do the paragraphs state every fact needed to answer the question? True means each step from the question to its answer is stated in the paragraphs; false means at least one step is not.
declines — does the candidate response decline to answer, or say the information is missing? True means it says it cannot answer or that the paragraphs do not say; false means it gives an answer.
supported — following the paragraphs step by step from the question, do you arrive at the candidate’s answer? True means the paragraphs connect the question to the answer through stated facts; false means they do not connect the two.
Read the three together and you can see why they are separate rather than one compound judgement. answerable describes the source, declines describes the answerer’s behaviour, supported describes the relation between the two. A single question like “is this answer correct?” would ask one model to solve all three at once, and would leave you unable to say whether a low score came from a bad paragraph set or from a model that refused good text.
From three probabilities to one risk number
The rule is short:
- if
declinesis 0.5 or above, the risk isanswerable— a refusal is wrong to the exact extent the paragraphs did answer - otherwise the risk is
1 - supported— an answer is wrong to the extent the paragraphs do not lead to it
Then a threshold, kept in the code as REVIEW_AT = 0.2: an answer with a risk at or above it goes to review, everything below it passes.
Two properties of this design are worth noticing, because they decide what the number means operationally.
First, the risk is not a probability of correctness. It is a review priority. A risk of 0.02 on an answer that happens to be wrong is not a bug; it is the cost of a check that reads text instead of knowing facts.
Second, the 0.5 gate on declines is a switch, not a score. An answer somewhere around 0.5 on that question is being classified by a coin flip, and it is the one case where you may prefer to compute both branches and take the higher. The published run does not do that; it follows the switch.
Four answers, scored
The cookbook prints four answers in order — a right answer, a right refusal, an answer the paragraphs do not support, and a refusal of a question the paragraphs do answer. These are the published scores, unchanged:
| Question | gpt-6-luna answer | answerable | declines | supported | risk | status |
|---|---|---|---|---|---|---|
| Where did the painter of The Mud Bath study? | The Slade School of Art. | 0.97 | 0.02 | 0.98 | 0.02 | PASS |
| Where is Assomada located? | The paragraphs do not say where Assomada is located. | 0.07 | 0.99 | 0.97 | 0.07 | PASS |
| What was the population density of Tammy Leitner’s city of birth in 2009? | 182.5/km² | 0.12 | 0.04 | 0.08 | 0.92 | REVIEW |
| When did the father of Dahyabhai Patel die? | The paragraphs do not say when Dahyabhai Patel’s father died. | 0.86 | 0.81 | 0.09 | 0.86 | REVIEW |
The first two pass for opposite reasons, and that is the useful part of the demo. The Mud Bath answer earns supported at 0.98, so the paragraphs demonstrably lead to it. The Assomada answer declines, and answerable is 0.07, so declining was the right call. Both are correct; one is correct by answering, the other by not.
The Tammy Leitner answer is the interesting failure. The city of birth is San Diego, and no paragraph gives San Diego’s population density in 2009, so supported falls to 0.08 and the risk lands at 0.92. The number 182.5/km² is not nonsense; it is a plausible number from outside the source. Only a check that walks the paragraphs can tell the difference between plausible and derived.
The Patel case is the mirror image. One paragraph names Sardar Vallabhbhai Patel as Dahyabhai Patel’s father, another gives the date he died, so the question is answerable — and the answerer still refused. answerable at 0.86 sends a refusal to review, which is exactly the class of mistake that a conventional “was the answer correct” check typically misses, because it never looks at refusals separately.
How the right/wrong labels were made
You cannot measure a check without knowing which answers were actually right. The cookbook does this in two stages, and the honesty of the second stage is what makes the numbers worth quoting.
Stage one: MuSiQue ships a reference answer for every question. One Jev request with a matches_reference Noul compares the answer to that reference without showing the paragraphs, accepting different wording for the same entity, value or date. A match at 0.5 or above is labelled right, and the run reports 739 answers matched.
Stage two: the remaining 261 go to gpt-6.1-sol with reasoning_effort set high and a strict JSON schema whose label is one of right, wrong, unclear, plus a rationale. That model sees the paragraphs and the reference, and is told that the human-written reference may itself be wrong about them. The split: 146 right, 115 wrong, 0 unclear.
739 plus 146 is 885, and 885 of 1,000 is the 88.5% base rate quoted everywhere on this page. The re-check step exists because the reference is sometimes wrong in one of two specific ways: MuSiQue counts a question as answerable while the paragraphs skip a step, or a question was meant to be unanswerable while another paragraph still carries its answer.
The dataset itself is worth knowing, since the difficulty of the check is set by it. The questions come from MuSiQue (Trivedi et al., CC BY 4.0), where each question joins two to four simple questions about Wikipedia into one, so earlier answers are needed for later steps. Each arrives with 20 paragraphs, most of them on nearby topics and useless for answering. The file used here is a fixed random sample of 1,000 questions from MuSiQue’s development set, one version of each question, and — this is the part that makes the measurement meaningful — about half are the version where a needed paragraph has been removed and the question cannot be answered from what remains. The run prints it directly: 1,000 questions, 473 answerable from their paragraphs.
Without that split, a check could look strong just by being pessimistic, so track exactly this: on an unanswerable question, a good answer must be a refusal, and the check has to agree with a refusal for the right reason.
Measured on 1,000 answers
Here is the published table, thresholds and all. “How many” is the number of answers used, “right” is the share of those that are correct, and “wrong held back” counts the LLM’s wrong answers diverted to review.
| answers used | how many | right | wrong held back |
|---|---|---|---|
| every answer | 1,000 (100%) | 88.5% | 0 of 115 |
| risk below 0.2 | 727 (73%) | 95.7% | 84 of 115 |
| risk below 0.3 | 825 (82%) | 94.2% | 67 of 115 |
| risk below 0.5 | 917 (92%) | 91.9% | 41 of 115 |
| risk below 0.7 | 965 (96%) | 90.3% | 21 of 115 |
Read the first pair of rows as the core result. Keeping every answer gets you 1,000 answers at 88.5%. Keeping only what the check passes at 0.2 gets you 727 answers at 95.7%, which means the check buys a little over seven points of precision for a quarter of the volume, and it moves 84 of the 115 wrong answers out of the answered stream.
The next three rows are the same trade at looser settings, and they are the reason the threshold should not be copied blindly. At 0.7 you keep 965 answers at 90.3%, catching only 21 wrong ones — that is the setting to choose when review is expensive and a wrong answer to a user is cheap. At 0.3 you keep 825 at 94.2% and catch 67. Where you sit on that curve is an operations decision, not a modelling one.
The distribution explains why the curve bends the way it does: right answers cluster near zero risk, wrong answers spread across the whole range. A threshold low enough to hold back most wrong answers necessarily sends some right ones to review too.
One more number worth stating plainly, because it is what reviewers feel on the second week: at 0.2 the check holds back 273 answers, and 84 of them are wrong. The other 189 are fine. Most of the review queue is confirmation work, and any plan for this check has to include a queue with someone in it — or a second, cheaper action for the passed-but-doubted cases.
What the check misses
The published caveat section is unusually specific, and it is the best guide to what this pattern can and cannot do.
31 wrong answers passed at 0.2. Eighteen of them answer a related question instead of the one asked — the band whose members were asked for comes back as “Leftover Salmon”, the thing the NFL is comes back as “the NFL”. The paragraphs do lead to those answers, so supported is high, and the check is behaving consistently: it measures support, not question interpretation. Catching this class needs a different question, not a lower threshold.
The other 13 are refusals of questions the paragraphs do answer, where answerable also came back low, between 0.10 and 0.19. This is the more sobering group: Jev missed the same connection between paragraphs that the LLM missed. If the source text makes a step genuinely hard to see, the check inherits that difficulty, and no threshold fixes it.
Two structural caveats from the cookbook also belong in any write-up you make internally. The right/wrong labels come from models, so a stricter or looser re-check would change the 88.5% base rate and therefore every precision number derived from it. And the three check questions were chosen over two other candidate sets by trying them on the first 100 questions of the sample — the results on those 100 are not a fair test of the choice.
Putting it in a pipeline
The shape that works in production is the shape of check_answer(): build the state, ask the three questions in one call, compute the risk, return the probabilities alongside the verdict so downstream systems can log all four numbers rather than a boolean.
Four practical details from the published run are worth carrying over:
- Concurrency, per side. The cookbook uses 8 workers on the answering side — about 1,250 OpenAI calls for the sample — and 4 on the Jev side for 2,000 calls, with an explicit note that more workers hit the public rate limit. Rate limits belong per provider, not per pipeline.
- Cache the model calls. The notebook wraps both callers in a JSON cache keyed on the prompt or state, and the client falls back to the placeholder key
cache-onlywhenTYPESAFE_API_KEYis missing, so a keyless kernel replays saved responses. A downloaded cache file is never overwritten; turning it live means deleting it, not editing it. - Keep the instruction in the state.
response_requirementis what lets you audit a historical score later. If you change the answering prompt, old risks stop being comparable, and without the field stored you will not be able to prove why. - Log the three probabilities, not only the risk.
answerableandsupporteddrift apart for different reasons, and the pair is the cheapest diagnostic you will ever have: a population of lowanswerablewith low risk usually means your retrieval is weak, not that your answers are strong.
Setting the threshold on your own data
The cookbook’s instruction here is direct: swap in your questions, your paragraphs and your reference answers, run the labelling step, and read the threshold off the table. The reason is that the whole curve above is a property of MuSiQue’s mix of answerable and unanswerable questions. Change the mix and the numbers move.
If you have no reference answers, there is a cheaper version that still produces a usable setting: human-label 200 answers at random, compute their risks, and plot the two histograms. You will not get the published precision figures, but you will learn where your own right answers sit, which is what the threshold decision actually needs. Then keep the labelled set — it is the only way to notice later that a model upgrade moved the distribution.
Cost, volume and what you send
The official models page prices Jev at $42 per billion input tokens, or $0.042 per million, with output tokens free. The billing unit is the state, which is why putting three questions in one request is the economical choice: you pay for the question, instruction, paragraphs and answer once, not three times.
Two context limits apply to every request: 64k per request and 32k per question, both raised to 262k by sending the TypeSafe-Danger-Zone-Long-Context: true header on requests that need it — available on 1.14 and 1.13 alike. Long context costs the same per token and is documented as slower and less accurate, so the sane default is to keep retrieval windows inside the normal limits and reserve the header for the rare long document.
The measurement run reports 2,000 Jev calls for 1,000 answers. That is two per answer: one for the check and one for the reference match used to label. In production you only need the first, so plan one request per answer — and remember that model versions move, so re-read the models page rather than trusting a price you quoted a quarter ago.
Where this sits next to the rest of the site
The check is built from primitives, so the earlier guides are the prerequisites rather than background reading. Three primitives explains why a Noul returns a probability instead of a boolean, and confidence and fallback covers what to do with the probabilities once you have them — the risk formula here is a worked instance of that pattern. Guardrails for LLM apps is the sibling check: it screens messages before and after the model, while this one asks whether the answer follows from the document. Request patterns covers batching several questions into one request, which is the same move this check makes.
On the retrieval side, re-ranking fixes which paragraphs arrive, and this check is what you run after that decision to see whether the answering model actually used them. The measurement harness itself — reference labels, a second model settling disagreements, thresholds chosen from a table — is the same machinery as the RAG chunk and rerank batch case; the difference is that this page measures groundedness rather than retrieval order.
If review capacity is the constraint rather than model spend, support routing describes where the review queue goes, and ticket triage shows the volumes involved when every inbound message is classified rather than answered.
FAQ
Is this the same thing as an LLM judge? No, and the difference is the shape of the evidence. A judge reads the answer and decides whether it looks good; this reads the paragraphs and decides whether they lead to the answer. The published result reflects that: the check’s scores are low for right answers and spread out for wrong ones, which is what you want from a measurement rather than a preference.
Can I skip the labelling step? For a live pipeline, yes — you need the check, not the measurement. You lose the ability to claim a precision figure, and you lose the ability to justify the threshold you picked.
What happens at 0.2 exactly? The answer goes to review: the cookbook’s own wording is retry, a stronger model, or a person. The check does not decide what review means.
Does the risk formula need tuning per language? Nothing published says so, and this page will not guess. The multilingual capability of 1.14 is documented as improved, but a per-language threshold is a claim you would have to measure on your own labelled set before relying on it.
What if the paragraphs are wrong but consistent? Then supported will be high and the check will pass the answer. This is a grounding check, not a fact check: it verifies derivation from what you retrieved, and it inherits the quality of that retrieval.