Jev Confidence Scores and Fallback Strategies That Work

Jev’s response carries a confidence score on every answer — and that single field is what turns the model from a novelty into a production component. The score tells you where the model is sure and where it is guessing, which means you can automate the sure cases and route the rest. This guide covers how to set thresholds, design the three fallback paths, run a cold-start when you have no labeled data yet, handle the batch case, and monitor the whole loop so it does not silently rot.

TL;DR: Start with a 0.8 threshold: automate above it, route below it. Low-confidence items go one of three ways — retry with a sharper question, escalate to a human, or a general-LLM pass. Watch your confidence distribution over time: a drift toward lower confidence is your earliest warning that inputs or question wording have shifted.

What confidence means in the response

Every Jev answer — judgment, choice or scoring — ships with the same shape. Here is a scoring example — example fixture, confirm exact field names in the official documentation:

{
  "answer": 8,
  "scale": [1, 10],
  "confidence": 0.86,
  "rationale": "The answer addresses the core question but omits the setup step."
}

For a judgment (yes/no) question the shape is the same but answer is "yes", "no", or "unclear". For a choice question, answer is the label of the chosen option. In all cases confidence is a float from 0.0 to 1.0 and rationale is a plain-language explanation.

Read it as a routing signal, not gospel. The useful mental model: confidence ranks items by how safely you can automate them. The top of the distribution gets automation; the bottom gets scrutiny; the exact cut belongs to your cost of errors.

One thing confidence does not tell you: whether the answer is objectively correct. A high-confidence wrong answer is possible when the question is well-formed but the source document is misleading. Confidence measures the model’s internal certainty, which correlates with accuracy but is not the same thing. This is why you validate the threshold against labeled data before automating anything with real consequences.

Choosing a threshold: start at 0.8, tune with data

The numbers are starting points, not laws. The tuning loop that matters:

1. Log every answer with its confidence for a week, without acting on low confidence yet.
2. Label a sample of the low-confidence answers by hand.
3. Compute where errors actually start — often the safe cut is higher or lower than 0.8.
4. Set the threshold there and revisit quarterly.

Cold-start tuning before you have labels

Most teams want the threshold set before they have labeled data. A reliable cold-start approach:

1. Shadow mode for five to seven days. Make every call in production but ignore the fallback logic — just log the confidence score alongside the input and the answer.
2. Pull a stratified sample. Take ~50 items from each decile of confidence. You want representation across the full range, not just the tail.
3. Label by hand. For each sampled item, mark the answer as correct or incorrect by your own judgment.
4. Plot error rate by score bucket. Group items into 0.0–0.69, 0.70–0.79, 0.80–0.89, 0.90–1.0. The bucket where error rate crosses your acceptable level is your threshold.
5. Set conservatively on the first deploy. A threshold that sends 20% of items to review and turns out to be too cautious is much easier to correct than one that automates too much.

If shadow mode is not possible (the task does not run in production yet), use a synthetic holdout: run 200–500 representative items manually, score them by hand, and use those labels to set the threshold. It is less accurate than real traffic but far better than guessing.

The three fallback paths

When confidence is below threshold, exactly one of three things should happen:

import json, os, requests

def judge_with_fallback(prompt: str, threshold: float = 0.8) -> dict:
    # Confirm the exact model slug on the OpenRouter model page
    resp = requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={
            "model": "typesafe/jev-1.13",
            "messages": [{"role": "user", "content": prompt}],
        },
        timeout=30,
    )
    result = json.loads(resp.json()["choices"][0]["message"]["content"])

    if result["confidence"] >= threshold:
        return result  # automate
    if result["answer"] == "unclear" or result["confidence"] < threshold / 2:
        return {"route": "human"}  # deep uncertainty: never retry-loop
    return {"route": "llm_review", "jev": result}  # middle band: full LLM pass

The wrapper returns the fixture-shaped answer when confident, or a routing decision otherwise; the shapes shown are example fixtures and official field names should be confirmed in the official documentation. The three routes in order:

• Retry with a sharper question — for the middle band. Often the problem is the question, not the model: re-ask with the boundary made explicit, per the question-design guide. Retry once, never in a loop.
• Escalate to a human — for the deep-uncertainty band and for anything with financial or legal stakes. This is what unclear answers are for.
• General-LLM pass — for middle-band items that a human should not see but a single call can resolve. The chat model writes a full justification; Jev’s low-confidence answer still travels with it as prior context.

Implementing the retry path correctly

A retry is not just resending the same prompt. The point is to give the model more information so the question can be answered with more certainty. Common ways to sharpen a question:

• Add the explicit criterion it was probably missing. If you asked “Is this a senior engineer?” with no definition, add “A senior engineer has at least five years of professional software experience and has led at least one project end-to-end.”
• Narrow the output scale. If a 1–10 scale returns low confidence, ask for a yes/no on a specific threshold instead: “Based on this resume, does this candidate meet a score of 7 or higher on technical depth?”
• Include a concrete example of a correct answer. A brief worked example in the question drastically reduces ambiguity for borderline cases.

Always set a single retry limit in code. A retry loop that fires again on a still-low second answer will burn cost and delay the item. One retry, then route to human or LLM:

def judge_with_one_retry(original_prompt: str, sharpened_prompt: str, threshold: float = 0.8) -> dict:
    first = call_jev(original_prompt)
    if first["confidence"] >= threshold:
        return first
    second = call_jev(sharpened_prompt)
    if second["confidence"] >= threshold:
        return second
    # Both below threshold — route, do not retry again
    return {"route": "human", "attempts": [first, second]}

Passing Jev context to the LLM fallback

When you route to a general LLM, send Jev’s low-confidence answer as prior context rather than starting blind. This lets the LLM focus on resolving the uncertainty rather than re-doing the full evaluation:

def llm_review_with_jev_context(item: str, jev_result: dict) -> str:
    system = (
        "You are reviewing a judgment made by a specialized AI model. "
        "The model returned a low-confidence answer. Your task is to confirm, "
        "correct, or override it with a full rationale."
    )
    user = (
        f"Item: {item}\n\n"
        f"Model answer: {jev_result['answer']}\n"
        f"Model confidence: {jev_result['confidence']}\n"
        f"Model rationale: {jev_result.get('rationale', '')}\n\n"
        "Please provide your final judgment and reasoning."
    )
    # Call your general LLM here — GPT-4o, Claude, etc.
    ...

Batch processing with confidence

Single-item calls are straightforward. Batch evaluation introduces two wrinkles: you need per-item confidence tracking, and you want to avoid paying for LLM fallback on every item in the batch just because the batch contains some low-confidence ones.

Structure the batch loop so it separates high-confidence items (straight to action) from low-confidence items (routed to fallback), processes both, and rejoins them in original order for downstream consumers:

from concurrent.futures import ThreadPoolExecutor, as_completed

def process_batch(items: list[str], threshold: float = 0.8) -> list[dict]:
    results = [None] * len(items)
    needs_fallback = []

    # First pass: call Jev for all items
    for i, item in enumerate(items):
        r = call_jev(item)
        if r["confidence"] >= threshold:
            results[i] = {"source": "jev", "result": r}
        else:
            needs_fallback.append((i, item, r))

    # Second pass: fallback only for the low-confidence subset
    with ThreadPoolExecutor(max_workers=5) as ex:
        futures = {
            ex.submit(llm_review_with_jev_context, item, jev_r): idx
            for idx, item, jev_r in needs_fallback
        }
        for future in as_completed(futures):
            idx = futures[future]
            results[idx] = {"source": "llm_fallback", "result": future.result()}

    return results

Tag the source field ("jev" vs "llm_fallback") so you can measure fallback rate per batch over time. A sudden rise in fallback rate for a specific item type is a signal that the question needs to be rewritten.

Common anti-patterns

Retrying in a loop until confidence is high enough. This is the most common mistake. If the model returns low confidence twice on the same input, the third call will usually return low confidence too. A loop does not resolve structural uncertainty; it burns money and adds latency. Set a single retry limit and route after that.

Using a single threshold for all question types. A 0.8 threshold on a yes/no judgment question is not the same as a 0.8 threshold on a 1–10 scoring question. Score questions tend to have more inherent ambiguity in the middle of the scale, so their confidence distributions are lower on average. Calibrate per question type, not per deployment.

Dropping the rationale field. The rationale field is cheap to log and expensive to lose. When a high-confidence answer turns out to be wrong, the rationale is usually the fastest way to understand why the model was certain about the wrong thing.

Conflating unclear with low confidence. unclear is a structural refusal — the question could not be answered from the provided context. Low confidence is an uncertain answer. Routing them to the same fallback wastes human review time on items that need the question fixed, not a human reviewer.

Setting the threshold once and forgetting it. Input distributions shift. Upstream data sources change format. New edge cases appear. A threshold that was right six months ago may be wrong today. The mean confidence metric described below is specifically designed to catch this.

Monitoring the loop

Three metrics catch most failures before users do:

Log the route taken (automate, retry, human, llm_review) alongside every answer. A weekly rollup of fallback route mix catches retry loops — if the retry share is growing week over week, your question wording has drifted and the sharpened question is not actually sharper.

The escalation rate is the fraction of items that reach a human. If it jumps without a deploy, one of three things happened: your upstream data changed, a new type of input is coming in that your questions were not written for, or a bug in your confidence-reading code is misrouting items. Check in that order.

See the cost and latency guide for how to budget the LLM fallback cost as a fraction of total spend. The rule of thumb: if your mean confidence is above 0.85, the LLM fallback affects fewer than 15% of calls, and the incremental cost is usually acceptable relative to the accuracy gain.

The invoice approval gate case is a good study in conservative thresholds — automate almost nothing, review almost everything — while the support routing scenario shows the opposite balance in a high-volume flow. And when an “error” is really a misread response shape rather than low confidence, the troubleshooting guide has the fix.

This guide applies to Jev 1.13.