Designing Jev Questions and State: A Practical Guide

A judgment model amplifies whatever you feed it: a well-bounded Jev question gives you stable, confidence-scored answers for months; a vague one gives you answers that drift with every slightly different input. Question design is therefore the highest-leverage skill in a Jev integration. This guide covers how to write stable judgment, choice and scoring questions, how to define boundaries the model can actually apply, and how to pass context state between calls.

TL;DR: State the answer space explicitly, define boundaries with concrete rules or examples, add 2–5 few-shot examples for the tricky cases, and pass context as explicit text in the question. If a question cannot be answered the same way twice, fix the question before blaming the model.

The anatomy of a stable question

A stable question has four parts, in this order:

1. The primitive. “Judgment question:”, “Choice question:” or “Scoring question (1-10):” — name the type so the answer space is unambiguous.
2. The criterion. One sentence stating exactly what makes the answer yes, or which option wins, or what a high score means.
3. The boundaries. What counts as the edge cases, stated as rules or examples.
4. The input. The item to judge, clearly delimited.

Compare a weak and a strong version:

Weak:   "Is this review bad?"

Strong: "Judgment question: should this product review be hidden for
        violating the no-promotional-links policy? The policy prohibits
        links to sellers, coupon codes, and contact info. Mentions of the
        product's own brand do not count. Review: \"...\""

The weak version invites drift: “bad” has no answer space. The strong version names the primitive, the criterion, the boundary cases and the input. Its response — example fixture, confirm exact field names in the official documentation — reads like this:

{
  "answer": "no",
  "confidence": 0.93,
  "rationale": "The review mentions the product's own brand only, which the policy explicitly excludes from violations."
}

Option enumeration for choice questions

Choice questions live or die on the option list. Three rules:

• Make the list exhaustive or add an escape option. If “other” is a legitimate outcome, enumerate it; otherwise the model is forced to pick a wrong bucket.
• Make options mutually exclusive in one sentence each. A one-line definition per option beats a name alone: “billing — disputes about charges and refunds” versus just “billing”.
• Keep the number manageable. Past roughly a dozen options, accuracy degrades; consider a two-stage choice instead.

import json, os, requests

prompt = (
    "Choice question: which queue does this ticket belong to? "
    "Definitions: billing - charges, refunds, invoices; "
    "technical - product errors and bugs; "
    "account - login, permissions, profile. "
    "Ticket: \"...\""
)
resp = requests.post(
    "https://openrouter.ai/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
    json={
        # Confirm the exact model slug on the OpenRouter model page
        "model": "typesafe/jev-1.13",
        "messages": [{"role": "user", "content": prompt}],
    },
    timeout=30,
)
route = json.loads(resp.json()["choices"][0]["message"]["content"])

The definitions turn bare labels into boundaries; confirm the exact model slug on the OpenRouter model page before shipping. The parsed route is an example fixture — the response shape is illustrative and official field names should be confirmed in the official documentation, taking the form {"answer":"technical","confidence":0.9,"rationale":"..."}.

Few-shot examples earn their keep at the boundary

Abstract rules cover the middle of the distribution; examples cover the edges. Put 2–5 examples in the question for the cases where you have been burned:

Examples:
- "Buy now at deals.example, 50% off!" -> hide (promotional link)
- "This product broke after two weeks, very disappointed." -> keep (genuine complaint)
- "Great product, contact me at me@example.com for bulk orders." -> hide (contact info)
Now judge: Review: \"...\"

Each example is a boundary made concrete. When answers drift, add an example for the drifting case rather than growing the rule text.

Passing state between calls

Jev calls are single-turn, so context must travel inside the question. Practical patterns:

Keep the state block short and factual — state is context for the judgment, not a second question. When a decision depends on an earlier Jev answer, copy the answer and the one-line rationale rather than the whole fixture.

The multilingual review moderation case shows these techniques combined — enumerated policy boundaries plus few-shots — across several languages, and the confidence-and-fallback guide explains what to do with the unclear answers your boundaries will occasionally produce.

This guide applies to Jev 1.13.