Designing Jev Questions and State: A Practical Guide

A judgment model amplifies whatever you feed it: a well-bounded jev api question gives you stable, confidence-scored answers for months; a vague one gives you answers that drift with every slightly different input. Question design is therefore the highest-leverage skill in a jev tutorial integration. This guide covers how to write stable judgment, choice and scoring questions, how to define boundaries the model can actually apply, how to pass context state between calls, how confidence interacts with state, and how to wire state persistence into automation workflows like n8n.

TL;DR: State the answer space explicitly, define boundaries with concrete rules or examples, add 2–5 few-shot examples for the tricky cases, and pass context as explicit text in the question. Gate downstream calls on the confidence field. If a question cannot be answered the same way twice, fix the question before blaming the model.

The anatomy of a stable question

A stable question has four parts, in this order:

1. The primitive. “Judgment question:”, “Choice question:” or “Scoring question (1-10):” — name the type so the answer space is unambiguous.
2. The criterion. One sentence stating exactly what makes the answer yes, or which option wins, or what a high score means.
3. The boundaries. What counts as the edge cases, stated as rules or examples.
4. The input. The item to judge, clearly delimited.

Compare a weak and a strong version:

Weak:   "Is this review bad?"

Strong: "Judgment question: should this product review be hidden for
        violating the no-promotional-links policy? The policy prohibits
        links to sellers, coupon codes, and contact info. Mentions of the
        product's own brand do not count. Review: \"...\""

The weak version invites drift: “bad” has no answer space. The strong version names the primitive, the criterion, the boundary cases and the input. Its response — example fixture, confirm exact field names in the official documentation — reads like this:

{
  "answer": "no",
  "confidence": 0.93,
  "rationale": "The review mentions the product's own brand only, which the policy explicitly excludes from violations."
}

Option enumeration for choice questions

Choice questions live or die on the option list. Three rules:

• Make the list exhaustive or add an escape option. If “other” is a legitimate outcome, enumerate it; otherwise the model is forced to pick a wrong bucket.
• Make options mutually exclusive in one sentence each. A one-line definition per option beats a name alone: “billing — disputes about charges and refunds” versus just “billing”.
• Keep the number manageable. Past roughly a dozen options, accuracy degrades; consider a two-stage choice instead.

import json, os, requests

prompt = (
    "Choice question: which queue does this ticket belong to? "
    "Definitions: billing - charges, refunds, invoices; "
    "technical - product errors and bugs; "
    "account - login, permissions, profile. "
    "Ticket: \"...\""
)
resp = requests.post(
    "https://openrouter.ai/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
    json={
        # Confirm the exact model slug on the OpenRouter model page
        "model": "typesafe/jev-1.13",
        "messages": [{"role": "user", "content": prompt}],
    },
    timeout=30,
)
route = json.loads(resp.json()["choices"][0]["message"]["content"])

The definitions turn bare labels into boundaries; confirm the exact model slug on the OpenRouter model page before shipping. The parsed route is an example fixture — the response shape is illustrative and official field names should be confirmed in the official documentation, taking the form {"answer":"technical","confidence":0.9,"rationale":"..."}.

Few-shot examples earn their keep at the boundary

Abstract rules cover the middle of the distribution; examples cover the edges. Put 2–5 examples in the question for the cases where you have been burned:

Examples:
- "Buy now at deals.example, 50% off!" -> hide (promotional link)
- "This product broke after two weeks, very disappointed." -> keep (genuine complaint)
- "Great product, contact me at me@example.com for bulk orders." -> hide (contact info)
Now judge: Review: \"...\"

Each example is a boundary made concrete. When answers drift, add an example for the drifting case rather than growing the rule text. Longer rule paragraphs rarely outperform a well-chosen example — the model infers the rule from the instance more reliably than from an abstract description.

If you find yourself writing more than five few-shot examples, that is usually a signal that the question criterion is still underspecified. Step back and ask whether the criterion sentence itself can be tightened before adding more cases.

State lifecycle across calls and requests

Jev api calls are single-turn by design. That means state does not persist between calls — you carry it. Understanding the lifecycle of jev state is what separates a brittle integration from a stable one.

Call-scoped state lives and dies with a single request. The transcript snippet you summarized, the current pipeline stage, or the immediate prior jev answer are all call-scoped. Pass them in the question text; do not try to cache them elsewhere.

Session-scoped state spans several calls within one user session: the accumulated judgments so far, a running violation count, or a user-declared preference from earlier in the conversation. Keep this in your application layer (an in-memory dict keyed by session ID is fine for short-lived sessions) and inject the relevant slice into each question.

Account-scoped state is long-lived: the user’s subscription tier, their historical escalation rate, or standing policy exceptions. This belongs in your database. Fetch it once per pipeline run and embed it in the state block.

The rule of thumb: the jev api only knows what you tell it in the current request. Anything that must be consistent across calls has to be managed by your code.

def build_prompt(criterion, item, session_state, account_state):
    state_block = (
        f"Account context: {account_state['tier']} plan, "
        f"violations this month: {account_state['violation_count']}.\n"
        f"Session context: {session_state['prior_judgments_summary']}\n"
    )
    return (
        f"Judgment question: {criterion}\n"
        f"{state_block}"
        f"Input: {item}"
    )

Keep the state block factual and minimal. If it exceeds two or three sentences, compress it — verbose state reduces the effective attention the model can place on the input.

When should state travel in the question versus be held server-side? Pass state in the question when it affects this specific judgment directly. Hold it server-side when it is needed across many calls but rarely changes the answer for any individual one (e.g., the user’s locale if your questions are already translated, or a global kill-switch flag). Over-passing state inflates token cost and can crowd out the actual input.

Questions design: granularity, count, and conflict

How many jev questions should one pipeline step contain? The practical answer is one judgment per call. Compound questions — “Is this spam AND should it be escalated?” — force the model to make two independent decisions and report them as a single answer. The compound form hides which sub-criterion caused a given result, making debugging and iterating harder.

One question, one criterion, one answer. If you need two judgments, make two calls. The added latency is usually less costly than the debugging time saved when one of the two criteria is misfiring.

Question granularity matters too. A question like “score this candidate on communication skills (1-10)” is too coarse if communication splits into written clarity, responsiveness, and articulation under pressure. Better to score each sub-dimension separately and aggregate in your code, where the aggregation logic is visible and adjustable.

Conflicting questions occur when the same item is scored by two questions whose criteria overlap but are not identical. For example, a “relevance” question and a “quality” question on the same document may produce contradictory signals because relevance is partly a quality signal. Define the criteria to be orthogonal — relevance is about topic fit; quality is about writing standard, not about whether the topic is correct — and the conflict disappears.

Repeated questions with slightly different wording are sometimes used as a consistency check. This works, but it is expensive. A cheaper consistency check is to keep your few-shot examples versioned: if the model’s answers on your example set shift between deploys, the question or model changed, and you should audit before promoting the new version.

EXAMPLE_FIXTURES = [
    {"input": "Great product!", "expected_answer": "keep", "expected_confidence_min": 0.85},
    {"input": "Buy at spam.example", "expected_answer": "hide", "expected_confidence_min": 0.90},
]

def smoke_test(prompt_template):
    for ex in EXAMPLE_FIXTURES:
        result = call_jev(prompt_template.format(input=ex["input"]))
        assert result["answer"] == ex["expected_answer"]
        assert result["confidence"] >= ex["expected_confidence_min"]

Run this smoke test on every question change before deploying. It takes under a second and catches the majority of regressions.

Confidence and state: handling low-confidence answers

The confidence field is not decorative. It is your signal that the model reached the boundaries of the information you provided. When confidence drops, the correct response is usually to re-ask with more context, not to accept the answer and move on.

Gate downstream state on confidence. Before embedding a jev answer into the next call’s state block, check the field:

CONFIDENCE_THRESHOLD = 0.75

def safe_embed_prior_answer(prior_result, state):
    if prior_result["confidence"] >= CONFIDENCE_THRESHOLD:
        state["prior_judgment"] = (
            f"Earlier judgment: {prior_result['answer']} "
            f"({prior_result['confidence']:.2f}) because "
            f"{prior_result['rationale']}"
        )
    else:
        state["prior_judgment"] = "Earlier judgment: unclear (low confidence — omit from this call)"
    return state

Passing a low-confidence answer as though it were certain is one of the most common jev state bugs. The downstream call treats it as a reliable signal; it is not.

Re-asking with expanded context is the right first move when confidence is below threshold. Add one specific few-shot example for the type of item that produced the low-confidence answer, then re-run the same call:

def reask_with_example(original_prompt, low_conf_item, correct_answer, explanation):
    extra_example = (
        f"Additional example: \"{low_conf_item}\" -> {correct_answer} ({explanation})\n"
    )
    return extra_example + original_prompt

If confidence is still low after re-asking, route to human review. Do not stack more than one re-ask automatically — two low-confidence answers on the same item indicate the question criterion needs reworking, not more examples.

Confidence and question count interaction: if you ask ten questions in sequence and each one has a 5% chance of landing below threshold, roughly half your pipeline runs will hit at least one low-confidence call. Design the fallback path before you hit production.

Passing state in multi-turn and multi-request scenarios

A multi-turn conversation or a multi-step automation pipeline both require explicit state management. The patterns differ slightly.

Multi-turn (interactive): The user is providing input at each step. State includes the conversation history summarized for context:

Context: User is screening a resume for a junior engineer role.
Prior turns:
  - Turn 1: scored technical skills 7/10 (confidence 0.91)
  - Turn 2: scored communication 6/10 (confidence 0.88)
Now judge: Scoring question (1-10): overall hire recommendation.
Candidate summary: "..."

Notice that only the prior scores and their confidence travel forward, not the full prior prompts. Summarize rather than concatenate to keep the state block short.

Multi-request (batch pipeline): Each item is independent, but pipeline-level context is shared. Keep a shared context dict that does not grow with the number of items:

pipeline_context = {
    "policy_version": "v2.3",
    "campaign_id": "camp_abc",
    "reviewer_tier": "enterprise",
}

def process_item(item, pipeline_context):
    prompt = build_prompt(
        criterion=CRITERION,
        item=item,
        shared_context=pipeline_context,
    )
    return call_jev(prompt)

If a pipeline step’s result affects subsequent steps — for example, a routing decision that changes the criterion for the next call — pass only the answer and the one-line rationale, not the full response object.

State persistence and idempotency in n8n and automation workflows

When wiring jev api calls into n8n or similar automation platforms, state persistence and idempotency become production concerns.

Persistence: n8n workflow runs can fail mid-execution. If a jev call succeeded but the downstream write failed, rerunning from the start doubles the call count. Use n8n’s built-in execution data to checkpoint state after each successful jev call:

{
  "item_id": "review_9182",
  "jev_answer": "hide",
  "jev_confidence": 0.93,
  "jev_called_at": "2026-09-28T08:42:00Z"
}

Store this checkpoint in a database node before continuing the workflow. On retry, check whether jev_called_at is already set for the item — if so, skip the jev call and use the stored answer.

Idempotency: A jev question is idempotent by nature — the same question with the same input returns the same answer. The risk is in the state you attach. If the session state block changes between a failed run and the retry (because another event updated it), the answer may change too. Snapshot the state block at call time and store it alongside the answer so you can audit which state produced which result.

checkpoint = {
    "item_id": item_id,
    "prompt_snapshot": prompt,        # the exact string sent
    "answer": result["answer"],
    "confidence": result["confidence"],
    "rationale": result["rationale"],
    "called_at": datetime.utcnow().isoformat(),
}
db.upsert("jev_checkpoints", checkpoint, conflict_on="item_id")

n8n-specific pattern: use the “Set” node to assemble the state block from upstream nodes before the HTTP Request node that calls the jev api. This makes the state construction visible in the workflow graph and easier to debug than building it inside a Function node:

[Fetch account data] → [Fetch session state] → [Set: build state block] → [HTTP: jev api call] → [Set: checkpoint] → [Downstream action]

If the HTTP node fails, n8n retries from that node; the account and session data are already set in the execution context and do not need to be fetched again.

Common errors and debug paths

Symptom: answers that were stable last week are drifting now. Most likely cause: the input distribution changed, not the model. Log a sample of the inputs alongside their answers and look for new patterns in the low-confidence cases. Add one targeted few-shot example for each new pattern.

Symptom: confidence is consistently around 0.5 for a subset of inputs. The question boundary is genuinely ambiguous for that subset. The model is not broken — it is telling you that the criterion does not cleanly separate yes from no for this class. Narrow the criterion or split into two questions.

Symptom: the jev api call succeeds but the parsed answer is missing expected fields. You are parsing a cached or fallback response that was not the full jev output. Confirm the Content-Type is application/json and that the outer choices[0].message.content is a valid JSON string before parsing. A network proxy or rate-limit response can return a non-JSON body that json.loads will raise on.

Symptom: the state block from a prior session is affecting current results unexpectedly. Session IDs are colliding, or you are reusing a state dict across requests without clearing it. Key every state dict by a combination of session ID and request timestamp. Clear call-scoped keys explicitly at the start of each call.

Symptom: n8n workflow retries are calling jev twice for the same item. The checkpoint write is happening after the HTTP node but before an error that causes the retry. Move the checkpoint write to immediately after the HTTP node, before any transformation step that might fail.

The multilingual review moderation case shows the enumerated-policy and few-shot techniques combined across several languages. The confidence-and-fallback guide covers the full decision tree for unclear answers, including human escalation thresholds and automatic re-ask strategies.

This guide applies to Jev 1.13.