Jev Case Study: Scoring Product Q&A Answers (Full Code)

Updated 2026-09-21 Applies to Jev 1.13 Example fixture

TL;DR: This Jev case study quality-gates marketplace product Q&A: every seller answer is scored on a 1-10 scale by one Jev 1.13 scoring question before it publishes. Scores of 7+ go live; the rest go back to the seller’s dashboard. Full code below.

Scenario

Our marketplace lets sellers answer buyer questions on their own product pages. In theory this is great documentation; in practice the answers range from genuinely useful to “ask the description” to marketing copy with the question’s keywords pasted in. Publishing everything made the Q&A section useless; reviewing every answer by hand did not scale past a few hundred a week.

The gate we settled on is one scoring question per answer:

Score how well this answer addresses the buyer’s question, on a scale of 1 to 10.

7 or above: publish automatically.
Below 7: send back to the seller’s dashboard with the Jev rationale attached, so the seller knows what was missing instead of just getting a rejection.
Low confidence at any score: route to the human moderation queue — the score is telling us it is not sure.

This uses the third of Jev’s three question types (judgment, choice, scoring — see the three primitives guide). Jev is TypeSafe AI’s System One judgment model, released September 2026; the call goes through the OpenRouter OpenAI-compatible endpoint. The scoring output is typed JSON, so the publish gate is a numeric comparison plus a threshold, with no text parsing in the hot path.

Request

{
  "model": "typesafe/jev-1.13",
  "messages": [
    {
      "role": "system",
      "content": "Scoring question. Score how well the answer addresses the buyer's question, on a scale of 1 to 10. Reply with JSON only: {\"answer\": <1-10>, \"scale\": [1, 10], \"confidence\": <0-1>, \"rationale\": <one sentence>}"
    },
    {
      "role": "user",
      "content": "Question: Is this desk made of solid wood or veneer? What is the load limit of the drawers?\n\nAnswer: Premium quality desk! Great value and fast shipping. Check our storefront for more deals."
    }
  ]
}

Response

Example fixture — illustrative output, not a live capture:

{
  "answer": 2,
  "scale": [1, 10],
  "confidence": 0.86,
  "rationale": "The answer does not address either the material or the drawer load limit and reads as promotional copy rather than a product answer."
}

This is the fixture we show on purpose: the gate earns its keep on bad answers, not good ones. A passing answer produces the same shape with a high answer and a rationale naming which part of the question it covered.

Reproduce

Set OPENROUTER_API_KEY and run:

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "typesafe/jev-1.13",
    "messages": [
      {
        "role": "system",
        "content": "Scoring question. Score how well the answer addresses the buyer'"'"'s question, on a scale of 1 to 10. Reply with JSON only: {\"answer\": <1-10>, \"scale\": [1, 10], \"confidence\": <0-1>, \"rationale\": <one sentence>}"
      },
      {
        "role": "user",
        "content": "Question: Is this desk made of solid wood or veneer? What is the load limit of the drawers?\n\nAnswer: Premium quality desk! Great value and fast shipping. Check our storefront for more deals."
      }
    ]
  }'

Parse the message content of the OpenAI-compatible response as JSON to get the fields above.

Key parameters

Field	Value	Why it matters
`model`	`typesafe/jev-1.13`	Pin the version so the publish threshold stays meaningful over time. Confirm the exact slug on the OpenRouter model page.
`messages[0]` (system)	Scoring question + scale + JSON shape	“Addresses the buyer’s question” is the rubric — say it exactly, every call.
`messages[1]` (user)	Question and answer, labeled	Two labeled blocks; multi-part questions are fine, the rubric rewards covering all parts.
`answer` (response)	`2`	The publish gate: `>= 7` → publish, else return to seller.
`scale` (response)	`[1, 10]`	Echoes the rubric; a cheap per-call sanity check.
`confidence` (response)	`0.86` (example fixture)	Low confidence routes to humans regardless of the score.
`rationale` (response)	One sentence	Returned to the seller as actionable feedback, not just a rejection.
Price	$0.0462 per 1M input tokens	One call per answer; verify current pricing on OpenRouter before budgeting.

Notes

Responses are example fixtures; verify field names against official docs before relying on them. Official API details live at typesafe.ai — see the System One announcement post.
Confirm the exact model slug (typesafe/jev-1.13) on the OpenRouter model page before shipping.
Watch the score distribution after launch: if nearly everything scores 8+, your threshold is too low or your rubric is too vague — tighten the rubric wording in the system prompt.
If scores look inconsistent between runs, see the troubleshooting guide first; identical inputs should give stable scores at the same pinned version.
Applies to Jev 1.13 (released September 2026) via the OpenRouter demonstration channel.