How to Use Jev for Product Q&A Scoring (Quality at Scale)

TL;DR: You cannot read every answer merchants write under your product questions, but you can grade a sample of them every day. One Jev scoring call per dimension — relevance to the question, consistency with the product page, tone — gives you structured scores with confidence and rationale. Thresholds turn scores into actions: coach, keep, or remove. Pipeline and settings below.

Scoring dimensions

Pick few dimensions and define each as its own question:

• Answer relevance — does it answer what was actually asked? Fixture:

{"answer":8,"scale":[1,10],"confidence":0.86,"rationale":"Directly answers the warranty length question with a specific number."}

• Product-page consistency — does it contradict the listed specs? A low score here is your strongest removal signal.
• Tone — respectful, no spam or promo links.

One scoring call per dimension keeps each question unambiguous and lets you act per dimension: a relevance failure means a bad answer, while a consistency failure might mean the product page itself needs fixing.

QA pipeline

Answers written under product Q&A
   |
   v
Daily sampler (e.g. every Nth answer per category)
   |
   v
Jev scoring call per dimension (1-10)
   |
   v
Combine scores per answer
   |
   +--> any dimension < 4 --> flag for removal or merchant rework
   |
   +--> all dimensions >= 4 and < 7 --> published, tracked in trend
   |
   +--> all dimensions >= 7 --> counts toward seller quality metrics
   |
   v
Weekly report: score trends per merchant / category

Sampling bounds cost; the trend report is where coaching targets come from.

Minimal implementation

import os, json, requests

URL = "https://openrouter.ai/api/v1/chat/completions"
# Confirm the exact model slug on the OpenRouter model page.
MODEL = "typesafe/jev-1.13"
DIMENSIONS = {
    "relevance": "Does this answer the customer's question?",
    "consistency": "Is it consistent with the product page specs below?",
    "tone": "Is it respectful and free of spam or promotion?",
}

def score(dimension: str, question: str, answer: str, page: str) -> dict:
    resp = requests.post(
        URL,
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={"model": MODEL, "temperature": 0, "messages": [
            {"role": "system", "content": f"{DIMENSIONS[dimension]} Score 1-10."},
            {"role": "user", "content":
             f"Question: {question}\nAnswer: {answer}\nPage: {page[:2500]}"}]},
        timeout=30,
    )
    resp.raise_for_status()
    # example fixture:
    # {"answer":8,"scale":[1,10],"confidence":0.86,"rationale":"..."}
    return json.loads(resp.json()["choices"][0]["message"]["content"])

def qa_answer(item: dict, low=4, high=7):
    dims = [score(d, item["q"], item["a"], item["page"]) for d in DIMENSIONS]
    worst = min(d["answer"] for d in dims)
    if worst < low:
        return "flag-rework", dims
    if worst < high:
        return "publish-track", dims
    return "quality-credit", dims

Threshold suggestions

Scoring is one of Jev’s three primitives, and the cost-and-latency guide shows why sampled cheap calls beat reading everything manually. The product FAQ answer scoring case study includes measured score trends across categories.

Applies to Jev 1.13.

FAQ

• What dimensions should answers be scored on? Three or four job-relevant ones — answering the question, consistency with the product page, tone — each as its own scoring call on a fixed scale.
• Do I have to score every answer? No: a daily sample flags the worst answers for rework or removal and tracks trends; full coverage is only for compliance-critical categories.
• How do scores become action? Action bands: below-threshold dimensions trigger merchant rework or removal, mid band stays published and tracked, top band feeds seller quality metrics — rationale stored with every score.