Jev Image Inputs and Box Questions: Sending Images and Reading Bounding Boxes

Updated Applies to Jev 1.14

Every guide on this site up to now has treated Jev as a text engine: you put text into state, you ask Choice, Score and Noul questions, you get typed answers back. That description was accurate for the primitives, and it is what nearly every tutorial repeats. It is also incomplete.

Jev reads images. It reads them on 1.13 as well as 1.14, it reads them in three different places inside a request, and since 1.14 it can answer a fourth question type — Box — that returns where something is inside the picture. This page is the missing chapter: what you actually send, what comes back, what the hard limits are, and what the published measurements say about whether any of it works yet.

Nothing here is inferred. Every figure comes from TypeSafe’s own documentation, read on 2026-11-11, plus one live read of the OpenRouter catalog for the model’s declared modalities. Where the docs and the catalog describe the same thing differently, both are quoted.

The one-line version

An image request is a normal System One request with two additions: a top-level assets map holding the images, and reference strings written where the image belongs. Add the header TypeSafe-Danger-Zone-Images: true, or the request fails with 400. A Box question takes only type and instructions, returns {top, bottom, left, right} as fractions plus a confidence, and always returns a box — including when the target is not there, which is why the documentation tells you to pair it with a Noul.

Jev reads images on 1.13 too

This is the part people get wrong first, because the 1.14 release notes announce images and Box in the same breath. The official changelog separates them:

The images page repeats it in one sentence: “Jev 1.13 and Jev 1.14 both accept images. Box questions need Jev 1.14.” The models table agrees — input is “Text, images” on both rows, while question types are Choice / Score / Noul / Box on 1.14 and Choice / Score / Noul on 1.13.

The practical consequence: if you are pinned to 1.13 and only need image input, you do not need to change versions at all. You need the header, and you need to be aware that you have no way to ask for a location.

There is a third gate on top of the version gate: images and Box are off by default and require the header on every request that carries either. The error table gives the exact 400 for forgetting it — “Images are not turned on: the request has an image or a Box question but no TypeSafe-Danger-Zone-Images: true header.”

The assets map and the reference rules

You never inline an image into a prompt string as a data URL. Instead each image is listed once in a top-level assets object, keyed by a reference string you choose, and you write that same string wherever the image belongs:

{
  "model": "jev-latest",
  "state": { "photo": "a3f1c2e8-found" },
  "assets": {
    "a3f1c2e8-found": { "type": "image", "format": "jpg", "data_base64": "/9j/4AAQSkZJRg..." },
    "77b0d4a1-poster": { "type": "image", "format": "jpg", "data_base64": "/9j/4AAQSkZJRg..." }
  },
  "questions": {
    "same_dog": {
      "type": "noul",
      "instructions": "Is the dog in `photo` the same dog as in this picture: 77b0d4a1-poster?"
    }
  }
}

That example is the documentation’s own three-way anatomy, trimmed to two images. The reference a3f1c2e8-found stands in as the whole value of state.photo; 77b0d4a1-poster is embedded mid-sentence inside a Noul’s instructions. The third position — a Choice option’s description — appears further down.

An asset entry has four possible fields. type is optional and must be "image" if present. part is the multipart file-part name. data_base64 is the file, standard base64 alphabet with = padding, no whitespace, no data: prefix. format is "png", "jpg" or "jpeg", lowercase, required with data_base64 and optional with part. If an asset somehow carries both part and data_base64, the API uses part.

The reference rules are where requests break, and they are all stated explicitly:

That fourth rule is the one that costs money if you miss it. An image that is 10 MiB after processing, referenced four times, is 40 MiB of expanded request and is over the 32 MiB limit before you send it.

The three places an image can go

In state

An image in state is part of the state, exactly like a string or a JSON field. Every question in the request is evaluated against it, so this is the cheap place to put an image that several questions share. A state can mix images with text and hold several images, labelled or nested — the documentation’s example is a short history of camera frames:

{
  "state": {
    "frames": [
      { "seconds_ago": 2, "frame": "5c0e9a12-f1" },
      { "seconds_ago": 1, "frame": "1d7be0c4-f2" },
      { "seconds_ago": 0, "frame": "e48a6f30-f3" }
    ]
  }
}

A question can then ask about the array as a whole (“Did the door in frames open during these frames?”) or address one element by path, such as `frames[2].frame`.

In instructions

An image inside a question’s instructions belongs to that question and is sent only with it. The documented use is comparison — one photo in the state, one reference photo in the question:

Is the person in door the same person as in this badge photo: 9a21c7d3-badge?

The cost note is explicit: an image in instructions is sent once for every question that includes it. When several questions need the same picture, put it in state once and refer to it by path — fewer tokens, less of the size budget.

In criteria

Choice options, Score levels, and a Noul’s true and false definitions can each carry images. This is the “define the options by example” pattern, and it is the most interesting use of images in Jev, because the model is not asked to describe the picture — it is asked which of your example pictures the input matches.

{
  "model": "jev-latest",
  "state": { "camera": "e48a6f30-cam" },
  "questions": {
    "gesture": {
      "type": "choice",
      "instructions": "Which gesture is the hand in `camera` making?",
      "criteria": {
        "thumbs_up": "0b6c51aa-thumb",
        "open_palm": {
          "description": "A hand held up with the palm facing the camera.",
          "examples": ["4fd2e7b9-palm1", "c83a0e15-palm2"]
        },
        "fist": "a7e94d02-fist",
        "none": "No hand, or a hand that matches none of the examples."
      }
    }
  }
}

An option can be an image on its own, a description with one or more example pictures, or plain text. Both description and examples are the documentation’s own naming choices in that example, not required fields — criteria accept any structure. A Score can use one example picture per level, and a Noul can define true and false with pictures: {"true": {"description": "Correctly assembled", "examples": ["<reference>"]}, "false": "Parts missing or in the wrong place."}.

The documentation summarises the whole design space as six patterns, and it is worth knowing which one you are in before you write the request:

PatternWhat it doesWhere the image goes
Compare against a known pictureSame person, product or place as a reference?Known picture in instructions, photo in state
Define options by exampleEach Choice option gets one or a few example picturescriteria
What changedSend the previous frame or a short history, ask about motion or arrivalsstate
Look closer when neededAsk on a small copy first, send a bigger one or a full-resolution crop only when the answer needs detailstate
Search a large imageCut into tiles, ask a Choice with the tiles as options, repeat on the chosen tilecriteria
Find where it isAsk a Box for the edges of the targetstate, optionally a target picture in instructions

Box: the fourth question type

A Box question has exactly two fields. type is always "box". instructions is what to find and in which image — and it does not have to be a question at all; the documentation’s example is the fragment “the coffee mug in photo”. Box takes no criteria.

curl https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer ***" \
  -H "TypeSafe-Danger-Zone-Images: true" \
  -F 'request={
    "model": "jev-latest",
    "state": {"photo": "6e1d0b7c-kitchen"},
    "assets": {"6e1d0b7c-kitchen": {"type": "image", "part": "kitchen-photo"}},
    "questions": {
      "mug": {"type": "box", "instructions": "the coffee mug in `photo`"}
    }
  };type=application/json' \
  -F 'kitchen-photo=@kitchen.jpg;type=image/jpeg'

The answer carries a box object and a confidence:

{
  "model": "jev-1.14.0",
  "answers": {
    "mug": {
      "type": "box",
      "box": { "top": 0.25, "bottom": 0.82, "left": 0.44, "right": 0.84 },
      "confidence": 0.92
    }
  },
  "usage": { "input_tokens": 290, "output_tokens": 21 }
}

left is always less than right and top always less than bottom. All four are fractions from 0 to 1 measured from the top-left corner — left and right against the image width, top and bottom against the height. That normalisation is the reason one answer works at any resolution: multiply each edge by the pixel dimension and you have a crop box.

from PIL import Image

image = Image.open("kitchen.jpg")
width, height = image.size
mug = response["answers"]["mug"]
edges = mug["box"]

box = (
    round(edges["left"] * width),
    round(edges["top"] * height),
    round(edges["right"] * width),
    round(edges["bottom"] * height),
)
image.crop(box).save("mug.jpg")

The always-returns-a-box problem

A Box answer is always a box. There is no “not found”. Ask where the mug is in a photo with no mug in it and you get four edges anyway, with some confidence attached.

The documented remedy is to ask a Noul in the same request and use it as a gate:

"questions": {
  "mug_visible": { "type": "noul", "instructions": "Is there a coffee mug in `photo`?" },
  "mug": { "type": "box", "instructions": "the coffee mug in `photo`" }
}

Then ignore the box when mug_visible is low. Both questions are evaluated against the same state in one request, so the gate costs one question, not one round trip.

Three more rules from the Box page are worth internalising. One box per question — Box is designed to find a single target, so several different targets means several Box questions. Describe the one you want when the image might contain several of the same thing: “the mug nearest the sink in photo”. Crop before looking for small things — a target a few dozen pixels wide in a very large image can be missed, so find the region first, for example with a Choice whose options are tiles, then Box on that tile and convert the edges back to fractions of the full image.

There is also no recommended confidence threshold. The documentation says to tune it on your own images, and offers two remedies when confidence is low: ask again on a better image (a larger version, or a higher-resolution crop around the box), and pad the box before cropping so a crop cannot cut off part of the target.

Hard limits

These are rejection thresholds, not resizing hints. An image outside them is rejected; the API does not shrink it for you.

LimitValue
FormatPNG or JPEG, still images only. Animated PNGs are rejected.
File size16 MiB per image file (after base64 decoding)
Width and height280 to 1568 pixels each; either side under 280 is rejected
Total pixels78,400 to 2,458,624 (a 280×280 to a 1568×1568 square)
Request body32 MiB, including JSON, base64 text and multipart framing; headers and compression do not raise it
All images in the request32 MiB, shared by every distinct image after processing
Expanded request32 MiB after every reference is replaced by its image
Reference strings1,024 bytes each, 65,536 bytes total

All limits include both ends of the range. If any image in a request is rejected, the whole request fails — there are no partial answers. How many images fit in one request depends on these limits and the context length, not on a separate image quota: images have no separate rate limit and count against the same request limits as text.

The 32 MiB “after processing” row deserves attention. Processing re-encodes each image and can make it larger than the file you sent, so a request of modest uploads can still exceed the budget.

Preparing images

Six instructions from the documentation, all of which prevent specific rejections:

The documented Pillow snippet does the first two together:

from PIL import Image, ImageOps

with Image.open("photo.jpg") as original:
    img = ImageOps.exif_transpose(original)   # rotate the pixels upright
    img.thumbnail((1568, 1568))               # longest side at most 1568
    # thumbnail only shrinks: upscale or pad an image under the minimum separately
    img.convert("RGB").save("photo-ready.jpg", "JPEG", quality=90)  # baseline JPEG

Under multipart, the JSON body goes as the first part, named request, with content type application/json; each image is its own file part, and the asset’s part field must name it. Part names must be unique and non-empty, request is reserved, each image part’s content type must be image/png, image/jpeg, image/jpg, application/octet-stream or absent, and a part that no asset names — or an asset whose part is missing — returns 400.

Does it actually work? The published numbers

Two cookbooks ship measured results, and they are the most useful thing in the documentation for anyone deciding whether to build on this.

The OCR check runs a three-request pipeline over two photographed supplier invoices. Three questions per field in the first request (a Box to locate it, a Noul to check the extracted value fits), a zoomed Box pass, then a Noul per crop. The published results, from jev-1.14.0 on 2026-10-15, with a review threshold of 0.25 on either check:

InvoiceFieldExtractedPrintedPage checkCrop check
Milkwoodinvoice numberAN400074IN4000740.880.18
Siyakhulainvoice numberINV03070INV030790.020.04
Siyakhuladate16/09/202816/09/20260.170.66

Read that table carefully, because the interesting finding is not that Jev caught the errors. It is that neither check catches all three alone. The page check passes the misread AN400074 at 0.88 while the crop check catches it at 0.18. The crop check passes the wrong year at 0.66 while the page check catches it at 0.17. Both catch INV03070. Every correct value on both invoices passed both checks, with the lowest correct scores being 0.35 (Milkwood account number, page check) and 0.31 (Siyakhula supplier name, crop check).

That is a genuinely useful shape of result: it says a two-angle check beats either angle, and it also says the correct-but-low scores at 0.31 and 0.35 sit close enough to the 0.25 bar that a tighter threshold would start flagging good data. Tune on your images, as the docs say.

The feeder-camera cookbook sends one frame per second with five questions per frame — a Noul for presence, a Choice for none/one/several, a Box for the most prominent bird, and two Choices for the leftmost and rightmost species. On its 101-second clip, Jev agreed with the hand labels on bird presence in all 101 frames and named the correct species in 132 of 133 labelled sightings.

The third image cookbook, on reading a scanned vaccination record, is the cleanest demonstration of the Choice-with-unreadable trick: rather than let a Box or a read fail, the Choice includes unreadable as an option, so a date hidden under a batch sticker comes back as a first-class answer instead of a guess.

Which channel, and what the catalog says

The model’s own capability declaration is a useful cross-check against the documentation. Read live on 2026-11-11, GET https://openrouter.ai/api/v1/models/typesafe/jev-1.14/endpoints reported:

Three things follow. The catalog independently confirms image input and nothing else — no audio, no video. It confirms that the output modality is decisions rather than text, which is why an image request through an OpenAI-compatible client is not the same thing as a chat completion with an image attached. And it shows the 262,144 context figure is the long-context number, not the default: the models page lists 64k per request and 32k per question unless you send TypeSafe-Danger-Zone-Long-Context: true.

The pricing arithmetic is the same as for text. At $0.042 per million input tokens, the documented example’s 290 input tokens for one image plus a Box question is about $0.0000122 — though the documentation states it will publish guidance on estimating image token usage later, so do not build a cost model on that figure yet.

What to expect to break

The honest summary is in the docs’ own header: Dreamer-ready. Images and Box are not up to TypeSafe’s standards to call production-ready. The request format, the answers and the limits can change between releases, and accuracy will change as the models are updated. If your workflow cannot tolerate that, keep a text-only path.

The error surface is small and documented:

StatusCause
400Image or Box question present but no TypeSafe-Danger-Zone-Images: true header
400Image rejected: not PNG/JPEG, damaged, incomplete, animated, below minimum dimension, or an asset/format problem
400max_tokens_exceeded: images plus text exceed the request’s context length
413A size limit: file, dimension, total pixels, request body, all images, expanded request, or reference lengths

Note the last 400 row’s remedy, which is specific and easy to miss: when images push a request over context, send fewer or smaller images, or move images out of instructions into the state. The second is free if several questions share the picture — it is the same advice as the token note, arriving as an error message.

If a request has several problems, the error reports one of them, so a rejection is not a full diagnosis. Check the limits table and the preparation steps before assuming the model refused.

When not to use a Box

Three cases where reaching for Box is the wrong call:

And one case where reaching for images at all is the wrong call: a scanned document with a text layer. Text extraction is cheaper, deterministic and testable. The documentation reaches for images when the information is genuinely visual — a gesture, a badge photo, a bird on a feeder, a value that must be checked where it is printed.

Where this fits on the rest of the site

The primitives page on this site covers Choice, Score and Noul as the three question types; Box is the fourth, and it is the only one that needs 1.14. The request patterns guide covers the batching patterns — speculative fan-out, Noul screen, confidence routing — and Box composes with all of them: a Noul screen in front of a Box is the documented way to avoid the always-a-box problem, and fanning several Box questions into one request is how you locate several targets at once.

If you have not sent a request yet, start with the first API call guide and the state and questions guide; images add two fields to a request you already know how to build, not a new request shape.

Frequently asked questions

+ Can Jev read images?

Yes. Jev accepts PNG and JPEG images anywhere text can go in a request: as a value in state, inside a question's instructions, and inside a question's criteria. The official models page lists input as 'Text, images' for both Jev 1.14 and Jev 1.13, so image input is not a 1.14-only feature. Audio and video are not accepted natively.

+ Why does my request return 400 even though the image is valid?

Images are off by default. Every request that carries an image or a Box question must send the header TypeSafe-Danger-Zone-Images: true. Without it, the documentation's error table lists exactly this cause for a 400: the request has an image or a Box question but no images header.

+ Does a Box question work on Jev 1.13?

No. Box is the only capability in the 1.14 changelog that requires 1.14. The changelog entry for images and for the long-context header both add the sentence 'This also works on Jev 1.13'; the Box entry does not. The models page lists question types as Choice, Score, Noul, Box for 1.14 and Choice, Score, Noul for 1.13.

+ What does a Box answer look like?

Four edges and a confidence, all fractions. The documented example returns {top: 0.25, bottom: 0.82, left: 0.44, right: 0.84} with confidence 0.92 and usage input_tokens 290, output_tokens 21. Edges are fractions of the image's width (left, right) and height (top, bottom) measured from the top-left corner, so the same answer is valid at any image size. Multiply each edge by the pixel dimension to get a crop box.

+ What happens if the thing I am looking for is not in the image?

You still get a box. The documentation states there is no 'not found' answer, so a Box question about an absent target returns four edges anyway. The documented remedy is to ask a Noul question in the same request, such as 'Is there a coffee mug in photo?', and ignore the box when the Noul probability is low.

+ Is image support safe to build on in production?

The official documentation answers this directly: images and Box questions are labelled Dreamer-ready, defined as not yet up to TypeSafe's standards to call ready for production. The request format, the answers and the limits can change between releases, and accuracy will change as the models are updated. Treat both as prototyping features and keep a text-only path for anything that must not break.

+ How are images priced?

As input tokens, at the normal per-token input price, with no separate image charge. The documentation says the response's usage.input_tokens includes the tokens for every image in the request, and image tokens count toward the request's context length. Jev 1.14 lists at $42 per billion input tokens ($0.042 per million) with free output tokens, and OpenRouter's endpoint entry for typesafe/jev-1.14 reported a prompt price of 0.000000042 per token and 0 for completion when read on 2026-11-11.

Keep reading