Jev Image Inputs and Box Questions: Sending Images and Reading Bounding Boxes
Every guide on this site up to now has treated Jev as a text engine: you put text into state, you ask Choice, Score and Noul questions, you get typed answers back. That description was accurate for the primitives, and it is what nearly every tutorial repeats. It is also incomplete.
Jev reads images. It reads them on 1.13 as well as 1.14, it reads them in three different places inside a request, and since 1.14 it can answer a fourth question type — Box — that returns where something is inside the picture. This page is the missing chapter: what you actually send, what comes back, what the hard limits are, and what the published measurements say about whether any of it works yet.
Nothing here is inferred. Every figure comes from TypeSafe’s own documentation, read on 2026-11-11, plus one live read of the OpenRouter catalog for the model’s declared modalities. Where the docs and the catalog describe the same thing differently, both are quoted.
The one-line version
An image request is a normal System One request with two additions: a top-level assets map holding the images, and reference strings written where the image belongs. Add the header TypeSafe-Danger-Zone-Images: true, or the request fails with 400. A Box question takes only type and instructions, returns {top, bottom, left, right} as fractions plus a confidence, and always returns a box — including when the target is not there, which is why the documentation tells you to pair it with a Noul.
Jev reads images on 1.13 too
This is the part people get wrong first, because the 1.14 release notes announce images and Box in the same breath. The official changelog separates them:
- New Box question type, which finds where something is in an image and answers with a bounding box. Box questions need Jev 1.14.
- Images can go in the
state,instructions, orcriteriaof a request. This also works on Jev 1.13.
The images page repeats it in one sentence: “Jev 1.13 and Jev 1.14 both accept images. Box questions need Jev 1.14.” The models table agrees — input is “Text, images” on both rows, while question types are Choice / Score / Noul / Box on 1.14 and Choice / Score / Noul on 1.13.
The practical consequence: if you are pinned to 1.13 and only need image input, you do not need to change versions at all. You need the header, and you need to be aware that you have no way to ask for a location.
There is a third gate on top of the version gate: images and Box are off by default and require the header on every request that carries either. The error table gives the exact 400 for forgetting it — “Images are not turned on: the request has an image or a Box question but no TypeSafe-Danger-Zone-Images: true header.”
The assets map and the reference rules
You never inline an image into a prompt string as a data URL. Instead each image is listed once in a top-level assets object, keyed by a reference string you choose, and you write that same string wherever the image belongs:
{
"model": "jev-latest",
"state": { "photo": "a3f1c2e8-found" },
"assets": {
"a3f1c2e8-found": { "type": "image", "format": "jpg", "data_base64": "/9j/4AAQSkZJRg..." },
"77b0d4a1-poster": { "type": "image", "format": "jpg", "data_base64": "/9j/4AAQSkZJRg..." }
},
"questions": {
"same_dog": {
"type": "noul",
"instructions": "Is the dog in `photo` the same dog as in this picture: 77b0d4a1-poster?"
}
}
}
That example is the documentation’s own three-way anatomy, trimmed to two images. The reference a3f1c2e8-found stands in as the whole value of state.photo; 77b0d4a1-poster is embedded mid-sentence inside a Noul’s instructions. The third position — a Choice option’s description — appears further down.
An asset entry has four possible fields. type is optional and must be "image" if present. part is the multipart file-part name. data_base64 is the file, standard base64 alphabet with = padding, no whitespace, no data: prefix. format is "png", "jpg" or "jpeg", lowercase, required with data_base64 and optional with part. If an asset somehow carries both part and data_base64, the API uses part.
The reference rules are where requests break, and they are all stated explicitly:
- References are matched as plain text anywhere in the request’s strings. This is a substitution, not a schema link. A reference named
catwould also match inside the wordcategory. The documentation’s own advice: use a fresh UUID per image per request. - Every asset must be referenced. An asset that appears nowhere in
state,instructionsorcriteriareturns 400. - Whole values match first. A string that is exactly a reference becomes that image. Inside longer strings, references are replaced left to right, and matches never overlap.
- Each use adds the image again. You upload the file once, but every place the reference appears adds another copy to what the model reads — and every copy counts against the expanded-request limit.
- Object keys are not replaced. A Choice option name, a question ID, any object key, stays text. To define an option with a picture you put the reference in the option’s description.
- Answers contain the reference, not the image, wherever they echo your criteria back — a Score’s
legend, for example. - References are capped at 1,024 bytes of UTF-8 each and 65,536 bytes total, measured in bytes, not characters.
That fourth rule is the one that costs money if you miss it. An image that is 10 MiB after processing, referenced four times, is 40 MiB of expanded request and is over the 32 MiB limit before you send it.
The three places an image can go
In state
An image in state is part of the state, exactly like a string or a JSON field. Every question in the request is evaluated against it, so this is the cheap place to put an image that several questions share. A state can mix images with text and hold several images, labelled or nested — the documentation’s example is a short history of camera frames:
{
"state": {
"frames": [
{ "seconds_ago": 2, "frame": "5c0e9a12-f1" },
{ "seconds_ago": 1, "frame": "1d7be0c4-f2" },
{ "seconds_ago": 0, "frame": "e48a6f30-f3" }
]
}
}
A question can then ask about the array as a whole (“Did the door in frames open during these frames?”) or address one element by path, such as `frames[2].frame`.
In instructions
An image inside a question’s instructions belongs to that question and is sent only with it. The documented use is comparison — one photo in the state, one reference photo in the question:
Is the person in
doorthe same person as in this badge photo: 9a21c7d3-badge?
The cost note is explicit: an image in instructions is sent once for every question that includes it. When several questions need the same picture, put it in state once and refer to it by path — fewer tokens, less of the size budget.
In criteria
Choice options, Score levels, and a Noul’s true and false definitions can each carry images. This is the “define the options by example” pattern, and it is the most interesting use of images in Jev, because the model is not asked to describe the picture — it is asked which of your example pictures the input matches.
{
"model": "jev-latest",
"state": { "camera": "e48a6f30-cam" },
"questions": {
"gesture": {
"type": "choice",
"instructions": "Which gesture is the hand in `camera` making?",
"criteria": {
"thumbs_up": "0b6c51aa-thumb",
"open_palm": {
"description": "A hand held up with the palm facing the camera.",
"examples": ["4fd2e7b9-palm1", "c83a0e15-palm2"]
},
"fist": "a7e94d02-fist",
"none": "No hand, or a hand that matches none of the examples."
}
}
}
}
An option can be an image on its own, a description with one or more example pictures, or plain text. Both description and examples are the documentation’s own naming choices in that example, not required fields — criteria accept any structure. A Score can use one example picture per level, and a Noul can define true and false with pictures: {"true": {"description": "Correctly assembled", "examples": ["<reference>"]}, "false": "Parts missing or in the wrong place."}.
The documentation summarises the whole design space as six patterns, and it is worth knowing which one you are in before you write the request:
| Pattern | What it does | Where the image goes |
|---|---|---|
| Compare against a known picture | Same person, product or place as a reference? | Known picture in instructions, photo in state |
| Define options by example | Each Choice option gets one or a few example pictures | criteria |
| What changed | Send the previous frame or a short history, ask about motion or arrivals | state |
| Look closer when needed | Ask on a small copy first, send a bigger one or a full-resolution crop only when the answer needs detail | state |
| Search a large image | Cut into tiles, ask a Choice with the tiles as options, repeat on the chosen tile | criteria |
| Find where it is | Ask a Box for the edges of the target | state, optionally a target picture in instructions |
Box: the fourth question type
A Box question has exactly two fields. type is always "box". instructions is what to find and in which image — and it does not have to be a question at all; the documentation’s example is the fragment “the coffee mug in photo”. Box takes no criteria.
curl https://api.typesafe.ai/v1/systemone \
-H "Authorization: Bearer ***" \
-H "TypeSafe-Danger-Zone-Images: true" \
-F 'request={
"model": "jev-latest",
"state": {"photo": "6e1d0b7c-kitchen"},
"assets": {"6e1d0b7c-kitchen": {"type": "image", "part": "kitchen-photo"}},
"questions": {
"mug": {"type": "box", "instructions": "the coffee mug in `photo`"}
}
};type=application/json' \
-F 'kitchen-photo=@kitchen.jpg;type=image/jpeg'
The answer carries a box object and a confidence:
{
"model": "jev-1.14.0",
"answers": {
"mug": {
"type": "box",
"box": { "top": 0.25, "bottom": 0.82, "left": 0.44, "right": 0.84 },
"confidence": 0.92
}
},
"usage": { "input_tokens": 290, "output_tokens": 21 }
}
left is always less than right and top always less than bottom. All four are fractions from 0 to 1 measured from the top-left corner — left and right against the image width, top and bottom against the height. That normalisation is the reason one answer works at any resolution: multiply each edge by the pixel dimension and you have a crop box.
from PIL import Image
image = Image.open("kitchen.jpg")
width, height = image.size
mug = response["answers"]["mug"]
edges = mug["box"]
box = (
round(edges["left"] * width),
round(edges["top"] * height),
round(edges["right"] * width),
round(edges["bottom"] * height),
)
image.crop(box).save("mug.jpg")
The always-returns-a-box problem
A Box answer is always a box. There is no “not found”. Ask where the mug is in a photo with no mug in it and you get four edges anyway, with some confidence attached.
The documented remedy is to ask a Noul in the same request and use it as a gate:
"questions": {
"mug_visible": { "type": "noul", "instructions": "Is there a coffee mug in `photo`?" },
"mug": { "type": "box", "instructions": "the coffee mug in `photo`" }
}
Then ignore the box when mug_visible is low. Both questions are evaluated against the same state in one request, so the gate costs one question, not one round trip.
Three more rules from the Box page are worth internalising. One box per question — Box is designed to find a single target, so several different targets means several Box questions. Describe the one you want when the image might contain several of the same thing: “the mug nearest the sink in photo”. Crop before looking for small things — a target a few dozen pixels wide in a very large image can be missed, so find the region first, for example with a Choice whose options are tiles, then Box on that tile and convert the edges back to fractions of the full image.
There is also no recommended confidence threshold. The documentation says to tune it on your own images, and offers two remedies when confidence is low: ask again on a better image (a larger version, or a higher-resolution crop around the box), and pad the box before cropping so a crop cannot cut off part of the target.
Hard limits
These are rejection thresholds, not resizing hints. An image outside them is rejected; the API does not shrink it for you.
| Limit | Value |
|---|---|
| Format | PNG or JPEG, still images only. Animated PNGs are rejected. |
| File size | 16 MiB per image file (after base64 decoding) |
| Width and height | 280 to 1568 pixels each; either side under 280 is rejected |
| Total pixels | 78,400 to 2,458,624 (a 280×280 to a 1568×1568 square) |
| Request body | 32 MiB, including JSON, base64 text and multipart framing; headers and compression do not raise it |
| All images in the request | 32 MiB, shared by every distinct image after processing |
| Expanded request | 32 MiB after every reference is replaced by its image |
| Reference strings | 1,024 bytes each, 65,536 bytes total |
All limits include both ends of the range. If any image in a request is rejected, the whole request fails — there are no partial answers. How many images fit in one request depends on these limits and the context length, not on a separate image quota: images have no separate rate limit and count against the same request limits as text.
The 32 MiB “after processing” row deserves attention. Processing re-encodes each image and can make it larger than the file you sent, so a request of modest uploads can still exceed the budget.
Preparing images
Six instructions from the documentation, all of which prevent specific rejections:
- Resize before sending. Longest side at most 1568 pixels. A typical 12-megapixel phone photo becomes 1568 × 1176. Scale up, or pad, anything under the minimum.
- Rotate the pixels upright. The API strips all metadata — EXIF, colour profiles, comments — and does not apply EXIF orientation. A photo that looks upright in a viewer can reach the model sideways.
- Send a still image. For video, export the frame you want. Convert GIF, WebP and HEIC to PNG or JPEG.
- Prefer standard JPEG. An 8-bit baseline JPEG is the safest JPEG to send; re-save progressive JPEGs as baseline.
- Re-export a rejected file. If a file opens fine locally but is rejected, export it again from a standard editor or library.
- Multipart over base64 when you call HTTP directly. Base64 inflates each image by roughly a third, and that inflation counts against the 32 MiB body limit.
The documented Pillow snippet does the first two together:
from PIL import Image, ImageOps
with Image.open("photo.jpg") as original:
img = ImageOps.exif_transpose(original) # rotate the pixels upright
img.thumbnail((1568, 1568)) # longest side at most 1568
# thumbnail only shrinks: upscale or pad an image under the minimum separately
img.convert("RGB").save("photo-ready.jpg", "JPEG", quality=90) # baseline JPEG
Under multipart, the JSON body goes as the first part, named request, with content type application/json; each image is its own file part, and the asset’s part field must name it. Part names must be unique and non-empty, request is reserved, each image part’s content type must be image/png, image/jpeg, image/jpg, application/octet-stream or absent, and a part that no asset names — or an asset whose part is missing — returns 400.
Does it actually work? The published numbers
Two cookbooks ship measured results, and they are the most useful thing in the documentation for anyone deciding whether to build on this.
The OCR check runs a three-request pipeline over two photographed supplier invoices. Three questions per field in the first request (a Box to locate it, a Noul to check the extracted value fits), a zoomed Box pass, then a Noul per crop. The published results, from jev-1.14.0 on 2026-10-15, with a review threshold of 0.25 on either check:
| Invoice | Field | Extracted | Printed | Page check | Crop check |
|---|---|---|---|---|---|
| Milkwood | invoice number | AN400074 | IN400074 | 0.88 | 0.18 |
| Siyakhula | invoice number | INV03070 | INV03079 | 0.02 | 0.04 |
| Siyakhula | date | 16/09/2028 | 16/09/2026 | 0.17 | 0.66 |
Read that table carefully, because the interesting finding is not that Jev caught the errors. It is that neither check catches all three alone. The page check passes the misread AN400074 at 0.88 while the crop check catches it at 0.18. The crop check passes the wrong year at 0.66 while the page check catches it at 0.17. Both catch INV03070. Every correct value on both invoices passed both checks, with the lowest correct scores being 0.35 (Milkwood account number, page check) and 0.31 (Siyakhula supplier name, crop check).
That is a genuinely useful shape of result: it says a two-angle check beats either angle, and it also says the correct-but-low scores at 0.31 and 0.35 sit close enough to the 0.25 bar that a tighter threshold would start flagging good data. Tune on your images, as the docs say.
The feeder-camera cookbook sends one frame per second with five questions per frame — a Noul for presence, a Choice for none/one/several, a Box for the most prominent bird, and two Choices for the leftmost and rightmost species. On its 101-second clip, Jev agreed with the hand labels on bird presence in all 101 frames and named the correct species in 132 of 133 labelled sightings.
The third image cookbook, on reading a scanned vaccination record, is the cleanest demonstration of the Choice-with-unreadable trick: rather than let a Box or a read fail, the Choice includes unreadable as an option, so a date hidden under a batch sticker comes back as a first-class answer instead of a guess.
Which channel, and what the catalog says
The model’s own capability declaration is a useful cross-check against the documentation. Read live on 2026-11-11, GET https://openrouter.ai/api/v1/models/typesafe/jev-1.14/endpoints reported:
architecture.modality:text+image->decisionsinput_modalities:["text", "image"]output_modalities:["decisions"]- endpoint context length: 262,144, with prompt pricing
0.000000042per token and0for completion
Three things follow. The catalog independently confirms image input and nothing else — no audio, no video. It confirms that the output modality is decisions rather than text, which is why an image request through an OpenAI-compatible client is not the same thing as a chat completion with an image attached. And it shows the 262,144 context figure is the long-context number, not the default: the models page lists 64k per request and 32k per question unless you send TypeSafe-Danger-Zone-Long-Context: true.
The pricing arithmetic is the same as for text. At $0.042 per million input tokens, the documented example’s 290 input tokens for one image plus a Box question is about $0.0000122 — though the documentation states it will publish guidance on estimating image token usage later, so do not build a cost model on that figure yet.
What to expect to break
The honest summary is in the docs’ own header: Dreamer-ready. Images and Box are not up to TypeSafe’s standards to call production-ready. The request format, the answers and the limits can change between releases, and accuracy will change as the models are updated. If your workflow cannot tolerate that, keep a text-only path.
The error surface is small and documented:
| Status | Cause |
|---|---|
400 | Image or Box question present but no TypeSafe-Danger-Zone-Images: true header |
400 | Image rejected: not PNG/JPEG, damaged, incomplete, animated, below minimum dimension, or an asset/format problem |
400 | max_tokens_exceeded: images plus text exceed the request’s context length |
413 | A size limit: file, dimension, total pixels, request body, all images, expanded request, or reference lengths |
Note the last 400 row’s remedy, which is specific and easy to miss: when images push a request over context, send fewer or smaller images, or move images out of instructions into the state. The second is free if several questions share the picture — it is the same advice as the token note, arriving as an error message.
If a request has several problems, the error reports one of them, so a rejection is not a full diagnosis. Check the limits table and the preparation steps before assuming the model refused.
When not to use a Box
Three cases where reaching for Box is the wrong call:
- You need to read text, not locate it. Box tells you where. If you want the value, the OCR cookbook’s shape is the answer: locate with Box, then check the candidate value with a Noul, then normalise in code. Do not expect the box itself to be the extraction.
- You need to count things. One box per question, and the docs point you at a Choice of
none/one/severalplus region splitting when the count matters more than the location. - You need a guaranteed answer. With no “not found” reply and no recommended threshold, an unguarded Box is a silent failure mode. Always pair it with a Noul.
And one case where reaching for images at all is the wrong call: a scanned document with a text layer. Text extraction is cheaper, deterministic and testable. The documentation reaches for images when the information is genuinely visual — a gesture, a badge photo, a bird on a feeder, a value that must be checked where it is printed.
Where this fits on the rest of the site
The primitives page on this site covers Choice, Score and Noul as the three question types; Box is the fourth, and it is the only one that needs 1.14. The request patterns guide covers the batching patterns — speculative fan-out, Noul screen, confidence routing — and Box composes with all of them: a Noul screen in front of a Box is the documented way to avoid the always-a-box problem, and fanning several Box questions into one request is how you locate several targets at once.
If you have not sent a request yet, start with the first API call guide and the state and questions guide; images add two fields to a request you already know how to build, not a new request shape.