Box and images: making Jev find things in pictures
In the first five lessons, the state held only text. But real decision needs often live in pictures: is there a person in the doorbell photo, where is the signature on the scanned form, where is the bird standing in this frame from the bird feeder camera. Hand such an image to GPT and it spits out a paragraph of description—what you want are coordinates you can crop directly and feed straight into if.
This lesson covers Jev Box: give it an image and a sentence, and it returns the four edges of the smallest rectangle enclosing the target. The whole thing takes 10 minutes.
Let's be upfront about the caveats: Images and Box are experimental features. The official wording is dreamer-ready—they haven't yet reached the production standard the team is willing to guarantee, and both the request format and precision may change across versions. Great for building prototypes and validating ideas; for production, weigh the risks yourself.
What you'll be able to do after this lesson
- Use the Python SDK's
Imageto put an image into state and send your first Box request with the Danger Zone request header - Read the Box answer: what fraction of the image's width and height each of the
top/bottom/left/rightfour edges represents, and convert back to pixel coordinates to crop out the target - Use the
confidenceincluded with the Box answer to decide whether to use the box directly, expand it before cropping, or switch to a better image and ask again - Use a Noul question as a gatekeeper to solve the problem that "Box has no 'not found' answer"
- State the limits of image requests: format, dimensions, size, and what error codes you'll get when you exceed them
Your current predicament
- Ask GPT to find something in an image and it returns a piece of prose like "there is a mug slightly left of center in the frame"—you get the direction but no coordinates, so you can't write the cropping code
- Using a dedicated detection model means annotating data, training, and tuning—overkill when you just want to find a mug's location
- Multimodal models return coordinates in a different format every time, and you still have to write a parser to catch all the fancy "can't see it clearly" or "not found" replies
When to use this move
When you need:
- The location of a target in an image: a signature on a scanned document, a person in a doorbell photo, a specific product on a shelf photo
- A location that goes straight into code: crop out the target region for OCR, draw a bounding box, sort by the
topedge - To ask judgment and localization together: is the person holding a package or a bag, is the target in the image
And you don't want to:
- Train or deploy a dedicated vision model
- Parse free-form natural language coordinate descriptions from a model
- Build a whole vision infrastructure just for the prototype stage
🎒 Prerequisites
- Completed State: know what state is and how questions use backtick paths to point at its fields
- Got the first request working:
.venvhastypesafe-sdkinstalled andTYPESAFE_API_KEYis in your environment variables - Have an image with something to find (the official example looks for a mug in a kitchen photo; if you don't have one, any PNG/JPEG will do—check the size limits in Step 1 first)
- Know that Box requires Jev 1.14 (
jev-latestalready points to it, so you most likely don't need to change anything); to confirm, see models and pricing
Core idea
Box answers "where". Official positioning: Box is the fourth question type (Choice picks options, Score assigns grades, Noul settles yes/no), and the first location question type designed purely for "output as data". Give it an image and the sentence "the coffee mug in the image", and it returns the smallest rectangle enclosing the mug.
The answer is just four edges, expressed as ratios.
| Field | Meaning | Range |
|---|---|---|
left / right | Distance from the left edge of the image, as a fraction of the width | 0–1, left < right |
top / bottom | Distance from the top edge of the image, as a fraction of the height | 0–1, top < bottom |
The origin is at the top-left corner: top: 0 is the top edge of the image, and bottom: 1 is the bottom edge. Everything is in 0–1 proportions, which means the same answer holds at any image size—a 4000×3000 photo taken on a phone and a 400×300 thumbnail will have the same relative position of the box. To draw a box or crop, just multiply each side by the image's pixel width and height (step 3).
How images get into a request: references + assets. Each image is registered once in the assets at the top level of the request, yielding a reference string; write that reference anywhere a string can appear—state, instructions, or criteria—and the model receives the image there. With direct HTTP, use multipart files or base64; the Python SDK generates references and uploads automatically for you, so Image(路径) goes straight into state and you're done (step 2).
One switch: the Danger Zone request header. Images and Box are off by default. Every request with an image or a Box must include:
TypeSafe-Danger-Zone-Images: trueIn the SDK, this just means passing extra_headers (step 2). If you forget, the request fails outright with 400.
Box can be mixed with other questions. Same as in previous lessons: all questions are evaluated in parallel against the same state, and adding questions adds almost no response time.
One key trait: Box always returns a box. There is no "not found" answer—if there's no coffee mug in the image, it still gives you four sides. The way to guard this is to add a Noul question in the same request, "Is there a coffee mug in the image", and if Noul says no, ignore that box (step 5).
Follow Along
Step 1: Prepare an image that meets the limits and install Pillow
Why: Image request limits differ from text ones—size, format, and file size are all strictly enforced, so prepare the image according to the checklist first, and the request in step 2 won't hit 413 / 400.
Official limits (images.md):
| Item | Limit |
|---|---|
| Format | PNG or JPEG, static images (animated PNGs are rejected) |
| Pixels per side | 280–1568; any side below 280 is rejected |
| Total pixels | 78,400–2,458,624 (i.e., the area from 280×280 to 1568×1568) |
| Single file | ≤ 16 MiB |
| Request body | ≤ 32 MiB (including all images, JSON, and multipart framing) |
Three things to do when preparing an image (from the official "Preparing images" guidelines):
- Resize: compress the longest side to within 1568 pixels. A 1200-megapixel phone photo typically becomes 1568×1176
- Rotate upright: the API strips all metadata (EXIF etc.) and does not apply EXIF orientation—an image that looks portrait in a viewer may appear landscape to the model. Rotate the pixels themselves; don't rely on EXIF
- Convert format: convert GIF, WebP, or HEIC to PNG or JPEG first; a standard 8-bit baseline JPEG is the safest
To get an image ready, three lines with Pillow do it (rotate upright + resize + convert to standard JPEG):
from PIL import Image, ImageOps
with Image.open("photo.jpg") as original:
img = ImageOps.exif_transpose(original) # rotate the pixels upright
img.thumbnail((1568, 1568)) # longest side at most 1568, same aspect ratio
img.convert("RGB").save("photo-ready.jpg", "JPEG", quality=90)If Pillow isn't installed, install it (we'll also need it later for Image.from_pil and cropping):
.venv/bin/pip install pillowYou should see: photo-ready.jpg is generated successfully; open it in an image viewer—the orientation is correct and the sharpness hasn't degraded.
- My image is PNG/JPEG, 280–1568 pixels per side, and already rotated upright
Step 2: Send your first Jev Box request with the Python SDK's Image
Why: With direct HTTP you have to assemble multipart yourself, manage references, and convert to base64; the Python SDK (with Image and Box support since v0.7.5) wraps all of this—put an Image object into the content, and references are generated and files uploaded automatically.
Image supports four ways to construct it:
| Construction | Syntax | Scenario |
|---|---|---|
| File path | Image(Path("kitchen.jpg")) | Most common |
| Raw bytes | Image(data) | Bytes from a network stream or database |
| base64 text | Image.from_base64(data_base64) | Receiving someone else's upload |
| PIL object | Image.from_pil(pil_image) | Send directly after cropping/resizing, saving a write to disk; requires typesafe-sdk[pil] |
Create a new box_kitchen.py. Use your photo-ready.jpg and pick a target in the image (the official example is a coffee mug; swap in something actually present in your image):
from pathlib import Path
from typesafe_sdk import Box, Image, TypeSafeClient
with TypeSafeClient() as client:
response = client.system_one(
state={"photo": Image(Path("photo-ready.jpg"))},
questions={
"target": Box(instructions="the coffee mug in `photo`"),
},
extra_headers={"TypeSafe-Danger-Zone-Images": "true"},
)
print(response.answers["target"].box)
print(response.answers["target"].confidence)Run:
.venv/bin/python box_kitchen.pyYou should see (values depend on your image and target):
top=0.25 bottom=0.82 left=0.44 right=0.84
0.92Three details:
A Box question has only two fields:
type(fixed as"box") andinstructions(what to find, and in which image). There is nocriteria—options, levels, and yes/no definitions are all inapplicable hereinstructionsdoesn't have to be a question; a single phrase is enough:"the coffee mug in`photo`"—the backtick path indicates which image to search in (official wording: It doesn't have to be a question)Image(Path(...))goes into state, and the SDK automatically generates a UUID-like reference—multiple images and nested structures won't get crossedI got four numbers between 0–1 and a confidence, with no 400 error
⚠️ Forgetting the Danger Zone header results in an outright 400
As long as the request contains an image or a Box question, TypeSafe-Danger-Zone-Images: true must not be missing. The error is blunt: 400, with the reason stated plainly. For those using direct HTTP, the API quick reference has a complete curl multipart example.
💡 Box requires Jev 1.14
Jev 1.13 and 1.14 both accept images, but the Box question is only available starting from 1.14. jev-latest already points to jev-1.14.0, so just follow it normally; projects locked to 1.13 that want to use Box need to upgrade the model first.
Step 3: Understand the four edges of the Box answer and convert them back to pixel coordinates
Why: The 0–1 ratios can't be used directly to draw a box or crop—you have to multiply them back by your image's pixel dimensions. That's exactly what the official "Reading a Box" code does.
First look at the raw response (if you call the API directly with curl, you get this JSON right away; the SDK receives the same structure):
{
"model": "jev-1.14.0",
"answers": {
"mug": {
"type": "box",
"box": {
"top": 0.25,
"bottom": 0.82,
"left": 0.44,
"right": 0.84
},
"confidence": 0.92
}
},
"usage": {"input_tokens": 290, "output_tokens": 21}
}How to read it (assume your image is 1568×1176):
| Field | Value | Conversion | Pixel position |
|---|---|---|---|
left | 0.44 | 0.44 × 1568 | 690px from the left |
right | 0.84 | 0.84 × 1568 | 1317px from the left |
top | 0.25 | 0.25 × 1176 | 294px from the top |
bottom | 0.82 | 0.82 × 1176 | 964px from the top |
Official cropping code (Pillow's crop takes pixel coordinates, in the order left, top, right, bottom):
from PIL import Image
image = Image.open("photo-ready.jpg")
width, height = image.size
mug = response["answers"]["mug"]
edges = mug["box"]
box = (
round(edges["left"] * width),
round(edges["top"] * height),
round(edges["right"] * width),
round(edges["bottom"] * height),
)
image.crop(box).save("mug.jpg")SDK users can copy it as-is, just replace mug = response["answers"]["mug"] with mug = response.answers["mug"] and edges = mug["box"] with edges = mug.box—the answer field is the same structure.
You should see: When you open mug.jpg, it's exactly the target you asked about—the coffee mug is boxed in cleanly, with basically no extra background on any side.
- My crop result encloses the target, and the four edges weren't converted backwards (left/right multiply by width, top/bottom multiply by height)
Step 4: Use confidence to decide whether to crop or double-check
Why: Box answers come with confidence (0–1, the higher the more certain). Same idea as the last lesson: use high-confidence answers directly, give low-confidence ones extra care—if one box is cropped crooked, the whole downstream pipeline is processing wrong data.
The official position: there is no recommended threshold, tune it on your own images. When it falls below your threshold, the official docs offer two paths:
| Approach | How to do it | When to use it |
|---|---|---|
| Re-ask with a better image | Send a larger version, or crop the region around the box, enlarge it, and re-ask the same Box question | The original image has low resolution or the target is too small |
| Expand the edges before cropping | Move the four edges outward a bit before crop, staying within the image bounds | When the crop must never cut off any part of the target |
The official padding code (PAD is the fraction to expand outward on each side; copy it and adjust for your image):
PAD = 0.05 # Fraction of the image to add on each side. Tune for your images.
MIN_CONFIDENCE = 0.6 # Example threshold. Tune it on your own images.
if mug["confidence"] < MIN_CONFIDENCE:
edges = {
"left": max(0.0, edges["left"] - PAD),
"top": max(0.0, edges["top"] - PAD),
"right": min(1.0, edges["right"] + PAD),
"bottom": min(1.0, edges["bottom"] + PAD),
}Then use this new set of edges to crop the way you did in step 3—the target goes into mug.jpg together with a ring of margin; better to include a bit of extra background than to slice off half a mug handle.
You should see: High-confidence answers are used as-is; answers below the threshold either come back with an enlarged box or go down the "enlarge and re-ask" path.
- I set a confidence threshold for my Box answers, with a clear handling path below it (expand the edges or re-ask)
Step 5: Use Noul as a gatekeeper and batch mixed questions
Why: Personality problem number 1—Box always returns a box, there's no "not found." If the target isn't in the image, it still returns four edges. The gatekeeping solution is already written up officially: add a Noul question to the same request.
The official vaccination record example is configured exactly this way (Noul asks "is it there," Box asks "where is it"):
questions = {
"mug_visible": Noul(instructions="Is there a coffee mug in `photo`?"),
"mug": Box(instructions="the coffee mug in `photo`"),
}Usage in one sentence: if Noul says no, ignore the box from mug. The two questions are evaluated in parallel, in one request, with each answer attached to its own ID.
Three other high-frequency pairing techniques, all from the official "Writing Box questions" guidelines:
| Scenario | Approach |
|---|---|
| There may be more than one target in the image | Narrow it down to a unique one in words: "the mug nearest the sink in `photo` " |
| Words can't pin down how many there are | Add a Choice alongside, with options none / one / several; if the answer is several, slice the image into pieces and ask about each piece separately |
| Finding a small target in a large image | First use Choice to pick which of the image's tiles contains the target, then ask Box about that tile |
| Using an image as the target | Write the target directly into the instructions: "the person in `photo` who matches this badge photo: 9a21c7d3-badge" |
The last one is the flip side of image references: criteria and instructions can also contain images. The official docs' lost-dog example uses exactly this mechanism: the photo of the found dog goes into state, the photo from the lost-dog flyer is written directly into the question, asking "is it the same dog."
You should see: Noul and Box (and even additional Choice / Score) coexist in one request, each answering its own part; when mug_visible is low, your code simply skips the box from mug.
- My Box request includes a gatekeeper question (Noul or Choice), and "not found" has a clear code path out
Checkpoint ✅
- I know Images / Box is an experimental feature (dreamer-ready): the format may change, accuracy may change, and the official docs advise against using it directly in production
- I can explain the four edges:
left/rightare ratios of the width,top/bottomare ratios of the height, with the origin at the top-left corner - I got a request with
TypeSafe-Danger-Zone-Images: trueworking end-to-end, and can crop out the target by converting to pixels - My Box answers have a confidence threshold; below it, I either expand the edges or re-ask with a different image
- My "not found" scenario has a Noul / Choice gatekeeper, so an empty box is never taken at face value
Pitfall Reminders
❌ The image is fine, but the request returns 400 / 413
400: Images are not turned on
413: The request is over a size limitCheck three things:
- Did any request involving an image or a Box question include the
TypeSafe-Danger-Zone-Images: trueheader? (400) - Did the image dimensions go out of bounds? Each side 280–1568, total pixels 78,400–2,458,624, single file ≤ 16 MiB—over the limit means outright rejection, no automatic scaling (
413or400) - Did a batch of images together exceed 32 MiB? The API re-encodes during processing, and the processed images may end up larger than the files you sent—even sending a bunch of small images can hit the line (
413)
The official docs also note: when a request has multiple problems, the error only reports one of them—go through the limits table item by item.
❌ Using a portrait-shot photo as a portrait image
It looks perfectly upright in your image viewer, but to the model it's horizontal—the API strips all metadata (including EXIF orientation) and does not apply it. When you ask about "the cup at the bottom of the image," the model sees a cup in a horizontal image.
The right approach: before sending, use ImageOps.exif_transpose to rotate the actual pixels upright (the three Pillow lines in step 1). Don't just fix the metadata.
❌ The cropped box doesn't match the target
First check whether the conversion got flipped: top/bottom multiply by height, left/right multiply by width; then check that Pillow's crop argument order is (left, top, right, bottom). If either is flipped, the box flies off the image or skews somewhere else.
Still wrong? Check confidence—low-confidence boxes can naturally be off. If it's below your threshold, pad the edges first (the PAD in step 4), or crop the region, scale it up, and ask the same Box question again.
❌ Asking one Box question for "the locations of all coffee mugs"
Box returns only one box per question—that's by design. For multiple locations of multiple targets: one question per target (sent in the same request); when same-kind targets are indistinguishable, add a Choice (none / one / several), and if the answer is several, split into pieces and ask separately. Don't cram "find all X" into a single instructions.
❌ The target isn't in the image at all, yet a box still comes back
Box has no "not found." If there's no mug in the image, the answer is still four edges—your cropping code will happily crop out a patch of background.
The right approach is exactly step 5: add a Noul alongside it ("Is there a coffee mug in `photo` ?"); if Noul's probability is low, ignore that box. The official vaccine-record, invoice-checking, and bird-feeder-camera cookbooks all use this pattern.
Advanced Tips (Optional)
Pick one of the three reference-attachment styles based on your scenario. Beyond auto-referencing, the SDK offers two finer-grained controls:
assetsparameter—name the reference yourself, using delimiters unlikely to collide (the official example is<product>):state="Inspect this returned product: <product>"paired withassets={"<product>": Image(photo)}client.attach_assets—attach the image to the client, embedstr(asset)into the body:state=f"The customer says {photo} arrived scratched.". You must use thewith client.attach_assets(photo):context manager; withoutwiththe SDK cannot map the references in the text back to the image data (quoting the official warning)
Send PIL objects directly and skip one write to disk. After cropping or scaling, no need to save and read back—send Image.from_pil(pil_image) directly (JPEG encoded with quality 95; pass format="png" for images with transparency or a palette). Installation: .venv/bin/pip install "typesafe-sdk[pil]". The official OCR cookbook takes this route: find fields on the full page, crop and scale up the region, then ask detailed questions about the crop.
Sort scanned documents by the top edge when reading. The vaccine-record cookbook's trick: one Box per line, with smaller box["box"]["top"] values first—pure code can turn jumbled lines into ordered records. Full case study in Image Walkthrough.
Images have no separate pricing. Each image is billed as input tokens, at the same rate as text; usage.input_tokens includes the tokens of all images. Images also count toward the context length (64k per-request / 32k per-question)—stuffing in too many large images triggers max_tokens_exceeded, see Models & Pricing for details.
The SDK's answer collection. Box can also be fetched from response.boxes["target"]—a Box-specific entry point into the same set of answers as response.answers["target"].
Lesson Recap
✅ Jev Box usage: put Image into state, use Box(instructions=...) with one sentence describing what to find, and turn on the Danger Zone header to start asking.
✅ Reading the answer: all four edges are 0–1 ratios (left/right multiply by width, top/bottom multiply by height), so the same answer holds at any image size.
✅ confidence: built into Box, with no officially recommended threshold—if it falls below the line you set, pad and re-crop, or re-ask with a better image.
✅ Gatekeeping: Box always returns a box; Noul asks "is it there," and a low answer means ignore the box.
✅ Experimental feature: dreamer-ready, format and precision may change across versions—best suited for the prototyping stage.
Key configuration fields recap:
| Field | Recommended usage | Notes |
|---|---|---|
type | "box" | Fixed value |
instructions | One sentence: what to find, in which image | Doesn't have to be a question; no criteria needed |
| Request header | extra_headers={"TypeSafe-Danger-Zone-Images": "true"} | Required for every request with images/Box |
Image(...) | path / bytes / from_base64 / from_pil | from_pil requires typesafe-sdk[pil] |
| Image dimensions | 280–1568 pixels per side, longest side ≤ 1568 | Rotate upright first (EXIF is not applied); over the limit means outright rejection |
| Answer access | response.answers["id"].box / response.boxes["id"] | .box is the four edges, .confidence 0–1 |
To see the full use of Box in real projects: Image Walkthrough: Reading Scanned Documents, Verifying OCR Results—line-by-line locating in vaccine records, field-by-field OCR verification for invoices, and species logging for a bird-feeder camera, three cookbooks end to end.
Frequently Asked Questions
What is Jev software?
Jev is a judgment model provided by TypeSafe, accessed via API calls, not a local software you need to install. The version used on this page is jev-1.14.0 (jev-latest already points to it); you can send requests with the Python SDK typesafe-sdk — see the code in step 2.
Jev How do I deploy it?
Jev does not require you to deploy it yourself: it runs on the server side, so as long as you install typesafe-sdk in .venv and set TYPESAFE_API_KEY as an environment variable, you can send requests. This is one of the reasons this page recommends it — you can do image grounding without training or deploying a dedicated vision model.
Jev Where can I find the official documentation?
The official limits and guidelines for image requests are in images.md, which this page cites; for model versions and billing details, see Models and Pricing, and for a curl multipart example of direct HTTP access, see API Quick Reference.
Jev What does it have to do with vaccine records?
Vaccine records are an example scenario in the official cookbook: use Box to locate each line of a scanned vaccine record, then sort the out-of-order lines into an ordered record by box["box"]["top"]. For the full case, see Image in Practice.
Next Lesson Preview
That covers everything about single-question usage: you've now gone through all five asking modes (Choice / Score / Noul / confidence / Box). Next, let's combine them into a system. Next lesson: Efficient Workflows: Four Architecture Patterns for Working with Coding Agents.
You will learn:
- The parallel pattern for asking a batch of questions in a single request: official tests show 12.2x lower cost and 10x faster speed
- Confidence-gated routing: answers tell you "what it is," while confidence tells you "whether to act"
- Composite scoring that breaks complex judgments into atomic Score items and then merges them with code-weighted aggregation
- Let Claude Code install the TypeSafe Agent Skill so it writes the code that calls Jev for you directly