Image Practice: Reading Scanned Documents and Verifying OCR Results
💡 One-sentence summary: use Jev image recognition to build two paper-document pipelines——for scanned documents, "Noul finds whether it's there, Box finds where, Choice reads the value"; for OCR results, "verify the full page once, take a close look at the crop once," routing low scores to humans.
In the prerequisite course Box and Images, you learned to send image-bearing requests and read the four edges. This lesson assembles it into a complete project. The official docs provide three image cookbooks: Reading a scanned vaccination record (reading vaccination record scans), OCR confidence and bounding boxes (verifying invoice OCR values), and Feeder-cam visit log (a bird-feeder camera species log). This lesson walks through the first two; the third is covered together in the advanced techniques section.
The official numbers first, to see how this method performs in practice: 16 candidate vaccines all judged correctly in one request——the 6 on the record with probability 0.97 or above, while the easily confused "hepatitis A and B combined vaccine" got only 0.02; for the date covered by a sticker, the model answered "unreadable" instead of guessing; all three misreads in the two invoices' OCR (AN400074, INV03070, 16/09/2028) were all caught by the two checks. This lesson takes about 20 minutes; the official examples include a json_cache.json answer cache——per the official notes, when the cache is placed in the run directory, the script replays the cache instead of making real requests, so you can run it without a Key and get the same numbers as this page.
What You'll Be Able to Do After This Lesson
- Write a scanned-document reading pipeline: Noul + Box to locate every row you need in a table, Choice to read the value, and get
unreadableinstead of a blindly guessed year when it can't be read - Split "reading the image" and "judging based on the reading" into two requests——official testing showed: the version merging them into one question let both problems slip past the review line
- Verify OCR-extracted values: check each field against the full page once, then take a close look at the cropped image again, route anything below 0.25 to a human, and draw review boxes on the original image
- Use the same set of questions on consecutive video frames: the bird-feeder camera's 101 frames judged perfectly on whether a bird was present (101/101), species recognized correctly 132/133
Why Use Jev Image Recognition Instead of Letting a Multimodal LLM Look at Images Directly?
"Send the scan to GPT and have it read it for me" is many people's first instinct——it works, but what you get is a paragraph of prose, not a judgment your code can consume.
| Capability | Multimodal LLM Looking Directly | Jev Image Recognition |
|---|---|---|
| Unreadable content | ❌ Tends to still give an answer——the date under a sticker gets "read" anyway | ✅ An unreadable option in Choice; in the official example the model actually chose it, with the adjacent readability Noul at only 0.05 |
| Location information | ❌ Prose descriptions like "a row toward the top of the table" | ✅ Box returns four edge ratios, letting you crop out that row directly for the reviewer |
| Confidence | ❌ Ask "are you sure" and the answer changes each time | ✅ Every question carries a probability: vaccines on the page 0.97+, those not on it 0.02, with thresholds written in your code |
| Judgment vs. reading | ❌ One prompt does everything; when it's wrong, you can't tell whether it misread or misjudged | ✅ One request to read the image, another to judge from the reading——each step's errors are visible on their own |
| Answer format | ❌ Free text; downstream needs a parser | ✅ Typed answers + probability distributions, used directly in an if |
In one sentence: an LLM "describes" the image for you, while Jev "judges" it for you——only judgments come with probabilities, and only probabilities let you set review lines.
Your Current Pain Points
- "Mountains of paper scans to enter into the system; OCR is done, but nobody knows which fields were misread——verifying them one by one takes longer than typing them by hand"
- "Have a multimodal model read a scan, and it dares to give me a year even for a date covered by a stain, with no warning at all when it's wrong"
- "Want the model to tell me where a target is in the image, and it replies with prose like 'toward the left of the frame'——impossible to write cropping code for"
When to Use This Technique
When you need to:
- Read structured information from scans or photographed documents: table rows, dates, numbers, amounts
- OCR has already extracted the values, but you need something to vet each field: is the value right, is it on the page, was it misread
- Get judgment and location together: what this row says and where it is in the image (handy for cropping it out for human review)
And you don't want to:
- Train or deploy a dedicated vision model
- Parse free-form natural-language replies from a multimodal model
- Guess which step went wrong in answers where "reading" and "judging" are tangled together
🎒 Preparation Before Starting
- Completed Box and Images: can send images with the Python SDK's
Image, knows theTypeSafe-Danger-Zone-Images: truerequest header, can read the four-edge ratios - Recommended reading: Noul Yes/No Questions and Choice Questions: every step of this lesson is a combination of these two question types plus Box
- Have run through Installation and the First Request:
.venvis usable. The first half of this lesson runs without a Key (official cache replay); setTYPESAFE_API_KEYif you want to send real requests - A cup of water ☕ — this lesson is the finale of the C track: three official cookbooks, reading scanned documents, verifying invoices, watching birds, all using the same "gatekeeping + locating + reading values" playbook
Follow Along
Create image_docs.py and paste the code block from each step below in order into the same file. Steps 1–5 use a scanned vaccination record (official example); step 6 switches to invoice OCR verification (another official example; changes: swap the image and the field table). No Key yet? It still runs — the official json_cache.json stores the original responses for all the numbers on this page; the script detects the cache and replays it instead of sending real requests (delete the cache file if you want a real run).
Step 1: Set up the environment, download the official scanned-document example and cache
Why: the official cookbook recorded every API response into json_cache.json — in the official words: when the cache exists, the script replays those responses instead of calling the API, so you get the same numbers as the official run. Get the materials ready first, so every later step has official numbers to compare against.
Install dependencies (the official vaccination-record cookbook's install command, plus pillow — needed for cropping in step 5):
pip install ipython matplotlib pillow 'typesafe-sdk>=0.7.5,<0.8.0' \
'cooksafe>=0.2.0,<0.3.0'Download the scanned document and cache from the official examples repo (typesafe-public-examples, place in the same directory as the script):
curl -sL -o vaccination_record.jpg https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks/reading_scanned_documents/vaccination_record.jpg
curl -sL -o json_cache.json https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks/reading_scanned_documents/json_cache.jsonPaste in the file header — client, threshold constants, and the request function with caching:
import hashlib
import json
import os
from datetime import date
from pathlib import Path
from time import perf_counter
from cooksafe import JsonCache
from typesafe_sdk import Image, TypeSafeClient
TYPESAFE_MODEL = "jev-1.14.0" # Box questions need Jev 1.14
IMAGES_ON = {"TypeSafe-Danger-Zone-Images": "true"} # opt in to the image preview
READ_CUTOFF = 0.90 # below this, a person checks the reading against the paper
LEGIBLE_CUTOFF = 0.50 # the legibility Noul gets its own, lower cutoff
IN_DATE_CUTOFF = 0.50 # below this, a nurse checks whether the dose still covers the trip
client = TypeSafeClient(
api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"), # keyless kernels replay
timeout=120.0,
)
json_cache = JsonCache(Path("json_cache.json"))
PAGE = Path("vaccination_record.jpg")
PAGE_SHA = hashlib.sha256(PAGE.read_bytes()).hexdigest()[:16] # cache key for the scan
@json_cache
def ask(model: str, page_sha: str, questions_json: str) -> dict:
"""One request: the scan as the state, plus a set of questions about it."""
started = perf_counter()
response = client.system_one(
state={"page": Image(PAGE)},
questions=json.loads(questions_json),
model=model,
extra_headers=IMAGES_ON,
)
return {
"answers": {k: a.model_dump(mode="json") for k, a in response.answers.items()},
"seconds": round(perf_counter() - started, 2),
}
def ask_page(questions: dict) -> dict:
if not questions: # nothing to ask, such as a page with no rows found
return {"answers": {}}
return ask(TYPESAFE_MODEL, PAGE_SHA, json.dumps(questions, sort_keys=True))
print("setup ok")Run python image_docs.py.
You should see: setup ok — this step is just assembly, no request sent yet.
⚠️ Experimental feature, the official words deserve to be pinned on the wall
Images and Box are an early preview (officially called dreamer-ready, not production-grade): requests with images or Boxes must include the TypeSafe-Danger-Zone-Images: true header, and the request format may change between versions. Box questions require Jev 1.14 — jev-latest already points to jev-1.14.0. Play freely during prototyping; weigh the risks yourself before production. The images themselves have limits too: PNG/JPEG, 280–1568 pixels per side, ≤ 16 MiB per file; downscale high-resolution scans before sending. See the limits table in the Box and Images lesson.
setup okprinted out;vaccination_record.jpgandjson_cache.jsonare both in the working directory
Step 2: Jev image recognition, first question — is the Noul present, and which row is the Box on
Why: Jev doesn't do "extract the image into a list of data" — it answers questions about the image. So the official breakdown is: you have a checklist (the 16 vaccines given by this clinic), and you ask two questions for each candidate. This is exactly the solution from Box Always Returns One Box: Noul handles "is it there", Box handles "where"; if Noul says it's not there, ignore that box.
One request sends 33 questions: 16 vaccines × (Noul + Box) + 1 Noul asking "does the table contain any vaccines outside the checklist" — that last one was added deliberately by the officials: the other 32 questions can't catch the case of "the list missed something". See Parallel Questions and Speculative Fan-out for the mechanism of asking mixed question types in parallel.
VACCINES = [
"Hepatitis A", "Hepatitis B", "Hepatitis A and B combined", "Typhoid",
"Hepatitis A and typhoid combined", "Yellow fever", "Rabies", "Japanese encephalitis",
"Tick-borne encephalitis", "Cholera", "MenACWY", "MenB",
"Tetanus, diphtheria and polio (Td/IPV)", "MMR", "Varicella", "Influenza",
]
PRESENT_CUTOFF = 0.5 # P(recorded) at or above this counts the vaccine as on the page
def find_questions() -> dict:
questions = {"unlisted": {
"type": "noul",
"instructions": {
"question": "Does the vaccines table in `page` record any vaccine that is not "
"in this list?",
"list": VACCINES,
},
}}
for i, vaccine in enumerate(VACCINES):
questions[f"vac_{i:02d}"] = {
"type": "noul",
"instructions": f"Is {vaccine} recorded in the vaccines table in `page`, with "
"exactly this name?",
}
questions[f"vac_{i:02d}|box"] = {
"type": "box",
"instructions": f"the whole {vaccine} row of the vaccines table in `page`",
}
return questions
found = ask_page(find_questions())
F = found["answers"]
present = []
for i, vaccine in enumerate(VACCINES):
p, box = F[f"vac_{i:02d}"]["noul"], F[f"vac_{i:02d}|box"]
print(f"{vaccine:<40}recorded {p:.2f} box top {box['box']['top']:.2f}")
if p >= PRESENT_CUTOFF:
present.append({"vaccine": vaccine, "present": p, "box": box["box"],
"box_confidence": box["confidence"]})
print(f"\nanother vaccine on the page: {F['unlisted']['noul']:.2f}")
rows = sorted(present, key=lambda r: r["box"]["top"]) # box tops give the page order
for n, row in enumerate(rows, 1):
row["id"], row["position"] = f"row-{n}", nRun python image_docs.py.
You should see (numbers recorded by the official jev-1.14.0):
Hepatitis A recorded 0.98 box top 0.33
Hepatitis B recorded 0.98 box top 0.40
Hepatitis A and B combined recorded 0.02 box top 0.36
Typhoid recorded 0.98 box top 0.48
Hepatitis A and typhoid combined recorded 0.01 box top 0.34
Yellow fever recorded 0.98 box top 0.55
Rabies recorded 0.97 box top 0.69
Japanese encephalitis recorded 0.01 box top 0.35
Tick-borne encephalitis recorded 0.01 box top 0.34
Cholera recorded 0.01 box top 0.34
MenACWY recorded 0.00 box top 0.34
MenB recorded 0.01 box top 0.35
Tetanus, diphtheria and polio (Td/IPV) recorded 0.98 box top 0.62
MMR recorded 0.01 box top 0.33
Varicella recorded 0.01 box top 0.34
Influenza recorded 0.01 box top 0.34
another vaccine on the page: 0.02Three ways to read this:
- Similar-looking names can't fool it: the checklist deliberately includes close relatives like "hepatitis A + B combined vaccine" and "hepatitis A + typhoid combined vaccine" — the 6 actually on the page all score 0.97 or above, the close relatives all 0.02 or below, cleanly separated by a wide margin
box topis a free row number: the 6 rows sorted by thetopedge are exactly their order on the page — the cropping in step 5 and showing results to reviewers both depend on this order- No vaccines outside the checklist slipped through either: that "gatekeeping Noul" answered 0.02 — the table contains no vaccines outside the checklist. If it had been above 0.5, the whole page would need to be flagged for human review (step 5)
The officials also noted one thing: 16 vaccines in 33 questions is fine, but when the checklist has thousands of items, the questions may not fit in one request (context limits) — first group by category, ask one Noul per group to narrow the scope, then ask item-by-item details for the groups that hit.
💡 This is the unique value of Jev image recognition
One probability per candidate lets you turn "is it there" into a threshold question: there's no gray zone between 0.97 and 0.02; PRESENT_CUTOFF = 0.5 cuts cleanly. "I see records of hepatitis A and hepatitis B" from a multimodal LLM contains none of these numbers — you can only choose to believe it or not.
- 16 row probabilities printed: 6 at ≥ 0.97, close relatives ≤ 0.02,
unlisted0.02; I can explain why that last Noul can't be skipped
Step 3: Choice reads the year; if unreadable, answer unreadable
Why: finding the row is only the first half; you still need to read the vaccination year to know whether the shot is still valid. The key pitfall: Choice always picks an option — when the date is covered by a sticker, without an escape hatch it will force-pick the year that looks most similar. The official solution: put other and unreadable at the end of the option list so the model can explicitly say "can't read it"; plus a separate Noul asking "is the date fully readable" as a second signal.
YEAR = {
**{f"y{n}": str(n) for n in range(2016, 2027)},
"other": "A year before 2016 or after 2026",
"unreadable": "The year is covered, stained, smudged or damaged and can't be read",
}
def read_questions(row: dict) -> dict:
where = f"the DATE GIVEN cell of the {row['vaccine']} row of the vaccines table in `page`"
i = row["id"]
return {
f"{i}|year": {
"type": "choice", "criteria": YEAR,
"instructions": "Dates on this record are written day/month/year, so the year "
f"is the last part. What year is written in {where}?",
},
f"{i}|legible": {
"type": "noul",
"instructions": f"Is the date in {where} fully legible, with nothing covering, "
"staining, smudging or blurring any of it?",
},
}
read = ask_page({k: q for row in rows for k, q in read_questions(row).items()})
A = read["answers"]
for row in rows:
i = row["id"]
row["year"], row["year_confidence"] = A[f"{i}|year"]["choice"], A[f"{i}|year"]["confidence"]
row["legible"] = A[f"{i}|legible"]["noul"]
UNRESOLVED = ("other", "unreadable") # picks that leave the row without a year
def year_given(row: dict) -> int | None:
return None if row["year"] in UNRESOLVED else int(row["year"][1:])
def as_text(row: dict) -> str:
year = year_given(row)
return str(year) if year else row["year"]
print(f"{'vaccine':<40}{'year':<12}legible")
for row in rows:
print(f"{row['vaccine']:<40}{as_text(row):<12}{row['legible']:.2f}")Run.
You should see (numbers recorded by the official jev-1.14.0):
vaccine year legible
Hepatitis A 2026 0.96
Hepatitis B 2026 0.86
Typhoid 2021 0.96
Yellow fever 2025 0.96
Tetanus, diphtheria and polio (Td/IPV) 2024 0.96
Rabies unreadable 0.05Two details are worth pausing on:
For the rabies row, the model answered
unreadable— its date was covered by a batch sticker. The readability Noul simultaneously gave 0.05, the two signals corroborating each other. That's what the escape hatch means: unreadable gets reported, not guessedFor readable dates, the readability Noul only reached 0.86–0.96 (one of those 0.86 rows was highlighted with a fluorescent marker) — lower than Choice's confidence, so the officials set a separate, lower 0.50 review line for readability instead of reusing 0.90. Different questions have their own score scales; don't apply one threshold to all
Six years printed; I can explain what
otherandunreadableeach catch (years outside the list vs. dates that can't be read at all)
Step 4: Plain-text request to verify validity — separating reading from judgment
Why: This step is one of the most instructive designs in the cookbook. Reading the date correctly isn't enough — the typhoid shot was given in 2021, the handwriting is clear, but protection lasts about three years, so it had long expired by the departure date of 2026-11. The official approach doesn't have the model make this judgment against the image; instead, it sends the year that was read out as plain text in a third request: no image attached, the judgment concerns only "is this shot still valid on the departure date," fully decoupled from reading the image.
Date arithmetic (how many years ago) is computed by code and put into state — date comparison is a known weakness of Jev, so arithmetic always stays in code.
DEPARTURE = date(2026, 11, 12) # from the traveller's booking
def before_departure(year: int) -> str:
years = DEPARTURE.year - year
return "less than a year" if years == 0 else f"about {years} year{'s' * (years != 1)}"
CHECK_CONTEXT = (
"Each entry is one line of an adult traveller's vaccination record, as read from a "
f"scan. The traveller leaves on {DEPARTURE.day} November {DEPARTURE.year}. Judge "
"only whether the dose will still protect the traveller on that date, by usual "
"travel-health guidance for adults."
)
IN_DATE_QUESTION = "Will this dose still protect the traveller on the departure date?"
IN_DATE_CRITERIA = {
"true": "The dose is within the usual protection period for this vaccine on the "
"departure date.",
"false": "The dose is past the usual protection period for this vaccine by the "
"departure date.",
}
def check_questions(readable: list[dict]) -> dict:
return {
row["id"]: {
"type": "noul",
"instructions": {"question": IN_DATE_QUESTION, "entry": row["id"]},
"criteria": IN_DATE_CRITERIA,
}
for row in readable
}
def check_state(readable: list[dict]) -> dict:
return {
"context": CHECK_CONTEXT,
"entries": [{"id": r["id"], "vaccine": r["vaccine"], "year_given": as_text(r),
"given_before_departure": before_departure(year_given(r))}
for r in readable],
}The third request carries no image and adds no images request header — the judgment concerns only the year text that was read out, and it gets the same caching decorator:
@json_cache
def check(model: str, state_json: str, questions_json: str) -> dict:
started = perf_counter()
response = client.system_one(
state=json.loads(state_json), questions=json.loads(questions_json), model=model
)
return {
"answers": {k: a.noul for k, a in response.answers.items()},
"seconds": round(perf_counter() - started, 2),
}
readable = [r for r in rows if year_given(r) is not None]
checked = check(TYPESAFE_MODEL, json.dumps(check_state(readable)),
json.dumps(check_questions(readable))) if readable else {"answers": {}}
for row in rows:
row["in_date"] = checked["answers"].get(row["id"])
year = year_given(row)
ago = before_departure(year) if year else "-"
shown = " -" if row["in_date"] is None else f"{row['in_date']:.2f}"
print(f"{row['vaccine']:<40}{as_text(row):<12}{ago:<20}{shown}")Run.
You should see (numbers recorded by the official jev-1.14.0):
Hepatitis A 2026 less than a year 0.94
Hepatitis B 2026 less than a year 0.83
Typhoid 2021 about 5 years 0.38
Yellow fever 2025 about 1 year 0.95
Tetanus, diphtheria and polio (Td/IPV) 2024 about 2 years 0.93
Rabies unreadable - -Typhoid 0.38 — it got singled out. Rabies has no score because the year wasn't read (step 3 answered unreadable), so the code simply doesn't send it into this request.
Why must it be split into two requests? The official cookbook ran a counterexample experiment in the appendix — asking the validity question directly against the image (with the departure date written in the page header):
vaccine from the image from the readings
Hepatitis A 0.88 0.94
Hepatitis B 0.85 0.83
Typhoid 0.50 0.38
Yellow fever 0.96 0.95
Tetanus, diphtheria and polio (Td/IPV) 0.84 0.93
Rabies 0.88 -Asked against the image: rabies 0.88 passed (even though the date was clearly covered by a sticker); typhoid 0.50 just barely squeaked by too. The officials pinpoint the cause: one answer covered two steps — you can't tell whether the model misread the date or the judgment itself was wrong. After the split (the main flow on this page), rabies is stopped at the reading step, and typhoid's judgment score of 0.38 also goes to human review.
💡 This is the unique value of Jev image recognition
Every station in the pipeline has its own probability and its own review threshold; which station has a problem and who should look at it is spelled out clearly in the answer. Merging them into one question is like welding two alarms into one — and both go dead.
- Typhoid 0.38 and rabies
-both matched up; I can state the two benefits of "separating reading from judgment" based on the appendix counterexample
Step 5: Flag low scores, crop by Box and send to humans
Why: Three reading steps plus one judgment step — now all the low scores are gathered into a single review list. The rules are all pure code — Jev only outputs probabilities, who gets sent to human review is decided by your thresholds.
def reading_score(row: dict) -> float:
"""The weakest of the answers that say the row was found and read."""
return min(row["present"], row["year_confidence"])
def review_reasons(row: dict) -> list[str]:
reasons = []
if (row["year"] in UNRESOLVED or reading_score(row) < READ_CUTOFF
or row["legible"] < LEGIBLE_CUTOFF):
reasons.append("Check the reading")
if row["in_date"] is not None and row["in_date"] < IN_DATE_CUTOFF:
reasons.append("Check the dose")
return reasons
def page_reasons() -> list[str]:
"""Flags that belong to the whole page rather than to one row."""
if F["unlisted"]["noul"] >= PRESENT_CUTOFF:
return ["Check the page: it may record a vaccine outside the list"]
return []
for row in rows:
in_date = "–" if row["in_date"] is None else f"{row['in_date']:.2f}"
flags = ", ".join(review_reasons(row)) or "–"
print(f"| {row['vaccine']} | {as_text(row)} | {reading_score(row):.2f} | "
f"{row['legible']:.2f} | {in_date} | {flags} |")
print(f"page: {', '.join(page_reasons()) or 'not flagged'}")Run.
You should see (numbers recorded by the official jev-1.14.0):
| Hepatitis A | 2026 | 0.98 | 0.96 | 0.94 | – |
| Hepatitis B | 2026 | 0.98 | 0.86 | 0.83 | – |
| Typhoid | 2021 | 0.98 | 0.96 | 0.38 | Check the dose |
| Yellow fever | 2025 | 0.98 | 0.96 | 0.95 | – |
| Tetanus, diphtheria and polio (Td/IPV) | 2024 | 0.98 | 0.96 | 0.93 | – |
| Rabies | unreadable | 0.97 | 0.05 | – | Check the reading |
page: not flagged (another vaccine on the page: 0.02)Two flagging rules and one page-level rule:
- "Verify reading": the year falls into
other/unreadable, the reading score (the weaker ofpresentandyear_confidence) is below 0.90, or readability is below 0.50 — any one triggers it - "Verify shot": validity score below 0.50. Note a high reading score for typhoid cannot rescue a low judgment score — in the officials' own words: the judgment is based on a year that may have been misread, so the two flags don't cancel each other out
- Page-level flag: if Noul for
unlistedis ≥ 0.5, the whole page goes to review — that vaccine has no row of its own, so the row-level rules can't catch it
Finally, the part for humans: crop out the full flagged row by Box (the official code draws with matplotlib; here is the cropping core — the box saved in step 2 now comes in handy):
PAD = 0.02 # fraction of the page added around each box before cropping
LOW_BOX_CONFIDENCE, LOW_PAD = 0.6, 0.05 # a less certain box gets a wider margin
def crop(box: dict, confidence: float):
from PIL import Image
page = Image.open(PAGE)
w, h = page.size
pad = LOW_PAD if confidence < LOW_BOX_CONFIDENCE else PAD
top, bottom = max(0.0, box["top"] - pad), min(1.0, box["bottom"] + pad)
left, right = max(0.0, box["left"] - pad), min(1.0, box["right"] + pad)
return page.crop((int(left * w), int(top * h), int(right * w), int(bottom * h)))
for row in rows:
if review_reasons(row):
crop(row["box"], row["box_confidence"]).save(f"review-{row['id']}.jpg")
print(f"saved review-{row['id']}.jpg ({', '.join(review_reasons(row))})")After running, review-row-3.jpg (typhoid, verify shot) and review-row-6.jpg (rabies, verify reading) land in the run directory — the reviewer doesn't have to hunt through the full scanned page, they just look at the cropped row.
The officials close with tuning advice: these thresholds (0.90 / 0.50 / 0.50) are a starting point, not gospel — use human-verified records to tune your own lines. The rationale behind the threshold approach is in Confidence.
- The two flags don't cancel each other: typhoid is flagged only for the shot, rabies only for the reading; my
review-*.jpggot cropped out
Step 6: Invoice OCR verification — two checks catch three misreads
Why: We've read scanned pages — what about the reverse? OCR extracts values from invoice photos — most are correct, but checking them one by one takes as long as typing them by hand. The official approach: Jev verifies every extracted value twice — once against the full page (page check), once against a cropped image (crop check); anything below 0.25 goes to a human. The reviewer only looks at the few flagged ones.
The scenario is very realistic: one is a deeply creased carbon-copy invoice (Milkwood, 1393×1080), the other has a shadow across the middle (Siyakhula, 899×1599) — OCR reads 8 fields (supplier, invoice number, date, amount due, four bank items), of which three are wrong:
| Invoice | Field | What OCR extracted | What's printed on paper |
|---|---|---|---|
| Milkwood | Invoice number | AN400074 | IN400074 |
| Siyakhula | Invoice number | INV03070 | INV03079 |
| Siyakhula | Date | 16/09/2028 | 16/09/2026 |
The officials' numbers come from jev-1.14.0 (recorded 2026-10-15). This part requires installing the typesafe-sdk[pil] extension separately and downloading a second set of examples; dependencies and downloads:
pip install ipython pillow 'typesafe-sdk[pil]>=0.7.5' 'cooksafe>=0.2.0,<0.3.0'The invoice photos, extractions.json (the values OCR extracted), and the second json_cache.json are downloaded from the ocr_confidence_bounding_boxes directory of the official examples repo; put the photos in the inputs/ folder. Note that this cache has the same name as the json_cache.json from step 1, so putting them in the same directory overwrites each other — open a separate folder for the invoice examples, or finish running the first five steps before downloading. The officials state plainly: the OCR and LLM steps (they use Apple Vision plus an LLM) are not in the code — any extractor works, because verification only needs the photos and the extracted values.
Three requests, run once per invoice. First request: full-page photo plus two questions per field (Box to find the location, Noul to check "is the extracted value this field"):
FIELDS = {
# field: what it is, in the words every question uses
"supplier_name": "the supplier's (seller's) business name",
"invoice_number": "the invoice or document number",
"invoice_date": "the invoice date",
"amount_due": "the final total amount payable",
"bank_name": "the name of the supplier's bank in the payment details",
"account_holder": "the account holder name in the supplier's bank details",
"account_number": "the supplier's bank account number",
"branch_code": "the supplier's bank branch code",
}
SAME_VALUE = "Ignore differences in spacing, punctuation, currency symbols and letter case."
# request 1: the whole photo, a Box + a Noul per field
def page_questions(values: dict) -> dict:
from typesafe_sdk import Box, Noul
questions = {}
for field, value in sorted(values.items()):
questions[f"box_{field}"] = Box(
instructions=f"{FIELDS[field]}, printed as “{value}” in `page`"
)
questions[f"fits_{field}"] = Noul(
instructions=f"Is “{value}” {FIELDS[field]} on `page`? {SAME_VALUE}"
)
return questionsThe second request crops the area around each field's box and re-asks Box at a larger scale (the region spans at least 30% of page width and 15% of page height, so even if the first box is off, it stays within the region); the third request crops each new box (with margins added on all four sides) and asks one Noul: "Is this value clearly printed on this cropped image? Answer no if it's cut off, illegible, or different" — Ignore differences in spacing, punctuation... makes differences in spacing, punctuation, capitalization, and currency symbols not count as errors. Each field's cropped image is embedded in the instructions of its own question and client.attach_assets() brings them all out at once (the mechanism for referencing images in body text is covered in the advanced techniques of the Box and images lesson).
Everything lands in one decision function: if any check is below BAR = 0.25, it's review. The officials' scoreboard after running:
photo # field extracted fits page crop verdict
milkwood.jpg 1 supplier_name MILKWOOD mushrooms 0.98 0.95 ok
milkwood.jpg 2 invoice_number AN400074 0.88 0.18 REVIEW
milkwood.jpg 3 invoice_date 01/09/2026 0.53 0.82 ok
milkwood.jpg 4 amount_due 504.70 0.95 0.80 ok
milkwood.jpg 5 bank_name Standard Bank 0.98 0.53 ok
milkwood.jpg 6 account_holder MILKWOOD 0.90 0.57 ok
milkwood.jpg 7 account_number 080 11 4941 0.35 0.92 ok
siyakhula.jpg 1 supplier_name Siyakhula Trading Services 0.96 0.31 ok
siyakhula.jpg 2 invoice_number INV03070 0.02 0.04 REVIEW
siyakhula.jpg 3 invoice_date 16/09/2028 0.17 0.66 REVIEW
siyakhula.jpg 4 amount_due R1,305.65 0.97 0.84 ok
siyakhula.jpg 5 bank_name Standard Bank 0.97 0.98 ok
siyakhula.jpg 6 account_holder Siyakhula Trading 0.93 0.96 ok
siyakhula.jpg 7 account_number 10 13 205 260 5 0.96 0.78 ok
siyakhula.jpg 8 branch_code 051001 0.97 0.94 okLook closely at the scores of the three misreads — this is the most important table in the whole lesson:
AN400074(the paper printsIN400074, the initial I was read as an A): the page check at 0.88 let it pass (the small text is too small in the full-page photo), the crop check at 0.18 caught it16/09/2028(what's printed on the paper is16/09/2026, with the last digit 6 misread as 8): the crop check 0.66 let it pass, while the page check 0.17 caught itINV03070(what's printed on the paper isINV03079, with the last digit 9 misread as 0): both checks scored low (0.02 / 0.04), a double catch
The official words: neither check alone catches all three cases. The two checks offer complementary perspectives—the full page asks "does this value look plausible in this layout," while the crop asks "are these specific characters actually correct." And all the correct values passed both checks (lowest 0.35 / 0.31).
- I can explain which check caught and which check let pass each of the three misreads;
BAR = 0.25is an "or" judgment, not an "and"
📋 Magic Prompt
The "Prompt" on this page is a questions-and-code template—copy it, replace the 【】 placeholders, and you can apply it to your own scanned documents. Three pieces: the line-finding duo, the value-reading escape hatch, and the OCR double check.
📝 Line-Finding Duo (Noul + Box edition)
Expected effect: for any need of the form "things on a list may appear in a document, find and locate them"—vaccines, signatures, stamps, entry rows—the same prompt works by just swapping the list.
# 必填项:【清单】换你要找的东西列表;instructions 里的 `page` 换你的图在 state 里的字段名
questions = {
"unlisted": Noul(instructions={
"question": "Does the table in `page` record any 【东西类型】 that is not in this list?",
"list": 【清单】,
}),
}
for i, item in enumerate(【清单】):
questions[f"item_{i:02d}"] = Noul(
instructions=f"Is 【东西】 recorded in the table in `page`, with exactly this name?")
questions[f"item_{i:02d}|box"] = Box(
instructions=f"the whole 【东西】 row of the table in `page`")
# 选填项:近亲名称故意放进清单里测区分度(官方放了 Hepatitis A and B combined)?
# 输出规范:P(yes) ≥ 0.5 视为在页上;按 box["box"]["top"] 排序得到页面顺序
# 约束条件:
# ✅ Noul 说不在页上,就不理会它那个 Box——Box 永远返回一个框
# ✅ unlisted 那道守门题保留:行级规则抓不到「列表漏了」
# ❌ 清单几千项别硬塞一个请求——先按组 Noul 预筛,再对命中组细问✅ Value-Reading Escape Hatch (Choice edition)
Expected effect: read any closed-set value from an image (year, month, denomination, edition number), and when it can't be read, get an explicit unreadable instead of a wild guess.
# 必填项:【选项集】+ 两个逃生舱;【值在原文里的写法】写进 instructions 帮模型定位
VALUE = {
**{【你的选项集】},
"other": "【选项集之外的值】",
"unreadable": "The value is covered, stained, smudged or damaged and can't be read",
}
question = Choice(
instructions="【值在原文里怎么写的(如 day/month/year,年份是最后一部分)】"
"What 【值】 is written in 【位置描述,如 the DATE GIVEN cell of the X row】?",
criteria=VALUE,
)
# 选填项:旁边加一道 Noul 问「这段完全可读吗」做第二信号(官方:可读性的分数尺度更低,阈值单独设)?
# 输出规范:answer.choice + answer.confidence;other/unreadable 让该行无值可判,直接送人工
# 约束条件:
# ✅ Choice 永远挑一个——没有逃生舱它就硬猜,这是本模板存在的理由
# ❌ 不要把「读值」和「按值判断」合并成一道题对着图问(见第 4 步附录反例)🔧 OCR Double-Check Pipeline (code edition)
Expected effect: verify every field extracted by OCR twice, routing to a human and drawing a review box when below threshold.
BAR = 0.25 # a check scoring below this flags the field
def check_invoice(photo, values) -> dict:
first = find_fields(photo, values) # 请求 1:整页 Box + Noul 页面核对
tight = zoom_fields(photo, values, first["boxes"]) # 请求 2:放大重框
shown = check_crops(photo, values, tight["boxes"]) # 请求 3:裁剪 Noul 细看
return {
field: {
"value": values[field],
"box": tight["boxes"][field],
"fits": first["fits"][field], # 页面核对分
"shows": shown["shows"][field], # 裁剪核对分
"review": first["fits"][field] < BAR or shown["shows"][field] < BAR,
}
for field in sorted(values)
}
# 必填项:【照片】【values:OCR 抽出的字段 map】;阈值 BAR 按你的容错调
# 选填项:复核框画在原图上——review 橙色、通过蓝色,复核的人对着位置看?
# 输出规范:每字段 {value, box, fits, shows, review};review=True 进人工队列
# 约束条件:
# ✅ 任一检查低于阈值就转人工——「或」不是「且」(单次检查抓不全)
# ✅ SAME_VALUE 句式保留:空格、标点、大小写、货币符号差异不算错
# ❌ 别指望整页一次看全:小字要靠裁剪放大,两次检查视角互补Checkpoints ✅
- I ran the full vaccine-scan workflow end to end: 33 line-finding questions → Choice reads the year (
unreadableescape hatch) → plain-text request checks validity → low-score flagging + cropping - I can explain why image reading and judgment must be split into two requests, and what the merged version missed (rabies 0.88 passed, typhoid 0.50 barely passed)
- I can go through the scoreboard and say which check caught each of the three OCR misreads:
AN400074by the crop,16/09/2028by the full page,INV03070by both - All my thresholds are labeled by purpose: reading 0.90, readability 0.50, validity 0.50, OCR 0.25—and I know to tune them myself using human-verified samples
- I know that both image requests carry
TypeSafe-Danger-Zone-Images: true, while the third plain-text request doesn't need it
Pitfall Reminders
❌ Without an escape hatch, Choice just guesses blindly
A date covered by a sticker, a number smudged by a stain—Choice always picks an option. With no "can't read it" exit, it will pick the most plausible-looking year for you, and with not-low confidence either. The official solution is a double safeguard: append other (out of list) and unreadable (unreadable) at the end of the option list, plus a separate readability Noul alongside. In the official example, the rabies line: Choice answered unreadable, readability Noul 0.05—both signals were correct.
❌ Merging image reading and judgment into one question
"Is this shot still valid on the departure date? The departure date is in the page header"—ask this directly against the image, and in official testing rabies got 0.88 and passed (the date was clearly covered), typhoid 0.50 barely passed. The reason: one answer covers two steps, so you can't tell whether it misread the image or misjudged. The correct split: the image-reading request produces structured readings, and the judgment request only receives text—rabies gets stopped at the reading step, and typhoid's judgment of 0.38 goes to a human.
❌ Taking Box's boxes at face value
Box always returns a box—even for things not in the image, it still gives four edges. In the vaccine record, 10 vaccines that aren't on the page each got a box like box top 0.33. The correct approach is the duo from step 2: only use the box when Noul says it's present (≥ 0.5); when the out-of-list gatekeeping Noul is ≥ 0.5, send the whole page for review. Also note: BoxAnswer's confidence may be null per the API schema (when the model doesn't provide it)—check for null before threshold comparison so your code doesn't hit a direct TypeError.
❌ Sending tiny images as-is, not downscaling big ones
Exceed the API's image limits and the request is rejected outright—no automatic resizing: static PNG/JPEG images, 280–1568 pixels per side, 78,400–2,458,624 total pixels, ≤ 16 MiB per file. For high-resolution scans, first compress the longest side down to 1568; conversely, if a cropped small image has a side smaller than 280, upscale it (the official examples upscaled the re-boxed region image to 1200 and the crop-check image to 800 before sending; if it's really too small, paste it onto the center of a white canvas and record the canvas position to convert coordinates back). One more gotcha: the API re-encodes during processing, so the processed image can end up larger than the original file—sending a bunch of small images can still hit the "all images in the request combined 32 MiB" cap.
Advanced Techniques (optional)
The same playbook applied to consecutive video frames: a feeder-cam species log. The official third image cookbook (Feeder-cam visit log): a 101-second feeder-cam video, one frame cut per second, with 5 questions per frame—is there a bird (Noul), how many (Choice: none/one/several), where is the most prominent bird (Box), and what species are the leftmost and rightmost birds (two Choices, 10 bird species plus other, where each option describes "what this bird looks like in the camera frame" rather than a field-guide-style description). Official results: bird_present vs. human labels 101/101 all correct, species recognized 132/133; median time per frame 0.54 seconds, with 4 threads in parallel.
Three designs worth copying: first, Box returns only one box per question—when two bird species share a frame, the second request asks for one box per species by species name ("the male northern cardinal (the bright red bird)"); when the overlap of two boxes (intersection over union) is ≥ 0.6, they're treated as the same bird counted twice, keeping only the higher-confidence one. Second, visits() turns frame answers into a "visit log"—its three parameters (species probability ≥ 0.6, gap ≤ 2 frames, at least 2 frames to count as a visit) are all code-side matters that Jev can't see—retuning parameters doesn't require resending requests. Third, birds of the same species in a flock can't be separated—the official suggestion is a zero-shot detector (such as Meta's SAM 3) to first find all boxes by "bird," and then Jev recognizes the species for each cropped bird.
Test Chinese-language scans before going live. The official docs state Jev is trained primarily on English, and other languages (including Chinese) work but with lower accuracy—before putting Chinese documents into production, first run a batch with your own scans, focusing on the probability distribution and confidence (for language support details, see Models and Pricing).
Know what the costs look like. Images are not priced separately—each image is billed by input tokens at the same rate as text, and usage.input_tokens includes the tokens for all images; images also take up context (64k per-request / 32k per-question), and stuffing too many images into one request will trigger max_tokens_exceeded. The caching mechanism in the official example saves you money during development: responses are cached by request content, so reruns don't resend.
Connect your own documents. The official docs explicitly say this approach fits any document where the set of answers is known: swap VACCINES for your list, YEAR for your value range, the validity-date question for your business judgment, and keep both other and unreadable as escape hatches. For the plain-text version of value extraction (dates, emails, phone numbers, amounts), see Value Extraction; for how to split extraction and verification across multiple models to save money, see Cascade Extraction.
Lesson Summary
✅ Jev image recognition reads scanned documents: Noul checks whether it's there, Box finds which line (top-edge ordering is the page order), and Choice reads the value with other/unreadable escape hatches—unreadable results get reported, not guessed.
✅ Separate image reading from judgment: the reading request carries the image, the judgment request is pure text. Official counter-example: merge them into one question, and rabies at 0.88 wrongly passes the gate while typhoid at 0.50 barely passes—one answer covers two steps and all the alarms go silent.
✅ Double OCR checks: full-page verification + cropped verification; if either falls below 0.25, route to a human. All three misreads were caught—either check alone would have let some slip through. Review boxes are drawn on the original image, so humans only look at the flagged parts.
✅ Same playbook for video frames: gatekeeper Noul + Box + Choice species question, 101/101 decides whether there's a bird; all visit-log thresholds live in code, so tuning doesn't require resending requests.
Key configuration fields recap:
| Field | Value on this page | Notes |
|---|---|---|
TYPESAFE_MODEL | jev-1.14.0 | Box questions need Jev 1.14; pin the version ID, don't use the drifting jev-latest |
IMAGES_ON | {"TypeSafe-Danger-Zone-Images": "true"} | Required on every request with images/Boxes; not needed for plain-text requests |
PRESENT_CUTOFF | 0.5 | Only use a Box when P(on page) ≥ 0.5 |
READ_CUTOFF / LEGIBLE_CUTOFF | 0.90 / 0.50 | Read score and legibility score are on different scales, so set thresholds separately |
IN_DATE_CUTOFF | 0.50 | Below this, validity judgment goes to a human; it doesn't cancel out read-score flags |
BAR | 0.25 | The review line for the double OCR checks; either check falling below it triggers review |
unreadable / other | escape hatches | the former catches "unreadable", the latter catches "outside the value range" |
| Image limits | 280–1568 pixels per side, ≤ 16 MiB per file | Oversized files are rejected outright, not downscaled; small crops must be upscaled before sending |
Related tutorials:
- Box and Images—the foundation of this page: how to attach images, four-edge conversion, Noul as gatekeeper
- Value Extraction—plain-text extraction of dates, emails, phone numbers, amounts: componentized questioning + regex candidates
- Confidence—three-tier usage (high: auto-execute / medium: flag / low: hand off to human) and tuning thresholds by risk
Next Lesson Preview
That wraps up the C track: you now have text judgment, value extraction, and image recognition. Next we move into the advanced handbook—another form of Jev: the browser Agent. Next lesson: Advanced Handbook: Jev Ultrafast Browser Agent and Pitfall Guide.
You will learn:
- What Jev Ultrafast is: give it a goal, Jev picks the action and the element, and a small LLM only writes strings when typing
- Dynamic indexed action space: a new element table is generated per observation, deciding action and target in a single round trip
- How it runs a real Google Flights search in 7 seconds, and how to install it
- The 9 known failure modes of Jev 1.13—the flip side of this page's "decompose the question" philosophy