Skip to content

Cascade extraction: the cheap model does the work, Jev verifies, the expensive model is the backstop ​

💡 One-sentence summary: use Jev to build a model cascade pipeline—the cheap model extracts first, Jev verifies field by field (one Noul per field per failure mode), and only an any_flag alarm escalates to the reasoning model, so cost is spent only on the few flagged items.

Tasks like "structured extraction"—turning web pages and documents into JSON records—have an awkward cost structure: large reasoning models extract well but slowly and expensively, while small models are cheap but commit the most insidious kind of error. The official SDE cascade cookbook puts it bluntly: cheap models make mistakes, and expensive reasoning models extract well but are both slow and costly. A cascade is the solution: the small model does the rough work first, TypeSafe verifies its output field by field, and only a verification alarm escalates to the large model.

This lesson is the "industrialized version" previewed at the end of value extraction. For single-field extraction, Jev alone can pick the right value; but when your task is batch-converting tens of thousands of web pages to JSON every day, having Jev extract field by field itself is too extravagant, while letting a cheap model handle everything is too risky to ship—this lesson's answer is a three-way division of labor: the cheap model extracts, Jev verifies, and the expensive model is the backstop. The official tested this loop on 100 real extraction tasks: the cascade curve sits above-left of every single model—cascade points at about $0.02 per item reach quality around 0.79, while the strongest gpt-5.5-reasoning alone at 0.82 costs about $0.045. This lesson takes about 20 minutes and is advanced level.

What you'll be able to do after this lesson ​

  • Build a three-tier model cascade: gpt-5.4-mini extracts → Jev verifies field by field in one request → escalate to gpt-5.5 reasoning only when any_flag raises an alarm—most items pay only for the cheap model
  • Give every record per-field flags: which field is suspicious and why (fabrication / misalignment / omission / format violation), instead of one vague overall score
  • Specifically catch fabrications that are "schema-valid but semantically wrong"—the kind that pass JSON Schema validation with flying colors
  • Use the official approach to pick your own operating point: sweep the alarm threshold from 0 to 1 and plot the cost/quality curve, rather than just settling on a 0.7

Why use Jev for verification instead of handing everything to the expensive model? ​

CapabilityExpensive model handles everythingCheap model + Jev cascade
Cost❌ Pay the reasoning price for every item: official curve gpt-5.5-reasoning about $0.045/item✅ Most items pay only the mini price plus one Jev verification: official curve about $0.02/item
Quality✅ Best among single models (0.82)✅ About 0.79, only 0.03 behind, at more than half the cost saved
Error visibility❌ All in one shot; when it's wrong, you don't know which field or which kind of error✅ One Noul per failure mode; flags name the field + cause
Plausible fabrications❌ JSON Schema only checks shape; it can't catch semantic fabrication✅ Semantic verification specifically catches schema-valid hallucinated values
Speed❌ Every item waits for deep reasoning✅ Items without alarms never enter the reasoning tier

In one sentence: the expensive model is "full insurance," while the model cascade is "cheap model does the work + a cheap inspector checks everything + only defective items get reworked"—the inspector must be far cheaper than the master craftsman for this math to work (Step 5 runs this calculation).

Your current predicament ​

  • "Tens of thousands of web pages need to become structured records every day; putting everything on reasoning models breaks the budget, but with only cheap models, you don't know which item is being fabricated"
  • "The cheap model's errors happen to be schema-valid: JSON Schema validation is all green, fabricated descriptions still land in the database, and when something goes wrong you can't even trace back which step failed"
  • "You want to add manual spot checks but don't know which items to sample—every record looks perfectly fine"

When to use this technique ​

When you need:

  • Large-scale structured extraction: web pages, documents → JSON records, at a volume where cost matters
  • The cheap model's error pattern is "plausible fabrication" rather than "broken formatting" (format errors can be caught by validation itself)
  • Each record carries a flag of "which field is suspicious and why," giving rework and spot checks a clear target

And you don't want:

  • To pay reasoning-model prices for every record
  • Verification results that are a blob of text that "looks reviewed" rather than numbers you can set thresholds on and sort by
  • To crash on parse failure — in the cascade, a response that can't be parsed is treated as an empty record, handed to verification to trigger an alarm and escalation, so the failure direction is safe

🎒 Preparation Before You Start ​

  • Completed Noul Yes/No Questions: every question in verification is a Noul, noul is the P(true) of 0–1, with no confidence
  • Successfully ran Installation and First Request: .venv is available and can read response.answers
  • Recommended reading Value Extraction: this page is the "industrialized pipeline version" teased at its end — how to pick the right single value is covered there; this page only handles the cost accounting of "multiple fields + multiple models"
  • Two keys: TYPESAFE_API_KEY (Jev verification) + OPENAI_API_KEY (the official example's rung 0 / rung 1 are OpenAI's gpt-5.4-mini / gpt-5.5)
  • A cup of water ☕ — 20 minutes, advanced level: a complete pipeline spanning three models

Follow Along ​

Create sde_cascade.py and paste the code block for each step below in order into the same file. First, let's be clear about how it runs: all outputs on this page are recorded numbers from the official cookbook, reproduced as-is so you can check your answers; when you run it live the numbers may differ slightly, but "which head raises the alarm" should be consistent. The official setup caches both LLM and TypeSafe calls in json_cache.json — reruns don't cost money again; delete that file if you want real requests.

Step 1: Install the environment, set the two keys — all three tiers in place ​

Why: the cascade spans three tiers — the two extraction rungs go through the OpenAI API, verification goes through TypeSafe. First install all dependencies and pin down three model constants and one alarm line; every later step only references them.

Install dependencies (official command verbatim; cooksafe is the official cookbook's companion utility library, 0.2.0 declares the dependency typesafe-sdk>=0.7.0,<0.8.0, which automatically installs the SDK as well):

bash
pip install openai jsonschema ipython 'cooksafe>=0.2.0,<0.3.0'

Set the two keys in your environment, then paste into the file header:

python
import json
import os
import urllib.request
from pathlib import Path

import jsonschema
from cooksafe import JsonCache, make_playground_link
from openai import OpenAI
from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

MINI = "gpt-5.4-mini"  # rung 0: cheap + fast
REASONING = "gpt-5.5"  # rung 1: strong, run with reasoning_effort="high"
TS_MODEL = "jev-1.14.0"  # the TypeSafe verifier model
FIRE_T = 0.7  # escalate if any per-field P(wrong) exceeds this

oai = OpenAI()
ts = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"], timeout=30.0)
print("setup ok")

Run python sde_cascade.py.

You should see: setup ok — this step is just assembly; no requests have been sent yet.

Read through the three constants and the whole cascade's staffing plan is set:

ConstantValue in the official exampleRole
MINIgpt-5.4-minirung 0: cheap + fast, does the rough work
REASONINGgpt-5.5 (with reasoning_effort="high")rung 1: strong but expensive, only appears when the alarm goes off
TS_MODELjev-1.14.0Verifier: scores P(problem) per field per failure mode
FIRE_T0.7Alarm line: escalate if any per-field head exceeds it

The two rungs are just two string constants — the code passes them verbatim to the chat completions API; to swap in your own cheap / expensive models you only change these two lines. When swapping the verification tier to a Jev version, lock the specific version ID (the jev-latest alias drifts with versions); see Models and Pricing.

  • setup ok printed out, and both keys are in the environment variables

Step 2: Prepare materials — an example where "the page has no answer" ​

Why: the touchstone for testing an extractor is not what it extracted correctly, but how it behaves when the page has no answer. The official materials include an example specifically designed to "test refusal to fabricate" for this lesson; download it first and look at it carefully.

python
EXAMPLE = Path("example.json")
EXAMPLE_URL = (
    "https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/"
    "cookbooks/sde_cascade/example.json"
)
if not EXAMPLE.exists():
    urllib.request.urlretrieve(EXAMPLE_URL, EXAMPLE)
example = json.loads(EXAMPLE.read_text())
schema = example["schema"]
prompt = example["prompt"]
content = example["content"]
print(prompt, json.dumps(schema, indent=2), content, sep="\n===========\n")

Run it and look at three things.

You should see (the official example's prompt and schema reproduced as-is; the page body is long, so it's summarized in bullet points):

text
Find the date fall registration opens at Wrenfield Community College in Wrenfield,
Oregon, for the 2026-2027 school year.
===========
{
  "title": "RegistrationOpen",
  "type": "object",
  "properties": {
    "registration_open_date": {
      "title": "Registration Open Date",
      "type": "string",
      "description": "The date registration opens for the fall semester, in mm/dd/yyyy format, such as 08/03/2026. Return a blank string if you are unsure."
    },
    "description": {
      "title": "Description",
      "type": "string",
      "description": "A short description of the registration open date, such as 'Registration opens for the fall semester'."
    }
  },
  "required": ["registration_open_date", "description"]
}
===========
(Wrenfield Community College's events calendar page: navigation, a search box,
the only event is "Fall 2026 Census Date — Monday, September 14, 2026",
i.e., the enrollment census date, which is the withdrawal deadline; the entire page contains no registration opening date, nor any description of it)

Two traps are buried in the material — you should spot them while reading the question:

  • The page has no answer: this is a university's events calendar page, and the only event is the fall census date (withdrawal deadline) — it contains neither a registration opening date nor any description of that date. A well-behaved extractor should refuse to fabricate these two fields

  • The schema carries a "sample value": the description of the description field says such as 'Registration opens for the fall semester' — a model looking for a shortcut will copy this sample verbatim into its answer. This is exactly the trap the cheap model will step into in Step 3

  • I can state the "test points" of this example: no answer on the page + a ready-made sample value lying in the schema

Step 3: First tier — cheap model extraction, deliberately without structured output ​

Why: both rungs extract with plain-text responses — no structured output, no tool calls, no JSON mode. The official rationale is counterintuitive: the errors this cascade is meant to catch are schema-valid; constrained decoding would only hide models that "break the schema because they misread the page," leaving verification nothing to work with. You can validate badly formatted errors yourself; only plausibly fabricated errors need semantic verification.

python
EXTRACT_SYSTEM = (
    "You extract structured data from documents. Return only values supported by the text. "
    "Follow any value format specified by the schema or its field descriptions."
)

# LLM and TypeSafe calls are cached in json_cache.json, so a re-run reuses earlier answers
# instead of calling the APIs again; delete the file to call them fresh.
json_cache = JsonCache(Path("json_cache.json"))


@json_cache
def extract(
    model: str,
    prompt: str,
    schema: dict,
    content: str,
    *,
    reasoning_effort: str | None = None,
    temperature: float | None = None,
) -> dict:
    user = (
        f"{prompt}\n\nReturn ONLY a JSON object matching this JSON Schema:\n"
        f"{json.dumps(schema, indent=2)}\n\nDocument:\n{content}"
    )
    kwargs = {
        "model": model,
        "messages": [
            {"role": "system", "content": EXTRACT_SYSTEM},
            {"role": "user", "content": user},
        ],
    }
    if reasoning_effort:
        kwargs["reasoning_effort"] = reasoning_effort
    if temperature is not None:
        kwargs["temperature"] = temperature
    text = oai.chat.completions.create(**kwargs).choices[0].message.content
    # The prompt asks for ONLY a JSON object, so parse the reply as-is -- no regex fishing a
    # substring out of a malformed reply. If json.loads fails, treat it as an empty extraction:
    # every field reads as absent, which the verifier flags and the gate escalates -- the safe
    # direction.
    try:
        return json.loads(text)
    except (ValueError, json.JSONDecodeError):
        return {}


mini_record = extract(MINI, prompt, schema, content, temperature=0)
print("mini extraction:\n", json.dumps(mini_record, indent=2))
print("\nschema-valid:", jsonschema.Draft202012Validator(schema).is_valid(mini_record))

Run python sde_cascade.py. With OPENAI_API_KEY set, one real mini request is sent, and the answer goes into json_cache.json.

You should see (official recording):

text
mini extraction:
 {
  "registration_open_date": "",
  "description": "Registration opens for the fall semester"
}

schema-valid: True

Pause for a moment — this record is valid but wrong, half right and half fabricated:

  • registration_open_date being empty is correct: the page indeed doesn't state a registration date, and the schema explicitly says to return an empty string when uncertain
  • description is fabricated: the page never described a registration date; mini copied the sample sentence from the schema field's description verbatim
  • jsonschema validation prints True — JSON Schema can't see this kind of error. What cheap models produce is exactly this kind of confident, all-green-schema fabrication; catching it is the semantic verifier's job

Now look at the except fallback: a response that fails to parse returns {} directly — every field reads as missing, verification raises the alarm, and the gate escalates. The failure direction is safe: better to spend one extra reasoning call than to let a bad record slip through.

  • mini's output matches schema-valid: True; I can point out why description is fabricated and why registration_open_date is correct

Step 4: Second level — Jev field-by-field verification, one Noul per failure mode ​

Why: one big question like "is this extraction correct" can only yield a single score, unable to say which field is wrong or why. The official approach is Decompose the questions — **each field × each failure mode, one narrow Noul each **, with wording uniformly framed as "true = something went wrong," so noul is P(problem). Flags carry the field name and the cause, and each question can have its wording adjusted and its threshold tuned individually. The whole question set is asked in parallel in a single system_one call (see parallel questions and speculative fan-out for the multi-question-one-request mechanism).

python
# metric -> (question, NoulCriteria)
MAIN_QUESTIONS = {
    "name_desc_mismatch": (
        "Does the `extracted_field` fail to match the field at `path` or the `description` in the "
        "`field_spec`? If the `description` is empty, judge against the `path` alone.",
        NoulCriteria(
            true="the `extracted_field` does not match the field name or its `description`",
            false="the `extracted_field` matches the field name and `description`",
        ),
    ),
    "type_mismatch": (
        "Does the `extracted_field` violate the `type` declared in the `field_spec`?",
        NoulCriteria(
            true="the `extracted_field` violates the declared `type`",
            false="the `extracted_field` conforms to the declared `type`",
        ),
    ),
    "unreasonable": (
        "Is the `extracted_field` one that a reasonable person would not have extracted for this "
        "`field_spec`?",
        NoulCriteria(
            true="a reasonable person would not have extracted this value",
            false="the extraction is reasonable",
        ),
    ),
    "hallucinated": (
        "Is the `extracted_field` unsupported by, or absent from, the source text?",
        NoulCriteria(
            true="the `extracted_field` is a hallucination -- not supported by, or absent "
            "from, the source text",
            false="the `extracted_field` is supported by the source text",
        ),
    ),
    "off_target": (
        "Does the source text fail to genuinely report the thing the `field_spec` describes, so the "
        "value was pulled from incidental text?",
        NoulCriteria(
            true="the source does not genuinely provide this field -- the value was pulled "
            "from incidental text",
            false="the source genuinely reports this field",
        ),
    ),
    "incomplete": (
        "Does the `extracted_field` fail to capture a value the source supports (note whether the "
        "`field_spec` is `required`)?",
        NoulCriteria(
            true="the field is wrongly empty, null, or missing a value the source supports",
            false="the field captures the value the source supports",
        ),
    ),
    "format_violation": (
        "Does the `extracted_field` violate the format or constraints implied by the `description`, "
        "the schema `type`, and the extraction instructions (e.g. date format, units, enum membership)?",
        NoulCriteria(
            true="the `extracted_field` violates the implied format or constraints",
            false="the `extracted_field` satisfies the format and constraints",
        ),
    ),
}
ABSENCE_QUESTION = (
    "The `extracted_field` is empty, null, or an empty collection. Does the source text contain the "
    "information the `field_spec` describes, making the empty result wrong?"
)
ABSENCE_CRITERIA = NoulCriteria(
    true="a value was wrongly omitted", false="returning nothing is correct"
)

OVERALL_JUDGE = (
    "Is this extracted record an incorrect extraction -- some value unsupported by the source or "
    "not conforming to the schema, required information missing or wrong, or some field hallucinated -- "
    "so it should be escalated to a smarter model?"
)
OVERALL_JUDGE_CRITERIA = NoulCriteria(
    true="the record is an incorrect extraction",
    false="the record is a correct extraction",
)


def is_empty(v) -> bool:
    return v is None or (isinstance(v, (str, list, dict)) and len(v) == 0)


def field_spec(name: str) -> dict:
    """Minimal spec pulled from the schema (unwrapping anyOf/null for optional fields)."""
    p = schema["properties"][name]
    branches = p.get("anyOf") or []
    typ = p.get("type") or next(
        (b["type"] for b in branches if b.get("type") != "null"), "unknown"
    )
    return {
        "path": name,
        "type": typ,
        "description": p.get("description", ""),
        "required": name in schema.get("required", []),
    }


def build_questions(record: dict) -> dict[str, Noul]:
    """The verify question set: one holistic __overall__ head plus a per-field battery,
    keyed field::metric."""
    questions: dict[str, Noul] = {
        "__overall__::judge": Noul(
            instructions=OVERALL_JUDGE, criteria=OVERALL_JUDGE_CRITERIA
        ),
    }
    # Loop over the schema, not the record, so a field the extractor left out still gets asked.
    for name in schema["properties"]:
        value = record.get(name)
        spec = field_spec(name)
        if is_empty(value):
            questions[f"{name}::absence_wrong"] = Noul(
                instructions={
                    "field_spec": spec,
                    "extracted_field": value,
                    "main_question": ABSENCE_QUESTION,
                },
                criteria=ABSENCE_CRITERIA,
            )
            continue
        for metric, (question, criteria) in MAIN_QUESTIONS.items():
            if metric == "type_mismatch" and spec["type"] == "unknown":
                continue
            questions[f"{name}::{metric}"] = Noul(
                instructions={
                    "field_spec": spec,
                    "extracted_field": value,
                    "main_question": question,
                },
                criteria=criteria,
            )
    return questions


@json_cache
def verify(record: dict) -> dict[str, float | str]:
    """Run the whole Noul battery over a record in one TypeSafe call; return {field::metric: P(true)}."""
    state = {
        "system_message": EXTRACT_SYSTEM,
        "instruction": "Extract the structured record from this document",
        "source_text": content,
        "schema": schema,
        "extraction": record,
    }
    questions = build_questions(record)
    answers = ts.system_one(state=state, questions=questions, model=TS_MODEL).answers
    return {qid: ans.noul for qid, ans in answers.items()} | {
        "playground_link": make_playground_link(state, questions)
    }


checks = verify(mini_record)
playground_link = checks.pop("playground_link")
print("playground link ready:", playground_link[:60], "...")
print(f"{'qid':<40}{'P(wrong)':>9}")
print("-" * 50)
for fld, p in sorted(checks.items(), key=lambda c: -c[-1]):
    flag = "  <== FIRES" if p > FIRE_T else ""
    print(f"{fld:<40}{p:>9.2f}{flag}")

Run it. With TYPESAFE_API_KEY set, a real Jev request is sent (question set in parallel, one round trip), and answers go into the cache.

You should see (official recording):

text
qid                                      P(wrong)
--------------------------------------------------
description::hallucinated                    0.98  <== FIRES
description::off_target                      0.97  <== FIRES
__overall__::judge                           0.90  <== FIRES
description::unreasonable                    0.86  <== FIRES
description::incomplete                      0.54
description::name_desc_mismatch              0.46
description::format_violation                0.41
registration_open_date::absence_wrong        0.05
description::type_mismatch                   0.04

Only the wrong fields score high: three heads of description cross the alarm line (fabricated 0.98 / misplaced 0.97 / unreasonable 0.86); the empty date field is correct, and absence_wrong is only 0.05 — Jev does not falsely accuse it just for being empty. __overall__::judge also reported 0.90, but it cannot say which field is wrong or why — the official version keeps it only to contrast "one big question per record" against "a field-by-field question set"; what actually decides escalation is the field-by-field question set.

There are three more places in the construction of the question set worth stopping to look at:

Design pointApproachWhy
7 failure-mode headshallucinated / off_target / unreasonable / incomplete / name_desc_mismatch / type_mismatch / format_violationfabrication, misplacement, missed capture, violating the field name or description, type violation, format violation — any non-empty field gets the full set asked (skip type_mismatch when type is unknown)
Empty fields only get absence_wrong askedan empty value has no "content," so the only thing that can be wrong is "the source text actually has information, so leaving it empty is wrong"an extractor that wants to fabricate a value and leaves the field empty will also be caught by this head
Loop over schema["properties"] instead of recordfields the extractor missed still get askeda per-record loop would let "missing fields" slip through as "not checked"

The instructions of the questions uses structured objects throughout — field_spec (the minimal spec pulled from the schema for the code), extracted_field, and main_question each occupy one field: the question and the data it references are separated, so swapping a field only swaps data without rewriting the question (see Noul yes/no questions for structured instructions usage). The state is also a structured five-field object: system_message / instruction / source_text / schema / extraction — the verifier can see the full context at extraction time, including the system prompt in effect then.

💡 This is the unique value of Jev

What the verifier gives is not a passage of text that "looks reviewed," but a calibrated P(problem) — one number per field per failure mode, sortable, thresholdable, plottable as curves. noul probabilities align with real outcomes (that's how TypeSafe is trained; see What is Jev), and the gap between 0.98 and 0.05 is the gap between "I caught this one" and "this one cannot be falsely accused." Ask an ordinary LLM to "check it over" and you get a sentence like "the description seems a bit off" — no number, no field name, and certainly no cause.

  • The checklist matches the official numbers; I can say which three heads description triggered and why the absence_wrong of registration_open_date is low

Step 5: Third level — any_flag gating, escalate to the reasoning model only when the model cascade alarms ​

Why: the checklist gives nine numbers, but "when to escalate" must be an explicit line in the code. The official gate is called any_flag: if any field-by-field head exceeds FIRE_T, escalate — take the max, not the average. With an average, the red flag of 0.98 would be diluted below the alarm line by the low scores of the other heads, and a confident alarm would be "averaged into silence."

A single threshold only works given the separation seen in step 4: wrong fields score high, correct fields score low. This accounting also happens at this step — the officials state outright that the verifier must be far cheaper than the reasoning rung, or the cascade saves no money at all: pay "mini + verification" for every record, pay the reasoning cost only for records that alarm — only this structure pays off.

python
# any_flag is a per-field gate: the holistic __overall__ head is shown above but not part of it
fired = {
    qid: p
    for qid, p in checks.items()
    if not qid.startswith("__overall__") and p > FIRE_T
}
escalate = bool(fired)

print(
    f"any_flag gate (threshold {FIRE_T}): {'ESCALATE' if escalate else 'ACCEPT cheap result'}"
)
for qid, p in sorted(fired.items(), key=lambda c: -c[1]):
    print(f"  fired: {qid}  (P={p:.2f})")

Run python sde_cascade.py. The gate is pure code and sends no requests.

You should see (official recording):

text
any_flag gate (threshold 0.7): ESCALATE
  fired: description::hallucinated  (P=0.98)
  fired: description::off_target  (P=0.97)
  fired: description::unreasonable  (P=0.86)

All three fired flags point to description; the 0.90 from __overall__::judge is explicitly excluded — it serves only as a contrast and does not enter the gate. Only alarmed records pay the reasoning cost: gpt-5.5 with reasoning_effort="high" re-extracts just this one record; records that didn't alarm use mini's result directly.

python
final_record = (
    extract(REASONING, prompt, schema, content, reasoning_effort="high")
    if escalate
    else mini_record
)

print("mini      :", json.dumps(mini_record))
print("reasoning :", json.dumps(final_record))
print("\nfield-level diff (mini -> final):")
for name in mini_record:
    if mini_record[name] != final_record.get(name):
        print(f"  {name}: {mini_record[name]!r}  ->  {final_record.get(name)!r}")

Run it. With OPENAI_API_KEY set, a real reasoning request is sent, and the answer goes into json_cache.json.

You should see (official recording):

text
mini      : {"registration_open_date": "", "description": "Registration opens for the fall semester"}
reasoning : {"registration_open_date": "", "description": ""}

field-level diff (mini -> final):
  description: 'Registration opens for the fall semester'  ->  ''

The diff is a single line: the reasoning model threw away the fabricated description, replacing it with an empty string — it saw that the page never described a registration date and refused to invent one. registration_open_date is an empty string on both sides, untouched. The cascade paid the reasoning cost on this record, and it paid it because the verification named the problem.

  • The gate prints ESCALATE, the three fired flags match the official ones; in the diff description becomes an empty string; I can say why the max is taken instead of the average and why __overall__::judge doesn't enter the gate

Step 6: understand the official cost/quality curve over 100 records ​

Why: FIRE_T = 0.7 wasn't picked out of thin air. The officials ran the same loop (extract → verify → escalate, gpt-5.4-mini → gpt-5.5-reasoning, an any_flag gate on the field-by-field heads) end to end over 100 ScrapeGraphAI-100k extraction tasks: TypeSafe scored the cheap rung's extraction for every record, swept the gate threshold from 0 to 1, and plotted each threshold as a (cost, quality) point.

Note the provenance: these are the officials' internal test results, from an earlier round of independent runs — the verification model in that round was jev-1.12 (the examples on this page use jev-1.14.0); the quality score is the score an LLM judge gave each extraction (mean llm_judge on the chart).

Official internal results: the cost/quality frontier over 100 extraction tasks

Four rules for reading it:

Chart elementWhat it isHow to read it
Black diamondsEach of the four models run on its ownCost rises with capability; the strongest gpt-5.5-reasoning sits in the top-right corner—the most expensive and the best
Blue dots (any_flag)Cascade configurations at many different gating thresholdsThe lower the threshold, the more items get escalated; the higher, the fewer; one threshold, one dot
Dashed lineThe pareto frontierConnecting the cascade configurations with the highest quality at the same cost
Position of the frontierup and to the left of every single modelCascade points cost about $0.02/item for about 0.79; gpt-5.5-reasoning run on its own 0.82 costs about $0.045

In one sentence: the cascade frontier sits up and to the left of every single model—the cheap rung takes care of the easy items, and only flagged items pay for reasoning, so quality is higher at the same cost and cost is lower at the same quality. 0.7 is not a magic number; it is just one point on the 0→1 sweep line. Where your own task should pin it must be decided by sweeping your own data (the magic Prompt includes the sweep method).

  • I can point out the pareto frontier on the official chart, explain what "up and to the left" means, and say which dot each pair of numbers—0.79 / $0.02 versus 0.82 / $0.045—corresponds to

📋 Magic Prompt ​

The "Prompt" on this page is a set of questions and code templates for the API—copy it, replace the 【】 placeholders (those marked with ? are optional), and you can hook it into your own extraction task. Three pieces: a per-field verification question set, any_flag gating, and a threshold sweep.

📝 Per-field verification question set (Noul version) ​

Expected effect: for any task where "a cheap model produces JSON records that need field-by-field vetting"—product parameters, job postings, real estate listings—the same question set works once you swap the schema; flags name the field + the cause.

python
# 必填项:【schema】你的 JSON Schema(字段 description 写清格式与示例);【记录】便宜模型的输出
#         失败模式头逐字抄官方七连:name_desc_mismatch / type_mismatch / unreasonable /
#         hallucinated / off_target / incomplete / format_violation
MAIN_QUESTIONS = {
    "hallucinated": (
        "Is the `extracted_field` unsupported by, or absent from, the source text?",
        NoulCriteria(
            true="the `extracted_field` is a hallucination -- not supported by, or absent "
            "from, the source text",
            false="the `extracted_field` is supported by the source text",
        ),
    ),
    【其余失败模式】: (
        【同一句式:这个字段是不是犯了这种错?】,
        NoulCriteria(true=【犯了这种错的描述】, false=【没犯的描述】),
    ),
}
ABSENCE_QUESTION = (
    "The `extracted_field` is empty, null, or an empty collection. Does the source text "
    "contain the information the `field_spec` describes, making the empty result wrong?"
)
# 选填项:按业务加头——单位、币种、语言、时效;某头总误报先改措辞再调线?
# 输出规范:qid 取 f"{字段名}::{失败模式}";true 一律 = 出问题,noul 就是 P(出问题)
# 约束条件:
# ✅ 循环 schema["properties"] 而不是记录——提取器漏掉的字段照样被问到
# ✅ 空字段只问 absence_wrong;type 未知跳过 type_mismatch
# ❌ 不要拿一道「这条提取对吗」的大题代替题组——大题说不出哪个字段、哪种错

✅ any_flag gating (code version) ​

Expected effect: the alert logic is one line of pure code—if any per-field head crosses the line, escalate to the expensive model; items with no alert pass through with the cheap result directly.

python
# 必填项:【FIRE_T】报警线(官方示例 0.7;上线前用自己的数据扫一遍再定)
fired = {
    qid: p
    for qid, p in checks.items()
    if not qid.startswith("__overall__") and p > 【FIRE_T】
}
escalate = bool(fired)
final_record = (
    extract(【贵模型】, prompt, schema, content, reasoning_effort="high")
    if escalate
    else mini_record
)
# 选填项:逐头设线——官方明说每道题可单独调阈值,编造类从严、格式类从松?
# 输出规范:fired 字典就是审计日志——哪个字段、哪种病因、多大概率
# 约束条件:
# ✅ 取 max 不取平均——一个自信的红旗不能被低分稀释成沉默
# ✅ 解析失败的记录当空记录进核查——每个字段读作缺失,报警升级,方向安全
# ❌ 别把 __overall__::judge 门进门控——它只作对照,决定升级的是逐字段头

🔧 Threshold sweep (choose-your-working-point version) ​

Expected effect: don't just pick 0.7 blindly—take a batch of tasks, sweep the threshold from 0 to 1, record one (cost, quality) point per threshold, draw your own frontier, then pin the working point.

python
# 必填项:【任务集】一批提取任务(官方用 100 条 ScrapeGraphAI-100k);【打分】质量分怎么算
for fire_t in [i / 10 for i in range(11)]:        # 0.0 -> 1.0
    total_cost = total_quality = 0.0
    for item in 【任务集】:
        record = extract(【便宜模型】, item.prompt, item.schema, item.content)
        checks = verify(record)                    # 每条一次核查
        fired = any(
            p > fire_t for qid, p in checks.items()
            if not qid.startswith("__overall__")
        )
        final = extract(【贵模型】, item.prompt, item.schema, item.content,
                        reasoning_effort="high") if fired else record
        total_cost += 【便宜模型单价 + 核查单价 +(fired 时的贵模型单价)】
        total_quality += 【质量分:官方用 LLM judge 给每条提取打分】
    point = (total_cost / len(【任务集】), total_quality / len(【任务集】))
# 选填项:质量分用人工抽检代替 LLM judge,先抽 20 条看头的分离度?
# 输出规范:一张(成本, 质量)散点图;工作点钉在你预算线与前沿的交点
# 约束条件:
# ✅ 扫描和上线锁同一个核查版本 ID——别用会漂移的 jev-latest
# ✅ 先确认头的分离度(错字段高、对字段低),分离不开什么阈值都白搭
# ❌ 别把官方 0.79 / $0.02 当你的预期——那是 ScrapeGraphAI-100k + jev-1.12 那一轮的数字

Checkpoints ✅ ​

  • The step 5 gate prints ESCALATE, and the three fired heads match the official recording; the reasoning re-draw replaces description with an empty string, and the diff is only one line
  • I can state the division of labor across the three levels of this model cascade: mini extracts, Jev verifies (one request), and only alerts pay for gpt-5.5 reasoning
  • I know why the gate takes the max instead of the average, and why __overall__::judge serves only as a control and never enters the gate
  • I can explain why "parse failure returns {}" is the safe direction: every field is read as missing → verification alerts → the gate escalates
  • I can point out the pareto frontier on the official 100-item chart, say which dot 0.79 / $0.02 and 0.82 / $0.045 each are, and that 0.7 is just one point on the sweep line

Pitfall warnings ​

❌ The gate takes the average, and red flags get diluted ​

any_flag is a max-type gate, and the officials specifically stress "not the average": one confident red flag is enough to escalate, rather than being averaged away by a row of low-scoring heads. In this page's example, description::hallucinated is as high as 0.98 while the lowest head is only 0.04—with the average, this fabricated record would slip into the database despite falling below the line; with the max, it gets caught on the spot. When flag separation is good (wrong fields high, correct fields low), max is the most sensitive gate.

❌ Counting __overall__::judge into the gate as well ​

In the gating code, not qid.startswith("__overall__") is an explicit exclusion. The officials keep this big question as a control to show the gap between "one question for the whole record" and "a per-field question set": it reports 0.90, but cannot say which field is wrong or why. What decides escalation is always the per-field heads—the value of flags lies in "field + cause," so the gate naturally gates on the flags.

❌ Turning on structured output / JSON mode for extraction to "prevent errors" ​

Both rungs use plain-text replies, and the official reasoning is counterintuitive: the errors this cascade is meant to catch are schema-valid. Constrained decoding guarantees the shape is right, but it hides models that "break the schema because they misread the page," leaving verification nothing to work with; errors with a broken shape can be blocked in one line by jsonschema. Handing the semantic verification job to the decoder means letting the sneakiest errors slip through while keeping the most obvious ones.

❌ The verifier is not cheaper than the reasoning model, so the cascade is wasted ​

The officials put it bluntly: the verifier must be far cheaper than the reasoning rung, otherwise the cascade saves nothing. The math of this structure is: every item pays "cheap extraction + one verification," and only alerted items additionally pay for reasoning. If verification costs the same as reasoning, you are effectively paying the reasoning price for every item, and alerted items pay an extra share. On the official 100-item curve, the cascade can push down to about $0.02/item precisely because verification is far cheaper + most items don't alert.

Advanced tips (optional) ​

Do arithmetic on the failure-mode heads. The officials explicitly say each question can be reworded individually and its threshold tuned individually—the question set is just a dictionary, and adding a head is adding one line. The official seven-question battery covers generic failure modes; if your domain has specific trouble spots (units, currencies, languages, timeliness), add heads for them. Before touching the questions, remember the next tip: editing questions does not invalidate the cache.

Editing questions does not invalidate the cache; only deleting the file triggers real runs. The keys of JsonCache contain only the function name and arguments—in the cooksafe documentation's own words: changing the function body does not invalidate existing entries. The question set of verify is assembled inside the function body, so rewording MAIN_QUESTIONS or adding heads leaves the cache key unchanged and old answers keep being served; to make new questions actually hit the API, delete json_cache.json first. Tuning FIRE_T is unaffected—the gate is pure code and never goes through the cache.

Play with the official verification question set for free. The official SDE cascade cookbook attaches a ready-made Playground share link below the verification output—the question set from step 4 comes with the complete state of example.json, so clicking it lets you reword questions and view probability distributions right in the browser. To generate links for your own records, the make_playground_link(state, questions) in the step-4 code is already wired up—the link carries the complete state and the questions, so anyone who gets the link can read them; don't put sensitive data in.

Hook up your own batch pipeline. All three pieces follow "schema + content in, records + flags out": swap the schema for your field table, swap the content for the pages you scraped, and reuse extract / verify / the gate as-is. Once it's running, don't rush to fix a threshold—accumulate a batch of data, draw your own curve with the sweep method from step 6, and pin the working point on your budget line.

Lesson summary ​

✅ Three-level model cascade = cheap extraction + Jev full verification + expensive model as backstop: gpt-5.4-mini plain-text extraction (structured output deliberately off) → Jev per-field verification in one request → any_flag escalate only on alert gpt-5.5 (reasoning_effort="high").

✅ Flags name the field + the cause: in the official example, the fabricated description triggers alerts on three heads (0.98 / 0.97 / 0.86), while the correctly empty field absence_wrong gets only 0.05; the reasoning model throws away the fabricated value, and the cascade pays for reasoning only on flagged items.

✅ Gating takes the max, not the average, and __overall__ serves only as a control: if any per-field head passes FIRE_T, escalate; returning {} on parse failure is the safe direction; this accounting holds only if the verifier is far cheaper than the reasoning rung.

✅ The official 100 curves define the operating point: the cascade frontier sits up and to the left of every single model—about $0.02 per item gets about 0.79, while gpt-5.5-reasoning in a single run at 0.82 costs about $0.045; 0.7 is just one point on the sweep line—run your own sweep for your own task.

Key configuration fields recap:

FieldValue on this pageNotes
MINI / REASONINGgpt-5.4-mini / gpt-5.5 (reasoning_effort="high")Two extraction rungs, plain-text replies; switching models only changes these two lines
TS_MODELjev-1.14.0Verifier model; the official 100-curve round used jev-1.12; live runs lock the version ID, don't use jev-latest
FIRE_T0.7any_flag Alarm line; the officials swept 0→1 to pick the operating point
Failure-mode heads7 + absence_wrong (plus __overall__::judge as a control)Full set for non-empty fields; empty fields only ask absence_wrong; unknown types skipped type_mismatch
Cachejson_cache.jsonLLM + TypeSafe calls all go into the cache; editing prompts doesn't invalidate it, only deleting the file triggers real requests
Requests per itemmini 1 time + verifier 1 time (question groups in parallel) + reasoning added only on alarm 1 timeMost items only pay for the first two rungs

Related tutorials:

Next lesson preview ​

That's the text chain verified—how do you vet scanned documents and OCR readings? Next lesson: Image in practice: reading scanned documents and checking OCR results.

You will learn:

  • Scanned-document reading pipeline: Noul checks whether it's there, Box finds where, Choice reads the value—a date covered by a sticker yields unreadable, not a guessed year
  • Checking OCR-extracted values: scan the full page once per field, then take a closer look at the cropped image; below 0.25 escalate to a human, and draw review boxes on the original image
  • The same "calibrated probability + thresholds in code" gating philosophy as this page—only the verification target shifts from text extraction to image readings
gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.