75-Category Industry Classification: Report the Division When Confidence Is Insufficient

💡 One-sentence summary: Use Jev to build a text classification pipeline with "automatic downgrade when uncertain" — a single Choice assigns to 75 industry groups; confidence ≥ 0.9 reports the industry group, otherwise the parent division is reported with pure code.

Every company that files an annual report (10-K) with the SEC explains what it does in Item 1, "Description of Business." The official cookbook uses this real-world material to pose a classic Jev text classification problem: assign each description to one of 75 industry groups under the Standard Industrial Classification (SIC) — one Choice per file, and the entire category table fits in a single request.

Most files are easy to classify: a regional bank is a regional bank. The hard ones are companies that just sold one of their two business segments, and startups describing an industry they "intend to enter." The model has to pick a group regardless, and the answers for hard cases look exactly like the easy ones. Telling hard from easy is usually where the cost lies: a second model, extra calls, human review.

Jev's Choice answers come with confidence (classification confidence) — a single number that separates "trustworthy answers" from "untrustworthy ones." And SIC labels are naturally hierarchical: above the industry groups sit 10 broader divisions. The official approach: when the model is unsure about the industry group, report its parent division — the division is obtained by looking up the group number, with zero second calls.

The official test on 60 annual reports: 0.9's confidence threshold splits the files exactly in half. The confident half is 90% correct; the unconfident half, if forced to report an industry group, is only 40% correct — after reporting one level up to the division, the usable rate rises to 70%. This lesson takes about 15 minutes and walks you through fully reproducing this gating with officially public data (the official cookbook rates it as Beginner).

What You'll Be Able to Do

• Use a single Choice question with 75 options to perform Jev text classification on company descriptions, obtaining the full probability distribution and confidence
• Write gating code for "confidence ≥ 0.9 reports the industry group, below that reports the parent division" — done with a lookup table, zero second API calls
• Reproduce the SEC annual report classification experiment with officially public data: 60 annual reports, forcing the industry group gives 39/60 correct → with gating, 48/60 usable
• Apply "escalate the report when confidence is insufficient" to your own category tree: works for AI industry classification, e-commerce categories, and ticketing systems alike

Why Use Jev Instead of Ordinary LLM Text Classification?

Capability	Ordinary LLM Classification	Jev
Uncertainty	Ask "are you sure?" and get a paragraph of prose; asking it to write probabilities as text adds yet another parsing failure point	Every answer comes with confidence (0–1), computed from the probability distribution — just set a threshold
Option Control	Freely invents category names; vocabularies like "loan brokerage" vs. "lending intermediary" drift daily	The answer is always one of the 75 options, where choice is the highest-probability option
Large Option Lists	75 options plus descriptions crammed into the prompt, and the output still needs validation against the table	A single Choice supports up to 255 options, reliable up to about 240 — 75 is well within the safe zone
Probability Calibration	Optimized for text generation; self-reported probabilities carry no calibration guarantee	RLCD calibration training aligns probabilities with real outcomes
Fallback for Hard Cases	Forces a wrong label, or makes another model call to ask "are you sure?"	The division is derived from the group number via a lookup table; escalation is a pure code mapping, zero second calls

One honest note: confidence gating lowers the proportion of "mistaking wrong labels for true ones," but does not eliminate errors — in the official test, even the confident half had 3/30 wrong. For the source and formula details of the calibration, see Confidence.

Your Current Predicament

• "I asked an LLM to sort 200 clients into 75 industries, and it answered every single one with absolute certainty — the misclassified ones look exactly like the correctly classified ones, so spot-checking is pure luck"
• "I wanted to add human review, but reviewing all 60 items costs more than the classification itself; and when I ask the model 'are you sure?', it always says yes"
• "The categories clearly have a hierarchy — when it can't reach the finest level, a coarse answer beats a wrong one — but the model just picks one anyway and never says 'I can only go down to this level'"

When to Use This Technique

When you need:

• Labels that are inherently hierarchical (industry classifications, e-commerce categories, organizational structures), where the coarse level is still usable when the fine level is out of reach
• Every piece of data has a destination—neither allowing "don't know" to be dropped outright, nor wanting everything routed to manual review
• Knowing which answers can be trusted automatically and which need review, so the review budget is spent where it matters

And you don't want to:

• Sending another request or calling another model just to ask "is the model sure"
• Parsing category names and probabilities out of the text the model spits out
• Forcing a fine-grained label when the model isn't confident, and downstream treating the wrong fine-grained label as truth

🎒 Preparation before starting

• completed Choice: knowing that criteria is the options map, probabilities is the distribution over all options, and the answer is always one of the options
• completed Confidence: knowing that confidence is computed from the probability distribution, the three-tier usage, and that thresholds scale with risk
• ran through the first request: .venv has typesafe-sdk installed, and the environment variable TYPESAFE_API_KEY is set

You don't need to create the data yourself: the officials have published the SIC industry table and 60 real annual reports in the typesafe-public-examples repository; step 1 walks you through downloading them.

Follow along

Step 1: Build the "industry group → division" mapping table (download the official SIC data)

Why: the reporting logic needs a "fine level → coarse level" mapping table. The SIC codes are themselves hierarchical: the first two digits of a four-digit industry code form the industry group (major group, 75 in total), and fixed ranges of industry groups fall into 10 divisions—this layer of mapping is derived purely in code from the coding rules, with no model involved.

Download the two official data files in your project directory (verified accessible locally on 2026-11-03):

curl -sL -o sic_codes.tsv "https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks/classification_using_confidence/sic_codes.tsv"
curl -sL -o filings.jsonl "https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks/classification_using_confidence/filings.jsonl"

Create classify_sic.py and paste in the mapping part:

# classify_sic.py —— 75 类行业分类：置信度不够就上报大类
# 依赖：pip install typesafe-sdk（Python 3.10+），环境变量里已有 TYPESAFE_API_KEY
from collections import defaultdict
from pathlib import Path

# 十个大类：SIC 行业组编码的固定区间（官方 cookbook 原表）
DIVISIONS = [
    (1, 9, "agriculture, forestry and fishing"),
    (10, 14, "mining"),
    (15, 17, "construction"),
    (20, 39, "manufacturing"),
    (40, 49, "transportation, communications and utilities"),
    (50, 51, "wholesale trade"),
    (52, 59, "retail trade"),
    (60, 67, "finance, insurance and real estate"),
    (70, 89, "services"),
    (91, 99, "public administration"),
]

# 444 个四位行业码 -> 75 个行业组（前两位）-> 10 个大类，全程零模型参与
INDUSTRIES: dict[str, str] = {}
for line in Path("sic_codes.tsv").read_text().splitlines()[1:]:
    code, _office, title = line.split("\t")
    INDUSTRIES[code] = title.lower()

GROUPS: dict[str, list[str]] = defaultdict(list)
for code in sorted(INDUSTRIES):
    GROUPS[code[:2]].append(code)


def division(group: str) -> str:
    """行业组组号 -> 所属大类，纯查表。"""
    number = int(group)
    return next(name for low, high, name in DIVISIONS if low <= number <= high)


print(f"{len(INDUSTRIES)} industries -> {len(GROUPS)} major groups -> {len(DIVISIONS)} divisions")
print(f"  group 35 = {division('35')} / {', '.join(INDUSTRIES[c] for c in GROUPS['35'][:3])} ...")

Run:

.venv/bin/python classify_sic.py

You should see (verified locally with the official data on 2026-11-03):

444 industries -> 75 major groups -> 10 divisions
  group 35 = manufacturing / engines & turbines, farm machinery & equipment, lawn & garden tractors & home lawn & gardens equip ...

• I got 444 → 75 → 10, with group 35 falling in manufacturing

Step 2: Write the 75 industry groups as a single Choice question

Why: names alone aren't enough for the options—of the 75 industry groups, only 42 carry an "umbrella name" in the SEC list (like group 20's food and kindred products); the other 33 have only a group number. The official approach is to describe each option using the industry details within the group—these details are exactly what someone reading annual reports compares against. Each group lists at most 8 details: the official notes say that's enough to characterize the group without piling up a wall of text.

Continuing from the code above, paste:

from typesafe_sdk import Choice

MAX_NAMED = 8  # 每组列出的细目数；官方值


def describe(group: str) -> str:
    """选项描述 = 伞名 + 组内细目；没有伞名的组只列细目。"""
    umbrella = INDUSTRIES.get(f"{group}00")
    inside = [INDUSTRIES[c] for c in GROUPS[group] if c != f"{group}00"][:MAX_NAMED]
    listed = "; ".join(inside)
    return (
        f"{umbrella} — includes: {listed}"
        if umbrella and listed
        else (umbrella or listed)
    )


print(f"group 20: {describe('20')[:150]}")
print(f"\ngroup 65: {describe('65')[:150]}")

# 问题原文来自官方 cookbook；完整问题写在 instructions 里
QUESTION = (
    "Which broad industry does this company operate in? Judge the company's own operations "
    "as this filing describes them."
)


def questions() -> dict:
    return {
        "group": Choice(
            instructions=QUESTION,
            criteria={group: describe(group) for group in sorted(GROUPS)},
        )
    }

Run again, and you should see (the new output below matches the officially recorded output; the prints from previous steps remain unchanged):

group 20: food and kindred products — includes: meat packing plants; sausages & other prepared meat products; poultry slaughtering and processing; dairy product

group 65: real estate — includes: real estate operators (no developers) & lessors; operators of nonresidential buildings; operators of apartment buildings; less

Aren't 75 options too many? No. The Choice lesson covered that a question supports at most 255 options; the official cookbook's experience is that roughly up to 240 remains reliable, and 75 is well within the safe zone—the entire category table fits in a single request.

💡 This is the unique value of Jev

The 75 options, together with their descriptions, are handed to the model in a structured way inside criteria, and the answer can only be one of these 75 group numbers—no vocabulary drift, no output parsing. Even more crucial is the next step: every answer comes with its own confidence, so you never need to send another request to ask "is it sure".

• I can point out which parts of the option description for group 20 are the "umbrella name" and which are the "details"

Step 3: One annual report, one request—get Jev text classification working

Why: filings.jsonl contains 60 real 10-K annual reports, each keeping only Item 1 "business description"—the part where the company describes its own business, the only part the industry code cares about, averaging 1438 words. One request per file, asking one Choice question: the answer returns choice (the winning group number), probabilities (the weights of each of the 75 options), and confidence (a single number for how concentrated the distribution is).

Steps 4 and 5 will repeatedly fetch answers for the same files, so first add a small cache storing results in my_cache.json—rerunning the script won't bill twice; delete it if you want to force a rerun (the official notebook uses the JsonCache from cooksafe to do the same thing):

import json

from typesafe_sdk import TypeSafeClient

client = TypeSafeClient(model="jev-latest")

FILINGS = [json.loads(line) for line in Path("filings.jsonl").read_text().splitlines()]
_cache: dict[str, dict] = {}
if Path("my_cache.json").exists():
    _cache.update(json.loads(Path("my_cache.json").read_text()))


def ask(filing_id: str, text: str) -> dict:
    if filing_id not in _cache:
        response = client.system_one(state=text, questions=questions())
        answer = response.answers["group"]
        _cache[filing_id] = {
            "group": answer.choice,
            "confidence": answer.confidence,
            "probabilities": dict(answer.probabilities),
        }
        Path("my_cache.json").write_text(json.dumps(_cache))
    return _cache[filing_id]


# 拿一份好分的文件先跑通：一家人寿保险公司
example = next(f for f in FILINGS if f["id"] == "33416_1998")
print(f"{len(FILINGS)} filings, {sum(f['words'] for f in FILINGS) // len(FILINGS)} words on average")
print(f"\n{example['id']} (filed {example['year']}, accession {example['accession']}):")
print(f"  {example['text'][:230]}...")
print(f"  filer's code: {example['sic']} {INDUSTRIES[example['sic']]}")

answer = ask(example["id"], example["text"])
print(f"\nmodel's answer: group {answer['group']} ({describe(answer['group']).split(' — ')[0][:46]}), conf {answer['confidence']:.2f}")

Run:

.venv/bin/python classify_sic.py

You should see (new output below; inspection lines verified locally on 2026-11-03, and the last line is the officially recorded value—jev-1.12 measured this file at conf 1.00 → group 63. If you run with jev-latest, the label for a file with such a clear business description will most likely stay the same, though the confidence value may differ slightly):

60 filings, 1438 words on average

33416_1998 (filed 1998, accession 0000033416-99-000001):
  ITEM 1. BUSINESS Erie Family Life Insurance Company (hereinafter referred to as "The Company", the "Company" or "Erie Family Life") was incorporated in the Commonwealth of Pennsylvania on May 23, 1967 and commenced business on Sep...
  filer's code: 6311 life insurance

model's answer: group 63 (life insurance; accident & health insurance; h), conf 1.00

A company that describes its business clearly gets a confidence right at the top. Among the officially recorded 60 answers, confidence ranges from 0.22 to 1.00—a spread-out distribution, which is precisely the precondition for gating to be useful. Cost reminder: running all 60 files = 60 requests, billed by input tokens with free output; see Models and Pricing for current rates.

• A single-file request returned choice, probabilities, and confidence, and the answer was one of my 75 options

Step 4: Add confidence gating: ≥ 0.9 report the industry group, < 0.9 report the division it belongs to

Why: this is the star of this lesson. With confidence ≥ 0.9, the answer is reported at the industry-group level; below 0.9, the same answer is instead reported as the division it belongs to—division(组号) does the lookup, with zero second calls. Every file comes back with a usable label: nothing dropped, no forced fine-grained labels, and no routing everything to manual review.

Continuing, paste:

CONFIDENT = 0.9  # 高于它报行业组组号；低于它上报所属大类（官方 SEC 场景实验值）


def classify(filing: dict) -> dict:
    answer = ask(filing["id"], filing["text"])
    sure = answer["confidence"] >= CONFIDENT
    return {
        "level": "group" if sure else "division",
        "label": answer["group"] if sure else division(answer["group"]),
        "confidence": answer["confidence"],
        "group": answer["group"],
    }


def show(filing: dict) -> None:
    result = classify(filing)
    named = describe(result["group"]).split(" — ")[0][:46]
    print(
        f"  {filing['id']:>13}  conf {result['confidence']:.2f}  -> {result['level']:<8} "
        f"{result['label']:<14} (group {result['group']}: {named})"
    )


results = [(f, classify(f)) for f in FILINGS]  # 60 份各一次请求，缓存防重复
by_conf = sorted(results, key=lambda fr: fr[1]["confidence"], reverse=True)

print("three filings the model was sure about:")
for f, _ in by_conf[:3]:
    show(f)
print("\nthree it was not:")
for f, _ in by_conf[-3:]:
    show(f)

Run:

.venv/bin/python classify_sic.py

You should see (new output below, officially recorded, jev-1.12, 2026-08-12; your own top and bottom lists and values will vary with version and data, but the pattern of "easy-to-separate files get high confidence, hard ones get low confidence" holds):

three filings the model was sure about:
    310158_1996  conf 1.00  -> group    28             (group 28: chemicals & allied products)
     33416_1998  conf 1.00  -> group    63             (group 63: life insurance; accident & health insurance; h)
    352541_1996  conf 1.00  -> group    49             (group 49: electric, gas & sanitary services)

three it was not:
   1372167_2013  conf 0.22  -> division manufacturing  (group 38: search, detection, navagation, guidance, aeron)
   1398633_2009  conf 0.23  -> division wholesale trade (group 50: wholesale-durable goods)
     46653_1999  conf 0.29  -> division services       (group 87: services-engineering, accounting, research, ma)

Look at the three rows at each end: confidence lines up perfectly with "how hard this file is to classify":

• The three 1.00: a pharmaceutical maker, a life insurance company, and a utility—all holding companies on paper, but each one's annual report names a dominant business.
• The three at the bottom: two are development-stage companies describing businesses they "intend to do" (one calls itself "intends to operate as a software developer," the other "organized to enter into the computer security software industry"); the third originally had two business segments and sold one just weeks before filing.
• So the three of them come back not with fine-level group codes, but with the three broad category labels: manufacturing / wholesale trade / services.

💡 Note on Chinese-language content

Official language support note: English is the primary training language; other languages, including Chinese, work but with lower accuracy. Before using Chinese company profiles for AI industry classification, run a batch on your own data first, focusing on the confidence distribution, and then set your threshold—this is the first step emphasized throughout this section.

• In the first and last three rows each, report group when confident and division when not (in the official recording, the confidence of the bottom three was all below 0.3)

Step 5: Quantify what "reporting the broad category" buys you

Why: Whether the gating is worth it must be judged by the numbers. The official work scored 60 recorded answers against the codes self-reported by filers, under two strategies: Strategy A forces reporting an industry group every time; Strategy B reports the broad category when not confident (confidence < 0.9).

Continuing, paste:

def correct(filing: dict, result: dict) -> bool:
    gold_group = filing["sic"][:2]
    if result["level"] == "group":
        return result["label"] == gold_group
    return result["label"] == division(gold_group)


sure = [(f, r) for f, r in results if r["level"] == "group"]
unsure = [(f, r) for f, r in results if r["level"] == "division"]

forced = sum(r["group"] == f["sic"][:2] for f, r in results)
broadened = sum(correct(f, r) for f, r in results)

print(f"forced to name a group every time      {forced}/{len(results)} right")
print(
    f"  of those, the {len(sure)} it was sure about  "
    f"{sum(r['group'] == f['sic'][:2] for f, r in sure)}/{len(sure)} right"
)
print(
    f"  and the {len(unsure)} it was not           "
    f"{sum(r['group'] == f['sic'][:2] for f, r in unsure)}/{len(unsure)} right"
)
print(f"\nletting it answer coarsely when unsure  {broadened}/{len(results)} useful answers")

After running ityou should see (new output below, official recording, jev-1.12):

forced to name a group every time      39/60 right
  of those, the 30 it was sure about  27/30 right
  and the 30 it was not           12/30 right

letting it answer coarsely when unsure  48/60 useful answers

Broken out into a table for clarity (percentages converted from the official four sets of numbers):

Strategy	Confident 30 filings	Not confident 30 filings	Total 60 filings
Always report industry group	27/30 correct (90%)	12/30 correct (40%)	39/60 (65%)
Report broad category when not confident	27/30 correct (90%)	21/30 usable (70%)	48/60 (80%)

Three ways to read it:

• The confident half: reporting the industry group is 90% correct—these can be trusted automatically with peace of mind.
• The not-confident half: forcing an industry group gets only 40% correct; after reporting the top-level broad category, 70% is usable. A coarser level is naturally easier to get right—the manufacturing broad category alone covers 20–39 twenty industry groups, so as long as the winner lands in the correct range, the label after table lookup is right.
• The cost: the labels for those 30 filings become coarser. In the official words: if the broad category is too coarse for your application to act on, this branch is where you hand it off to a human.

Don't forget the reconciliation: the confident half is also wrong 3/30 of the time—gating reduces the proportion of "mistaking wrong labels for true," it doesn't eliminate errors.

• I can state the meaning of each of the four numbers: 39/60, 27/30, 12/30, 48/60

📋 Magic Prompt

The "Prompt" on this page is a question template for the API—copy it, replace the 【】 placeholders (those with ? are optional), and it's ready to use.

📝 Two-level classification template with confidence gating (any category tree)

Expected effect: turn any hierarchical label system into a classifier that "reports the fine level when certain enough, the coarse level when not"—one request, zero second calls.

from typesafe_sdk import Choice

# 必填项：【】里换成你的类目树
CONFIDENT = 【置信度门槛：官方 SEC 场景用 0.9；从保守值起步，用自己的数据调】
FINE_LABELS = {
    "【细级标签 A】": "【什么内容算 A——写成可对照的情境描述】",
    "【细级标签 B】": "【什么内容算 B】",
    # 细级标签总数 ≤ 255（硬上限），约 240 内可靠
}
COARSE_OF = {
    "【细级标签 A】": "【A 所属的粗级标签】",
    "【细级标签 B】": "【B 所属的粗级标签】",
}

QUESTIONS = {
    "label": Choice(
        instructions="【一句话问题：这条内容最属于哪个标签？只根据给定材料判断】",
        criteria=FINE_LABELS,
    ),
}

# 输出规范：answer.choice / answer.confidence / answer.probabilities
# 门控：confidence >= CONFIDENT 报 answer.choice；否则报 COARSE_OF[answer.choice]
# 约束条件：
# ✅ 粗级映射用纯代码查表——不发第二次请求
# ✅ 每个细级标签写「可对照的情境描述」，光有名字往往不够
# ❌ 不要把「不确定」做成一个选项——门控是代码层的，不是模型标签
# ❌ 单级标签超过约 240 个别硬塞一题——改走逐层分类（见相关教程）

✅ Option description generator (assemble criteria from your own category file)

Expected effect: the official work describes each option as "umbrella name + at most 8 sub-items within the group"—do the same with your category tree, automatically assembling each option's description from the category data.

# 必填项：【】里换成你的类目数据源（伞名和子项列表）
# 选填项：没有伞名的标签 umbrella 传空串?，函数自动只列子项
def describe_label(umbrella: str, children: list[str], max_named: int = 8) -> str:
    """官方 describe() 的通用版：伞名 + 最多 max_named 个子项。"""
    listed = "; ".join(children[:max_named])
    if umbrella and listed:
        return f"{umbrella} — includes: {listed}"
    return umbrella or listed


# 约束条件：
# ✅ 子项挑最有代表性的（官方按编码顺序取前 8 个）
# ✅ 描述统一大小写风格——官方代码把全部行业标题统一转小写
# ❌ 别堆全部子项——官方注释：足够刻画特征即可，不堆文字墙
# ❌ 没有伞名的标签别硬造——官方 describe() 对 33 个无伞名组只列子项

🔧 Threshold calibration script (set CONFIDENT using your own labeled data)

Expected effect: 0.9 is the experimental value for the official SEC scenario on jev-1.12. Change the category, the language, or the model version, and the threshold must be recalibrated—this script outputs two cards per candidate threshold: "auto-trust ratio × accuracy".

# 必填项：【】里换成你的标注样本（挑真实业务里最难判的一批：代表性案例 + 难例 + 反例）
GOLD = [
    ("【文本 1】", "【正确细级标签】"),
    ("【文本 2】", "【正确细级标签】"),
]
THRESHOLDS = [0.5, 0.6, 0.7, 0.8, 0.9, 0.95]

answers = {}
for text, _gold in GOLD:
    response = client.system_one(state=text, questions=QUESTIONS)
    a = response.answers["label"]
    answers[text] = {"choice": a.choice, "confidence": a.confidence}

for t in THRESHOLDS:
    auto = [(gold, answers[text]) for text, gold in GOLD if answers[text]["confidence"] >= t]
    fine = sum(a["choice"] == gold for gold, a in auto)
    coarse = sum(
        COARSE_OF[a["choice"]] == COARSE_OF[gold]
        for gold, a in auto
        if a["choice"] != gold
    )
    print(f"threshold {t:.2f}: 自动采信 {len(auto)}/{len(GOLD)}, 细级正确 {fine}/{len(auto)}, 上报后粗级可用 {fine + coarse}/{len(auto)}")

# 约束条件：
# ✅ 标定样本要覆盖难例——全是好分的样本会把你骗到过低的门槛
# ✅ 门槛和问题定义放同一个文件，方便人审（官方 Agent Skill 建议）
# ❌ 别拿官方 0.9 直接上生产——官方 NOTE：正确阈值取决于你的领域和用例表现
# ❌ 金标先抽检再标定——过期的标签会把门槛带偏（官方实验前先筛样本）

Checkpoint ✅

• classify_sic.py produced the mapping 444 → 75 → 10, with group 35 falling under manufacturing
• All 60 filings ran to completion; both group and division appear in the level column, and the confidence of the not-confident batch is clearly lower
• Single-file answers carry the three fields choice, probabilities, and confidence, and the answer is always one of the 75 options
• I know the meaning of 39/60, 27/30, 12/30, 48/60, and that they come from the official jev-1.12 (2026-08-12) recording
• I know that 0.9 is the experimental threshold for the SEC scenario—changing the category, language, or version requires recalibrating with labeled data

Pitfall warnings

❌ Making "reporting the broad category" a second request

Symptom: when confidence is below the threshold, sending another request asking "which broad category does it belong to?"

Why it's wrong: the broad category is uniquely determined by the group code—division(组号) with a table lookup suffices. A second request costs more money and introduces new uncertainty. In the official words: the broad category is derived from the industry group, so there is no second call.

Correct approach: complete the reporting in pure code within the gating branch. By the same logic, don't make "uncertain" a 76th option for the model to choose—whether to report upward is decided by code based on confidence, not by a model label.

❌ Being lazy with option descriptions, writing only label names

Symptom: in criteria, each option is just a single word like "manufacturing" or "services"; answer quality is poor and confidence is generally low.

Why it's wrong: of the 75 industry groups, 33 have no umbrella name in the SEC list; given only group codes or single words, the model can only guess. The official work describes each option as "umbrella name + at most 8 sub-items within the group"—those sub-items are exactly what people reading annual reports compare against.

Correct approach: following the approach of step 2's describe(), write a "comparable situational description" for each option; both the label name and the description are sent to the model, and the description is what distinguishes the options.

❌ Treating confidence as a guarantee that "the answer is correct"

Symptom: answers with conf 1.00 go straight into the database without spot checks.

Why it's wrong: confidence describes how concentrated the probability distribution is, not the probability that the answer is correct. In the official evaluation, 3 of the confident 30 filings were still wrong (27/30). When option descriptions are vague or the material lacks information, the model can bet wrong very confidently.

Correct approach: treat confidence as a gate for "whether to trust it automatically"; measure question quality separately — for the three directions of investigating low-confidence cases (overlapping options / the question tests more than one thing / insufficient state information), see Confidence.

❌ Copying the 0.9 threshold verbatim while the version drifts with the alias

Symptom: you hard-code 0.9 into your own system and keep using jev-latest, until one day a new version is released and the behavior changes.

Why it's wrong: 0.9 is an experimental value from the official SEC scenario on jev-1.12; the official NOTE explicitly says the correct threshold depends on your domain and use-case performance. Moreover, the jev-latest alias moves with new versions, and the official recommendation is that requests with adjusted thresholds should pin a version ID.

Correct approach: calibrate the threshold with your own annotated data (template 3 of the magic Prompt); pin the version ID in production requests, and response.model lets every answer in the logs be traced to a version. For more Jev failure modes, see the Jev 1.13 pitfall guide.

❌ Not self-checking the gold-standard data makes the measured "accuracy" meaningless

Symptom: you compute the correct rate from a batch of "model answers vs. human labels," the numbers look bad, and you start doubting the model.

Why it's wrong: SEC industry codes are chosen by the filers themselves and can go stale — after a company sells the business a code refers to, the code stays. Before running experiments, the officials filtered the 60 items down to "codes supported by the text" before measuring: what's being tested is this method, not the quality of EDGAR metadata.

Correct approach: spot-check the gold standard before calibration and evaluation — fix expired-label samples or archive them separately; don't let dirty labels contaminate your threshold.

Advanced techniques (optional)

🔧 Tune the gate with alternative signals. Beyond confidence, the officials have listed two alternative metrics: the top probability p_max (its meaning changes with the number of options, so set the threshold per question setup) and the top-to-second ratio p_max / p_second (most handy for questions where two strong candidates compete).

A geometric intuition: with 75 options, confidence and p_max nearly coincide numerically — its value lies in normalizing the option count, so the same 0.9 threshold means the same thing with 3 options and with 75 options. For formulas and trade-offs, see Confidence.

🔧 When a single level can't hold your categories, switch to level-by-level classification. 255 options per question is a hard limit for Choice, reliable within roughly 240. If your category tree exceeds that in a single level, or you need labels at every level, see the previous lesson Hierarchical classification: the officials use Choice probabilities to walk the category tree level by level with parallel beam search. The "flat, ask-once + confidence reporting" approach in this lesson suits scenarios like 75 where a single level suffices — the two routes complement each other.

🔧 Try the official example with zero code. The official cookbook page includes a Playground share link — a complete question with one annual report + 75 options; open it to see the probability distribution and confidence without writing any code. The console Playground is at console.typesafe.ai/playground.

Lesson summary

✅ A single Choice question with 75 options completes Jev text classification: criteria describe each option with "an umbrella name + up to 8 named items"; the answer is always one of the options.

✅ Confidence gating: confidence ≥ 0.9 reports the industry group, below that it reports the parent division — done by table lookup, zero second calls; every piece of data comes back with a usable label.

✅ Official real-world test on 60 SEC annual reports: forced industry-group reporting 39/60 (65%), with gating 48/60 (80%) — the confident 90% are correct, and the unconfident improve from 40% to 70%.

✅ 0.9 is an experimental threshold for the SEC scenario on jev-1.12: change the categories, language, or version, and you must recalibrate with your own annotated data.

Key configuration fields recap:

Field / constant	Value on this page	Description
CONFIDENT	0.9	≥ reports the industry group, < reports the parent division; experimental value from the official SEC scenario
MAX_NAMED	8	Upper limit on the number of named items listed in each option's description
Number of options	75	Hard limit for Choice 255, reliable within roughly 240
Decision input	answer.choice / .confidence / .probabilities	choice i.e., the highest-probability option
division(group)	Table lookup	Industry group number → 10 divisions, pure code

Related tutorials:

• Self-consistency: adding an "uncertain" exit to review systems — the same "escalate/route to humans when uncertain" idea, applied to content moderation
• Hierarchical classification: sorting content to the deepest level of the category tree — the level-by-level approach for when a single level can't hold your categories
• Smart routing: intent classification with confidence gating — an expansion of confidence gating into a full routing system

Next lesson preview

If you ask the same question only once, how do you know whether it's "stable"? Next lesson: Self-consistency: adding an "uncertain" exit to review systems.

You will learn:

• A self-consistency checkup: re-ask the same content 15 times with a fresh uid to find questions that flip labels
• Choice questions with top probability below 0.60 are automatically routed to humans; Noul questions get an uncertainty band at 0.30–0.70
• Official real-world test: after adding the exit, decision consistency rose from 90.8% to 99.2%, at the cost of 25.8% routed to humans
• Relation to this lesson: this lesson uses the confidence of a single answer to decide "report coarse or fine"; the next lesson uses repeated sampling to decide "which questions should never run automatically"