Jev 1.13 Pitfall Handbook: 9 Failure Modes and Alternatives
There is one particularly special page in the official documentation: the jaggedness documentation—TypeSafe's own compiled "list of Jev 1.13 known flaws". It opens with the line Jev isn't perfect, then lists 9 failure modes, each with an alternative. This lesson brings that entire document over: a quick-reference table, item-by-item breakdown, and official copyable code, taking about 15 minutes in total.
This lesson is fully standalone and can be read without touching a browser—pure API users can jump straight from the Advanced Handbook to this lesson without missing anything. The documentation is marked as applying to jev-1.13 (last reviewed 2026-10-02), and the officials also say many of these will be fixed in later versions; but the questioning habits of "arithmetic in code, date components extracted, large state filtered first" are worth developing no matter which version you use.
What you'll be able to do after this lesson
- Recite the names of the official 9 failure modes, and for each state one alternative (self-check against the quick-reference table in this lesson's summary)
- Turn "make Jev count" questions into the official pattern: code iterates over candidates with one Noul per item, and the count is added up in code
- Change date comparison into "model extracts components + code assembles": a Choice enum of 12 months, 31 days, and bounded years, with a "not stated" escape hatch
- Slim down large state: send only the fields the question needs; if you can't filter, use Noul as a relevance gate
- Run all three checks before launch: option reordering consistency, boundary cases, and alignment of instructions with criteria
Your current predicament
- Answers are "right nine out of ten": the same kind of input occasionally goes down a weird branch, you can't find a pattern, and you don't dare launch
- It's clearly a judgment question, but the model acts like it's counting or doing arithmetic—the numbers are never right, and rewording doesn't help
- You want Jev to output a summary, so you've been chaining Choices forever—slow and bad—and you don't know whether the method is wrong or the model is incapable
When to use this technique
When you need to:
- Avoid Jev 1.13's known pitfalls in advance when writing questions, instead of tracing back case by case after launch
- Rewrite a "wrong answer" question into a robust one—counting, dates, large state, and wording are the four high-frequency causes
- Decide who a task belongs to: Jev's judgment, code's arithmetic, or a generation model's text
And you don't want to:
- Learn by stepping into all 9 pitfalls one by one—the officials have already stepped in them for you and written documentation
- Get schooled by adversarial input after launch: vague criteria, untested boundaries, and one dirty data feed from users makes it all fall apart
🎒 Preparation before starting
- Completed Confidence: knowing that confidence is "a shape summary of the answer's own probability distribution"—this lesson's pitfall reminders rely on this premise
- Have used the three basic question types: Choice, Score, Noul
- (Optional) You have a Jev app in development—this lesson's steps are "revise your own problem", so you'll gain the most with a real problem
- (Optional) If you want to run the official counting sample: a Python ≥ 3.10 environment with typesafe-sdk installed (Installation Tutorial)
Core idea
First look at the official self-assessment (the jaggedness document's opening): jev-1.13 is fast, calibrated, and good at common-sense judgment, performing best on System One tasks. It also has three categories of inherent weaknesses:
| Weakness | Meaning | Corresponding failure mode |
|---|---|---|
| Multiple layers of indirection | Tasks requiring extra levels of indirection are a struggle | #4 Indirect reference |
| Literal reading | Understanding can be quite literal | #1 Literal reading |
| Numerical precision | Tasks that need numerical precision will struggle | #2 Math and numbers, #3 Date and time comparison |
9 failure modes are essentially the same thing: handing System Two (slow thinking) work to System One (fast judgment). So all the alternative approaches can be compressed into one mantra:
Leave judgment to Jev, arithmetic to code, generation to generative models.
At the end of the documentation, the officials also provided a four-item avoid list, corresponding one by one:
- ❌ Asking the model about things code can compute exactly
- ❌ Hiding several judgments inside one question
- ❌ System Two tasks: more layers of indirection
- ❌ Stuffing more context into state than the question needs—the official wording: Jev suffers from context rot, and irrelevant material directly costs you accuracy
💡 The attitude of this document itself
The officials are not hiding anything: it opens with Jev isn't perfect, and ends with a Discord entry point to collect new failure modes. Documenting known problems, each with an alternative approach—this is also the attitude you should take toward your own Jev applications: record the pitfalls, and attach the workarounds.
Follow along
Step 1: Use the official 9-failure-mode table to run a Jev pitfall checkup
Why: know the name of the disease before changing the question. The officials have organized the known pitfalls into 9 failure modes, each with a one-line alternative approach—running your existing questions through this table first saves a lot of back-and-forth compared to troubleshooting after errors occur.
The official full table (symptoms and alternative approaches are all from the jaggedness document):
| # | Failure mode | Typical symptoms | Official alternative approach |
|---|---|---|---|
| 1 | Literal reading | Qualifiers, negations, and implicit conditions are all read literally; it answers the question you wrote, not the one you meant | Put exact conditions into instructions and edge cases into criteria; if unavoidable, split into two literal questions and combine in code |
| 2 | Math and Numbers | Counting is unreliable, with error growing with scale; hex / RGB works worse than English names | Put all arithmetic in code; counting = code iterating over candidates + one question per item |
| 3 | Date and time comparison | Dates read as text; ordering, intervals, and window judgments are unreliable | Extract components, compare in code |
| 4 | Indirection | Double negation, attributes of attributes; the more hops, the less accurate | Reduce hops, point directly to the relevant state by name |
| 5 | Large state full of irrelevant details | Irrelevant details are distractors; when errors occur, it's hard to pinpoint which part of the input is at fault | Filter first, send only what the question needs |
| 6 | Adversarial content | Injected instructions, misleading framing, and self-justifying text can shift the answer | Write precise questions, test edge cases before going live |
| 7 | Contradictory instructions and criteria | When instructions and criteria ask for different things, the model gets confused | Treat criteria as an extension of instructions, align the two |
| 8 | Choice option order | Order affects the answer, with a bias toward options listed first | Reorder the options, recheck that answers stay consistent |
| 9 | Needs generation | Chained Choice forced to generate: poor results and very slow | Switch to a generative model; convert bounded extraction to Choice |
You should see: each of your draft questions falls into one of three categories—usable as is; hitting a certain row and needing a rewrite; or something that shouldn't be asked of the model at all (code can compute it exactly). The following steps only deal with the second category.
- My question found a matching row in the table (or I confirmed no changes are needed)
Step 2: Move arithmetic back into code—the triple pitfall of counting, numerical representation, and Score interpolation
Why: the official wording is Jev is not a calculator—any mathematical logic is strongly recommended to go in code; Jev performs better on semantic questions than on math questions. Failure mode #2 has three sub-pitfalls; let's look at each one.
Pitfall A: counting. jev-1.13 counting is unreliable—character counts within words, occurrences of a term in a paragraph, and the number of entries in a long list all count. The official description is vivid: the model recognizes "the shape of the answer," not a tally one by one, and the error grows with the scale of the objects being counted. Before starting, ask yourself: why does this count need the model? If the units are something a regex or parser can find, the count belongs to code, and the model can't help.
The official example (from the jaggedness document, copyable verbatim)—to count how many fruits are in a list, change it to "one Noul per item + code sums them up":
from typesafe_sdk import Noul, TypeSafeClient
client = TypeSafeClient(model="jev-1.13")
YES = 0.5 # up to you on what you want the threshold to be, depends on your usecase.
items = ["typesafe", "apple", "california", "banana", "likes", "calibration", "orange", "vertex"]
result = client.system_one(
{"items": items},
{
f"item_{i}": Noul(instructions=f"Is `items[{i}]` the name of a fruit?")
for i in range(len(items))
},
)
count = sum(result.nouls[f"item_{i}"].noul > YES for i in range(len(items)))The model only made 8 "is it a fruit" judgments and did not count anything; count is what added up the 8 answers in code. Based on the list contents, apple, banana, and orange should all be judged as fruits, and you should see count = 3.
Pitfall B: numerical representation. Semantic representation beats numerical representation: for color questions, hex values work worse than English color names; judging whether two values are close given RGB triplets or hex values—unreliable. Likewise, questions about high-level programming languages perform better than assembly or binary-encoded instructions. Alternative: do the conversion in code, pass in computed numbers or named buckets, and save the model for the part that genuinely needs judgment—for example, "does this color read like a warning."
Pitfall C: Score interpolation. Don't use Score outputs (expectations, probabilities) to compute precise values between two levels. Expectations can be used to check "whether something passes a threshold," but the score levels of jev-1.13 are weakly numerically calibrated—reconstructing precise numbers by interpolating between the nearest two levels is not something it can help with.
- Search your code once for questions that "ask the model to output counts / differences / precise values," and change them all to code computation or Choice enumeration
Step 3: Don't compare dates—the model extracts components, code assembles
Why: jev-1.13 reads dates as text, not as ordered quantities. "Which of two dates comes first, how far apart they are, whether they fall within a window" are all unreliable; mixed formats, relative references (next Thursday), and domain boundaries (quarters, settlement windows, accrual periods) make it worse.
The official division-of-labor principle in one sentence: extraction is judgment, so it goes to the model; arithmetic is not, so it stays in code.
Every component of a date is a small closed set: 12 months, 31 days, a bounded range of years. This turns extraction into a Choice over enumerated options rather than free-form parsing; it also allows an explicit "not stated" option—missing components are reported, not guessed. The code assembles the components into a real date, and everything after that—sorting, durations, offsets, weekdays—belongs entirely to the code.
The 7 Choice questions from the official date extraction cookbook (all sent in a single request):
| Question | What it determines | Escape hatch |
|---|---|---|
| mode | absolute (a calendar date with a month written) / relative (relative to today) / none (no date stated) | none |
| month | the month of an absolute date (12 options) | none: not stated in the text |
| day | the day of the month (1–31 one option each) | none |
| year | which year (1900–2050 one option per year) | none: no year stated, the code fills it in; out_of_range: stated but not in the list, the code flags it and does not guess |
| day_anchor | today / tomorrow / day_after / weekday (a named day of the week) | none |
| weekday | day of the week (7 options) | none |
| week_offset | next (the next calendar week) / current (this week) / none (a bare weekday) | — |
The assembled confidence is "the lowest among the components actually used," and anything below 0.60 (the cookbook's REVIEW_BELOW) goes to a human. The full runnable version (including relative dates and confidence gating) is in Value extraction: dates, emails, phone numbers, amounts.
What you should see: a single request returns 7 component answers; the code assembles a date object; "next Thursday" is resolved by code convention (a bare weekday = the next one from today, next = the next calendar week, current = this week)—the model never compared any dates at any point.
- There are no questions like "which date is earlier / how many days apart / is it within the window" in the code; all of them were replaced with component extraction + code comparison
Step 4: Split literal interpretation into two questions, reduce hops for indirect reference
Why: Failure modes #1 and #4 both stem from how the question is worded. Literal interpretation: jev-1.13 answers the question you wrote, not the question you meant—qualifiers, negations, and implicit conditions are all read literally; humans read intent into it, it does not. Indirect reference: questions with double negatives or complex indirection lose reliability, and attribute-stacking plus multi-hop reasoning both burn accuracy.
Three alternatives for literal interpretation:
- Write the exact conditions into
instructions, and be specific - Put edge cases into
criteria - The official golden quote: when you look at a wrong answer and find yourself explaining "what I really wanted to ask was...", that explanation is the missing other half of the instructions
Where an explanation truly cannot be avoided, split it into two literal questions and combine them in code. For example, "keep comments that are relevant and friendly" hides two judgments in one sentence:
from typesafe_sdk import Noul, TypeSafeClient
client = TypeSafeClient(model="jev-1.13") # 本页示例固定用 jaggedness 文档适用的版本
result = client.system_one(
{"comment": "这个服务真好用,就是贵了点"},
{
"is_on_topic": Noul(instructions="Is `comment` about the product itself?"),
"is_friendly": Noul(instructions="Is `comment` friendly in tone?"),
},
)
# 组合逻辑在代码里:改阈值、改逻辑,都不用动 prompt
keep = result.nouls["is_on_topic"].noul > 0.7 and result.nouls["is_friendly"].noul > 0.7The alternative for indirect reference: write instructions as directly as possible; wherever possible by name point out the relevant parts of the state (use backtick dot-path references, e.g. ticket.messages[0].text), and don't make the model do the hopping itself.
What you should see: after the rewrite, each question contains only one literal judgment; connectors like "moreover" or "not not" disappear from the instructions; all the combination logic lives in the code.
- Each set of instructions contains only one judgment; negation words appear at most once
Step 5: Slim down large state—code filters first, Noul as the relevance gate
Why: Failure mode #5. The more content in the state that is irrelevant to the decision, the lower the accuracy—irrelevant details are distractors, and a large state makes it hard to pinpoint "which piece of input caused the wrong answer." The official wording: Jev suffers from context rot, and irrelevant material in the state directly costs you accuracy.
The alternative has two layers:
- Code retrieves and filters first, sending only the fields the question needs
- When state-level filtering isn't possible, use Noul to ask about relevance. The official RAG cookbook's approach: 4 Nouls per candidate passage—is it relevant, does it contain usable evidence, does it contradict the query's premise, does it contain injection instructions; anything with
is_relevantbelow 0.45 (the cookbook'srelevant_min) is excluded outright
⚠️ Do the context accounting along the way
The official jaggedness documentation reminds you: state counts once against jev-1.13's two context limits—the per-request 64k (state + all questions) and the per-question 32k (state + a single question). If content overflows, you can enable the long-context request header (TypeSafe-Danger-Zone-Long-Context: true, both limits become 262k), but the official docs state plainly that it is slower and less accurate—the correct order is to filter first, not to turn on the header first.
For the full case study see RAG passage vetting; for a review of the three forms of state and dot-path references, see State.
What you should see: the state sent out contains only the fields usable for that question; the request body shrinks noticeably; answers to the same question stabilize.
- Every question's state has passed the "is this field usable for this question" check
Step 6: Three checks before going live—option reordering, edge cases, instruction alignment
Why: Failure modes #6, #7, #8 cannot be fixed once and for all by rewording; the alternatives the official docs give are all "pre-launch checks." Before facing many users, run through these three checks.
Check one: Choice option reordering. The official docs observed that option order can influence answers, with jev-1.13 favoring options listed earlier. How to verify: reorder the options and ask again—the answer should stay consistent.
from typesafe_sdk import Choice, TypeSafeClient
client = TypeSafeClient(model="jev-1.13")
state = {"ticket": {"subject": "无法登录", "message": "从今天早上开始一直报 401"}}
def urgency(options: dict) -> str:
return client.system_one(
state,
{"urgency": Choice(instructions="How urgent is `ticket`?", criteria=options)},
).answers["urgency"].choice
print(urgency({"low": None, "medium": None, "high": None}))
print(urgency({"high": None, "medium": None, "low": None})) # 重排后再问一遍
# 两次不一致 → 这个问题的答案不可信:选项定义要更互斥、更具体Check two: adversarial content. State is data, and jev-1.13 does not treat it as hostile content by default—injected instructions, deliberately misleading framings, or text defending its own classification can all move the answer (the official docs say this will improve in the future). Alternative: write criteria explicitly; thoroughly test your integration before launch. For a full security-check case study in this direction, see LLM guardrails.
Check three: alignment between instructions and criteria. When the two ask for different things, the model gets confused. The official example: a Noul where true maps to "no" and false maps to "yes" performs worse. Alternative: treat criteria as an extension of instructions, aligning them in clear, precise language; the goal is that an ordinary person can understand at a glance.
You should see: answers are consistent after reordering; edge cases such as empty input, overlong input, and injected input have all been tested; instructions and criteria say the same thing.
- All three checks have been run, and inconsistent option definitions have been rewritten
Checkpoint ✅
- Can recite the names of 9 failure modes and give one alternative practice for each (against the quick reference table in this lesson's summary)
- Counting, date comparison, and exact values are all computed in code—the model only makes judgments
- Every instructions contains only one literal judgment; the combined logic lives in code
- state sends only the fields a question needs, and the budget for the 64k / 32k limits has been accounted for
- Before going live, the three checks have been run: option-reorder consistency, edge cases, and instruction alignment
Pitfall reminders
❌ Making Jev generate text
Symptom: you want Jev to write a summary, write a reply, or output a piece of free text, and you use chained Choice to assemble it character by character.
Cause: failure mode #9. jev-1.13 is not trained to generate text; the official docs explicitly say that forcing chained Choice to generate "works poorly and is very slow".
Correct approach: when the answer space is bounded, turn extraction into a Choice over those options instead of asking for the value itself; if you need free text, use a generation model. The official recommended division of labor for data extraction: regex or a generation model first extracts the candidate options, and jev-1.13 is only responsible for picking the correct one—value extraction is the full implementation of exactly this division of labor. The official wording is blunt: If you really need to generate text... there are other models for that.
❌ Using a confidence threshold as a band-aid
Symptom: answers are occasionally absurd, so you add confidence thresholds everywhere and route anything below the threshold to a human, assuming that makes things stable.
Cause: confidence is "a shape summary of the answer's own probability distribution," not proof of "whether the answer is correct." When it hits pitfalls like literal interpretation, the model will confidently answer the literal question—the distribution is sharp, confidence is high, but the answer is beside the point. A badly written question cannot be saved by a threshold.
Correct approach: first fix the question according to the 9 patterns in this lesson, then use the three tiers of confidence usage for layered automation. Also, the official agent-skill documentation's FAQ has an entry: when you only care about "picking the best," just take the highest-probability option directly—no need to set confidence thresholds everywhere; if you have statistical algorithm needs, use probabilities.
❌ "These pitfalls are all fixed in 1.14, no need to learn them"
Symptom: you see the docs marked as applying to jev-1.13, while jev-latest already points to jev-1.14.0, so you think the whole page can be skipped.
Cause: the officials only said "many will be fixed in later versions," without saying which ones were fixed; the jaggedness document was last reviewed 2026-10-02, earlier than the 1.14 release (2026-10-14), and the official docs site has no 1.14 version of the jaggedness list. Also, jev-1.13.0 is still in service, and officials plan to give 60 days' notice before deprecating it.
Correct approach: build the 9 patterns into your questioning habits—arithmetic goes in code, dates are extracted as components, large state is filtered first; these are version-independent and equally good practice for 1.14. Chinese users should add one more official reminder: CJK content accuracy is lower than English, so test with your own data before going live, paying particular attention to confidence (see the language support section of models and pricing).
Advanced tips (optional)
Turn the quick reference table into a team checklist. The official agent-skill documentation recommends centralizing questions and threshold constants in a single file for easy human review. The 9 failure modes naturally form a question review checklist: before a new question is merged, run it against the quick reference table in the summary. Combine with the TypeSafe Agent Skill to let Claude Code help you rewrite questions—the officials have warned that agents are bad at writing questions, so expect collaborative editing.
Feed back into your Ultrafast goals. These questioning habits also apply to writing goals for browser tasks: literal, specific, with stop conditions—official example goals all end with Stop when…, and "writing goals narrowly" is also an official recommendation (see Ultrafast troubleshooting).
If you discover a new failure mode, report it to the officials. The end of the jaggedness document is a call for submissions: if you find a failure mode that deserves to be listed, report it via Discord.
Lesson summary
✅ Jev's master rule for avoiding pitfalls: leave judgment to Jev, arithmetic to code, and generation to generative models; only send the state that the current problem needs.
✅ Among the 9 failure modes, #2 #3 are "work that belongs in code," #1 #4 are "question wording," #5 is "state organization," #6 #7 #8 rely on "pre-launch checks," and #9 is "time to switch to a generation model".
✅ Remember two official templates: counting = code iterating over candidates + one Noul per item; dates = Choice enumerating components + code assembling them, with a "not specified" escape hatch.
✅ confidence is a summary of distribution shape and cannot save a badly written question—fix the question first, then set thresholds.
✅ These habits are version-independent; for Chinese content, additionally watch out for CJK accuracy and self-test before going live.
Quick reference for the 9 failure modes (symptom → alternative practice):
| # | Failure mode | One-line alternative practice |
|---|---|---|
| 1 | Literal interpretation | Put exact conditions in instructions and boundaries in criteria; split into two questions and combine in code |
| 2 | Math and numbers | Put all arithmetic in code; counting = code iteration + one question per item |
| 3 | Date-time comparison | Extract components (Choice enumerates), compare in code |
| 4 | Indirect reference | Reduce the number of hops; point to state by name |
| 5 | Large state full of irrelevant details | Filter first; if filtering isn't possible, use Noul as a relevance gate |
| 6 | Adversarial content | Write criteria explicitly and test boundaries before going live |
| 7 | Conflicting instructions and criteria | Criteria are an extension of instructions; align the two |
| 8 | Choice option ordering | Reorder the options and re-check answer consistency |
| 9 | Generation needed | Switch to a generation model; convert bounded extraction to Choice |
Next lesson preview
This concludes the advanced handbook. The next lesson, API quick reference: requests, responses, and errors, condenses the request-layer knowledge into a single page. You will learn:
- Request body rules: 32 MiB limit, UTF-8 JSON without BOM, nesting ≤ 64 levels, numbers must be finite
- Status code table: 400 / 401 / 413 / 415 / 422 / 429 / 529 — their meanings and retry strategies
- Compression rules: how to use gzip / zstd, and when 415 occurs
- Field mapping for requests and responses between the SDK and direct curl calls
Related tutorials
- Noul yes/no questions: get the true probability from 0 to 1
- Score rating questions: rate according to the scale you define
- Value extraction: dates, emails, phone numbers, amounts
Appendix: source code references
— click to expand and view the official documentation sources
Last updated: 2026-11-04 (local information source for official docs: sources/jev/docs/, URL = https://docs.typesafe.ai/<本地路径>)
| Content | File path | Key locations |
|---|---|---|
| 9-failure-mode summary table with per-section alternative approaches | docs/model-jaggedness/jev-1.13.md | Opening positioning (fast / calibrated / common-sense judgment; three weaknesses: indirect hierarchy, literal comprehension, numerical precision); failure-mode table; closing four avoid items and the Discord feedback entry point |
| Official counting example (Noul per-item judgments + code summation) | docs/model-jaggedness/jev-1.13.md | Math and Numbers → Counting: 8 items, YES = 0.5, summing result.nouls |
| Numeric representations and Score interpolation | docs/model-jaggedness/jev-1.13.md | Numeric representations (hex vs English names, high-level languages vs assembly); Math using score (expectations used only for threshold checks) |
| Literal comprehension / indirect references / large state / adversarial / conflicting instructions / option ordering / generation | docs/model-jaggedness/jev-1.13.md | The respective same-named subsections; the context limit NOTE in the Large state subsection |
| Complete implementation of date component extraction | docs/cookbooks/date_extraction_cookbook.md | 7 Choices (mode / month / day / year / day_anchor / weekday / week_offset); year 1900–2050 + none / out_of_range; REVIEW_BELOW = 0.60; confidence takes the minimum of the components used; published code pins TYPESAFE_MODEL = "jev-1.12" |
| Noul relevance gate and injection checks | docs/cookbooks/classifying_rag_passages.md | 4 Nouls (is_relevant / contains_answer_evidence / contradicts_query_premise / contains_prompt_injection); relevant_min = 0.45 |
| Context limits and long context | docs/models.md | 64k per-request / 32k per-question; TypeSafe-Danger-Zone-Long-Context: true → 262k, slower and less accurate |
| Version timeline and language support | docs/models.md; blog post typesafe.ai/blog/jev-1-14 (local site/jev-1-14.md) | jev-1.14.0 released 2026-10-14; jev-latest now points to 1.14.0; 1.13 planned to give 60 days' notice before deprecation; CJK accuracy is lower, test it yourself before going live |
| Common issues such as "no need to set confidence thresholds everywhere" | docs/agent-skill.md | The Common issues section |