Search and Matching: Reranking, Line-by-Line Search, Entity Alignment
💡 One-sentence summary: Use Jev to add a "rank—locate—adjudicate" layer to search—Jev reranking sends one Noul per pair and reranks by probability, line-by-line search uses Choice to point to the line plus one Noul to check whether an answer exists, and entity alignment lets one Score with three levels make the call. Fast search handles recall, Jev handles putting the right one first.
"Search" is really two different jobs: the first half (fast search like BM25 and embedding similarity) is fast enough to pull a shortlist of the possibly-matching top 30 from thousands of documents; the second half—which of these 30 is the right one—it cannot answer. The measured data from the official Re-ranking cookbook says it best: 100% of the correct segments were pulled into the top-30 by BM25, but only 5% ranked first.
Jev reranking fills in that second half: on the CLERC legal query data, the official approach sends one Noul per "query-candidate" pair, obtains a comparable 0–1 score, and reranks the shortlist by it—top-1 hit rate 5% → 18%, top-10 38% → 62%. This lesson covers all three cases from the Search and matching group in the official cookbook index in one go: besides reranking, there is also line-by-line search (218 lines of the GitHub Terms of Service, locating which line the answer is on in a single request and, along the way, determining whether the document actually contains an answer) and entity alignment (450 candidate pairs from two beer catalogs, determining which two rows refer to the same beer). Three cases, three question-type combinations, but only one division-of-labor principle: whatever can be computed stays in the code; judgments that require reading the text go to Jev.
What You'll Be Able to Do After This Lesson
- Add a reranking layer on top of your existing BM25 / embedding retrieval: one Noul per "query-candidate" pair, reranking the shortlist by the 0–1 probability score—official measurements show top-1 hit rate 5% → 18%, top-10 38% → 62%
- Write
find(query): for a document within 255 lines, a single request returns "how relevant each line is" (Choice probabilities) plus "whether the document contains an answer" (Noul), and it dares to say there is no answer when there isn't one (official measurements show exists dropping as low as 0.14) - Write
route(pair): determine whether two catalog records are the same thing, with the three-level outcome (keep separate / send to human review / merge) decided by the nearest level of one Score—no threshold constants anywhere in the process - Have Noul name the discrepancies for human review: in the same request, ask field by field "is the name the same, is the brewery the same, is the type the same," so the curator knows which field to look at as soon as they open the ticket
Why Use Jev Instead of Relying Only on Fast Search (BM25 / Embedding Similarity)?
Fast search solves "pulling the top 30 from thousands of entries"; it is powerless for the judgment that comes after.
| Capability | Fast Search Only | Jev Reranking and Matching |
|---|---|---|
| Ranking basis | ❌ Based on literal overlap / vector distance—one change of wording and the ranking goes wrong; the correct answer is in the shortlist 100% of the time, but only 5% of the time at rank one | ✅ Reads each pair with "can this candidate answer this question"—official measurements show top-1 5% → 18%, top-10 38% → 62% |
| The "no answer" case | ❌ Retrieval always returns a "most similar" result and forces an answer | ✅ Line-by-line search paired with one exists Noul; when the document has no answer, the score drops to 0.14, explicitly saying "not in this document" |
| What about the middle ground | ❌ Cut once on similarity—where to cut depends entirely on tuning thresholds, and it breaks as soon as the data changes | ✅ Entity alignment uses Score with three semantically meaningful levels (different / related / same), no threshold constants anywhere in the file |
| Can you tell where the disagreement is | ❌ Only a single total score—no way to trace why it's judged "uncertain" | ✅ One Noul per field: name 0.95, brewery 0.94, style 0.35—the disagreement is in the style wording, visible at a glance |
| Form of the answer | ❌ The score is a black box—switch libraries or tokenizers and it drifts | ✅ Typed answers + probability distributions; the sort key is the noul of 0–1, no extra layer of parsing |
In one sentence: fast search handles recall (the right ones must get in), and Jev handles ranking and adjudication (of those that got in, which is right, and whether they're the same thing)—the two stages each do their own job, and whichever breaks can be fixed on its own.
Your current predicament
- "BM25 always fishes the answer into the top 30, but the one ranked first is almost never it—users only look at the first result, so the correct answer further down might as well not exist"
- "The document clearly doesn't contain the answer, yet search still insists on returning the 'most similar' line—following it leads straight to hallucination"
- "Merging two product catalogs with different name conventions and errors in each field—no threshold is right: too high and you miss merges, too low and you merge wrongly—one wrong merge takes half a day to untangle by hand"
When to use this technique
When you need to:
- In large-corpus retrieval, after fast search produces a shortlist, push the right one to first place—the reranking layer for legal retrieval, document Q&A, and enterprise search
- Locate the lines that answer the question in a medium-length document (within 255 lines), while also being able to recognize "this document doesn't say at all"
- Decide whether two records / products / items are the same thing, allowing an intermediate tier of "uncertain, send to human review"—the higher the cost of a wrong merge, the more you should use it
And you don't want to:
- Tune rerank thresholds until dawn—reranking has no threshold; the noul itself is the sort key
- Parse ranking JSON spat out by a generative model—that's making it re-answer every candidate all over again
- Manually split data after wrongly merging entities
🎒 Preparation before starting
- Completed Choice question: line-by-line search treats 218 line IDs as options, with one Choice holding at most 255 options—the source is also on that page
- Completed Score question: entity alignment adjudicated by a three-level scale, where scores can land between two levels (e.g. 1.30)
- Recommended reading Noul yes-no question: in reranking, one Noul per pair—that 0–1 probability is the sort key
- Working environment:
typesafe-sdkinstalled andTYPESAFE_API_KEYin environment variables (Installation and first request). All three official examples have published their answer cachesjson_cache.json, so you can replay the official numbers without setting a Key (step 1 explains how) - A cup of water ☕—this lesson covers three official cases in one go: three ways of asking the same thing
Follow along
The three cases are independent of each other, one per question type; every step's official numbers come from the corresponding cookbook's recording on jev-1.12. First, here are the three question-type combinations in one table, so you have a map in mind before starting:
| Case | Data | Question-type combination | Requests | Output |
|---|---|---|---|---|
| Reranking | CLERC: 3,565 passages of case law, 40 queries | 1 Noul call per pair | 40 × 30 = 1,200 calls | Reranked shortlist (top-1 5%→18%) |
| Line-by-line search | GitHub ToS: 218 lines, 43,980 characters | 1 Choice + 1 Noul, same request | per query 1 time | per-line relevance score + whether an answer exists |
| Entity alignment | two beer catalogs: 450 candidate pairs | 1 Score + 3 Noul, same request | per pair 1 time | three-way outcome: separate / manual / merge |
Step 1: Match them up—three kinds of search problems, three question-type combinations (including environment setup)
Why: the common thread of the three cases is "pushing the truly matching entry to the front"; they differ only in how the question is asked. First see clearly which question type fits which job—every step after that just swaps in different data.
Install dependencies (the official install commands of the three cookbooks merged; cooksafe is the official cookbook companion tool library, whose dependency declaration pins typesafe-sdk at >=0.7.0,<0.8.0—copy it and pin it the same way):
pip install 'typesafe-sdk>=0.7.0,<0.8.0' bm25s datasets matplotlib ipython 'cooksafe>=0.2.0,<0.3.0'To replay the official numbers without a Key: download json_cache.json from the corresponding cookbook directory in the official examples repo typesafe-public-examples and put it in your run directory—cooksafe.JsonCache will directly replay the recorded answers without sending a single request. To run for real, set TYPESAFE_API_KEY and delete the cache file.
- Dependencies installed; I know which question-type combination each of the three cases uses
Step 2: Jev reranking—one Noul per pair on the BM25 shortlist, reordered by probability
Why: BM25's problem is that it can't rank "who is actually cited." Look at the two official numbers together: the correct passage lands in BM25's top-30 100% of the time (recall is fine), but only 5% of the time is it ranked first (ranking is not). The one and only thing reranking does: take these 30 candidates and re-sort them with a more accurate score.
The official two-step approach:
Thousands of candidates
│ Step one: fast search (BM25)—runs over the whole corpus, fast enough
▼
Shortlist of 30 passages per query
│ Step two: reranking—one Noul per query-candidate pair
▼
Reranked shortlist (still the same 30 candidates, different order)Problem definition (a simplified version of the official text):
from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient
question = Noul(
instructions="Is this candidate the cited case?",
criteria=NoulCriteria(
true="The candidate states the specific rule the query cites.",
false="The candidate is only on a similar topic.",
),
)
response = client.system_one(state={...}, questions={"is_cited_source": question})
response.answers["is_cited_source"].noul # -> 0.87Three key points:
- Why Noul instead of a yes/no question: the official words — pure yes/no can't rank 30 candidates; Noul returns a 0–1 probability, and that noul is exactly the score the program sorts on
- criteria pins down both sides:
true= the candidate states the specific rule the query cites;false= merely topically similar — "similar topic" is exactly the root cause of BM25 misranking, and it must be explicitly excluded in the question - the state holds just this one pair:
{"query_excerpt": query, "candidate_passage": candidate}, each request sees only one pair
Reranking is a single line of sorting code (official original):
nouls = {candidate: ask_typesafe(query, candidate) for candidate in shortlist}
reranked = sorted(shortlist, key=lambda c: nouls[c], reverse=True) # highest noul firstIn the official implementation, 40 queries × 30 shortlist passages each = 1,200 independent calls, run concurrently with ThreadPoolExecutor(max_workers=12) — no pair ever sees another, so it's naturally parallelizable. The CLERC corpus is 3,565 passages of US court opinions (170 rows pooled into a shared corpus, 40 rows sampled as queries; each query is an opinion excerpt with its citation removed, and the gold is the passage the citation points to).
You should see (official jev-1.12 recording, fast search vs. adding Jev reranking, the proportion of correct passages reaching the top N across 40 queries):
Top 1: 5% → 18%
Top 5: 15% → 35%
Top 10: 38% → 62%The correct passage moved up at every tier — but note its ceiling: reranking only reorders the 30 passages already in the shortlist; anything fast search didn't pull in can't be recovered (100% recall is what BM25 happened to achieve this time, not something your scenario is guaranteed to have; see pitfall reminder).
💡 This is the unique value of Jev
Reranking sets no threshold at all — it doesn't judge "this pair matches / doesn't match"; it only needs 40 × 30 comparable 0–1 scores, and sorting is left to code. The official docs also note: this demo asks only one question per pair for clarity, while a real application would ask several questions about the same pair in a single call — for the mechanics and measured gains of multiple questions in one request, see parallel questions and speculative fan-out.
- I can explain why the sort key uses Noul rather than a plain yes/no judgment, and where the 1,200 calls come from (40 × 30)
Step 3: line-by-line semantic search — 218 lines of terms in one request, Choice picks the line, Noul checks whether it exists
Why: reranking handles "a pile of candidate passages," while this step handles one specific document: the GitHub Terms of Service (218 lines, 43,980 characters), asking in plain language "who owns the code I upload," with the answer located to a line — and the ability to discover that "the document simply doesn't say." The official three-part structure: first assign an ID to every line, then use Choice to score the line IDs, and finally add a Noul question in the same request to check existence.
First assign an ID to every line and stitch them back into one document — the model uses these IDs to point out its answer (official original):
DOCUMENT_URL = (
"https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks/semantic_find/github-tos.txt"
)
LINES = fetch_document(DOCUMENT_URL).splitlines()
def line_id(i: int) -> str:
return f"L{i:03d}"
DOCUMENT = "\n".join(f"{line_id(i)}| {line}" for i, line in enumerate(LINES))The stitched document looks like this:
L052| You own Your Content. If you post Content you did not create, you are responsible for...
L053| You grant us and other Users the licenses in Sections D.4–D.8. These licenses apply...
L054| 4. License Grant to UsTwo questions (official original): one Choice uses line IDs as options — the option descriptions say None, because the line text is already in the document:
def where_question(query: str) -> Choice:
return Choice(
instructions=f'Which line of the document contains the answer to: "{query}"?',
criteria={line_id(i): None for i in range(len(LINES))},
)The other Noul checks whether an answer exists — it must be paired with its reason: Choice probabilities always sum to 1, so even when the document has no answer, some line always ranks first; Noul's probability doesn't depend on other options and can drop to near 0 when there is no answer:
def exists_question(query: str) -> Noul:
return Noul(
instructions=f'Does any line of the document address or answer: "{query}"?',
criteria=NoulCriteria(
true="At least one line of the document states or directly implies the answer",
false="No line of the document addresses this",
),
)Both questions go into the same request, and the document is sent only once:
@json_cache
def _find(model, state, where, exists) -> dict:
response = client.system_one(
state=state,
questions={"where": where, "exists": exists},
model=model,
)
probabilities = response.answers["where"].probabilities
return {
"exists": response.answers["exists"].noul,
"relevance": [probabilities.get(line_id(i), 0.0) for i in range(len(LINES))],
}exists is judged in three tiers — the official comment says: with an answer, the reading is typically ≥ 0.9; without one, ≤ 0.05; tune the threshold on your own documents before going live:
FOUND, ABSENT = 0.7, 0.35 # present answers typically read >=0.9, absent <=0.05
def verdict(exists: float) -> str:
if exists >= FOUND:
return "answered in this document"
return "not in this document" if exists < ABSENT else "partially addressed"You should see (official jev-1.12 recording, four queries; the first two have answers, the third doesn't, the fourth has only half of one):
218 lines, 43,980 characters
"who owns the code I upload?"
exists 0.98 -> answered in this document
L052 0.95 You own Your Content. If you post Content you did not crea... ← 就是它
"can GitHub kick me off the platform without warning?"
exists 0.97 -> answered in this document
L168 0.97 GitHub has the right to suspend or terminate your access t...
"do I have to take disputes to arbitration?"
exists 0.14 -> not in this document
L205 0.86 Except to the extent applicable law provides otherwise, th...
"can minors use GitHub with parental permission?"
exists 0.46 -> partially addressed
L029 0.90 You must be age 13 or older. While we are thrilled to see...The third query is the most valuable lesson in the whole design: the top-ranked line got a relevance score of 0.86, yet exists was only 0.14 — the most similar line is not the same as the answer. The official punchline: the ranking tells you where to look, and the exists score tells you whether the result counts as an answer. The fourth is half an answer: the age-rule line tops the list at 0.90, but "whether parental consent can change the rules" isn't stated — partially addressed, for a human to follow up.
💡 This is the unique value of Jev
Choice 选项 = 行 ID turns "locate the line" into a multiple-choice question: 218 options are all listed in the criteria, and the model points to L052 instead of reproducing the original text — citations always point to lines that actually exist in the original, so it can't hallucinate an L499. Paired with the Noul exists, it dares to refuse to answer when "the document doesn't say" — something pure similarity retrieval cannot do.
- I can explain what 0.86 and 0.14 each measure in the arbitration query, and why both questions must go in the same request
Step 4: entity alignment — one Score for a three-tier verdict, three Nouls to name the conflicting fields
Why: when merging data sources, a wrong merge is the more expensive kind of error — the official words: once merged incorrectly, every fact about both entities describes the merged entity and all links come along with it, and undoing it requires disentangling which side each fact came from; a missed merge just leaves a duplicate. So the decision needs a third way out: pairs where you dare neither merge nor discard go to a human.
The data is 450 candidate pairs from two beer catalogs in the Magellan dataset (a coarse-screening pass has already picked out the pairs worth a closer look), with four fields per entity: name, brewery, style, abv. The text is kept as-is with no preprocessing — HTML entities not restored, apostrophes split into separate tokens, a few characters mis-decoded — all left in place.
One Score measures the verdict across three tiers, and the three tier descriptions are the entire decision (official original):
LEVELS = [
"They describe two different products.",
"They describe closely related products that may or may not be the same one: "
"a variant, a special edition, or a name that could plausibly refer to either.",
"They describe one and the same product.",
]
OUTCOME = {0: "leave unlinked", 1: "curator queue", 2: "assert sameAs"}
QUESTIONS = {
"link_state": Score(
instructions="How do the two entity descriptions relate as products?",
criteria=LEVELS,
),
"same_name": Noul(instructions="Do the two entities state the same beer name?"),
"same_brewery": Noul(instructions="Are the two entities from the same brewery?"),
"same_style": Noul(instructions="Do the two entities describe the same beer style?"),
}Four questions go into the same request, and the state is just this pair: {"entity_a": ..., "entity_b": ...}. Three details:
- Why Score and not Noul/Choice: the official selection testimony — to attach semantic labels (the three tier descriptions) directly to each outcome, including the middle tier; Noul can only do this indirectly via output thresholds, while Choice throws away the ordinal relationship among the three tiers
- The middle tier deserves careful writing: "variants, special editions, names that could refer to either" — the wording decides which pairs go to the curator and which are left unlinked; this is your "decision logic," and it can be written down before looking at any score
- abv gets no Noul: the official words — comparing two numbers is arithmetic; if you want it, do it in code. Each of the three text fields gets one Noul, serving as the discrepancy breakdown for the curator
The outcome is decided by a single line of "take the nearest tier" code — note there is no threshold constant anywhere in the file (official original: There is no threshold constant anywhere in this file):
def route(score_value: float) -> str:
"""The whole decision rule: the nearest level names the outcome."""
return OUTCOME[min(int(score_value + 0.5), len(LEVELS) - 1)]You should see (official jev-1.12 recording, four sample pairs — one same-product, one unrelated, two landing in the middle tier for different reasons):
c446 score 1.94 confidence 0.92 -> assert sameAs
Thomas Hooker Old Marley Barleywine | Thomas Hooker Brewing Company
Thomas Hooker Old Marley Barleywine | Thomas Hooker Brewing Company
name 0.97 brewery 0.99 style 0.81 ← same product, style written differently (Barley Wine)
c427 score 0.03 confidence 0.95 -> leave unlinked
Frost Quake Bourbon Barrel Aged Barley Wine | Wellington County Brewery
Lompoc Bourbon Barrel Aged Proletariat Red A | Lompoc Brewing
name 0.02 brewery 0.09 style 0.08 ← unrelated
c100 score 1.30 confidence 0.27 -> curator queue
Belle Gueule Rousse | Brasseurs R.J.
Belle Gueule Rousse | Brasseurs RJ
name 0.95 brewery 0.94 style 0.35 ← only the style wording differs, confidence 0.27
c428 score 1.10 confidence 0.77 -> curator queue
Ambleside Amber Ale | Bridge Brewing Company
Bridge Ambleside Amber Ale - Pomegranate & Galena Hops | Bridge Brewing Company
name 0.63 brewery 0.98 style 0.74 ← pomegranate-flavored version from the same breweryRan all 450 pairs (one request per pair, ThreadPoolExecutor(max_workers=6) small pool concurrency — official note: the public endpoint starts rate-limiting at roughly 8 concurrent requests):
assert sameAs 40 ( 8.9%) ← auto-merged
curator queue 50 (11.1%) ← sent to humans
leave unlinked 360 (80.0%) ← kept separate80% of pairs were auto-resolved; about one in five pairs needed a human. Two points worth pausing on: in c100 both name and brewery scored above 0.94, with only style disagreeing at 0.35 — the Noul details tell the curator directly which field to look at; route()'s two cutoff points (0.5 and 1.5) were not tuned — they follow entirely from how you write the three-level descriptions.
- I can state why a wrong merge costs more than a missed merge, what the middle-level description is for, and the share of 450 pairs sent to humans (50, 11.1%)
Step 5: swap in your own data — three replacement points: shortlist, document, fields
Why: across the three cases there are only three replacement points in total, and they are independent of each other — your scenario is most likely a variant of one of them.
- Swap the shortlist source: the fast search in the reranking sequence uses BM25, but the officials explicitly say this is a side issue — reranking only sees the passages that made it into the shortlist; swapping the fast search to embedding retrieval or a full-text index does not affect the second layer. The only thing to change is the 30 entries fed into
shortlist - Swap the document: for line-by-line search, replace
DOCUMENT_URLwith your own contract / policy / clauses (official words: swap out this URL and the rest of the script runs as-is). One request handles a document within 255 lines; for longer ones use the official two-stage approach — a first Choice picks which window the answer is in, and a second ranks within the window - Swap the fields: to migrate entity alignment to product, music, or organization catalogs, you only need to rewrite
QUESTIONSandLEVELS— numeric fields (price, quantity) should not go through Noul, compare them in code; one Noul per text field. Cost scales with the number of candidate pairs, regardless of how large each catalog is (official words)
Two habits for saving money and controlling concurrency during development:
All answers are stored on
json_cache.jsonofJsonCache, so changing code and rerunning costs zero requests; to actually rerun, just delete the cache file — the cache key is built from the function name and arguments, so changing the document / question / candidates automatically missesWhen running many pairs live, use a small thread pool (the officials use 6): rate limits are per organization, and exceeding them returns
429; the SDK retries automatically with exponential backoff, but there's no point in running right at the limitI know which point to swap in my own scenario: shortlist source / document / fields
📋 Magic Prompts
The "Prompts" on this page are question and code templates for the API — copy them, replace the 【】 placeholders (those with ? are optional) and you can hook them up to your own retrieval or catalog. Three sets: rerank scoring, line-by-line locating, alignment adjudication.
📝 Rerank scoring task (Noul version)
Expected effect: any scenario where "a fast search produces a candidate list that needs reranking by relevance" — legal retrieval, document Q&A, the rerank layer of enterprise search — apply this one template per query-candidate pair to get a ranking key.
from typesafe_sdk import Noul, NoulCriteria
# 必填项:state 放【查询原文 + 这一个候选】,true/false 分别写清「真匹配」和「最容易混的假匹配」
rerank_question = Noul(
instructions="【候选段回答得了这个问题吗?一句话,例如:这段能不能是被引的那条判例?】",
criteria=NoulCriteria(
true="【候选陈述了查询引用的具体规则/给出了查询要的答案】",
false="【候选只是在相近话题上——把快搜索最典型的误排写在这里】",
),
)
score = client.system_one(
state={"query": 【查询原文】, "candidate": 【这一个候选】},
questions={"is_match": rerank_question},
).answers["is_match"].noul # 0–1,直接当排序键
# 选填项:真实应用在同一个请求里对同一对多问几道——支持度、原创度、时效各一题,一组分数一起排?
# 输出规范:每对一个 0–1 的 noul;reranked = sorted(shortlist, key=noul, reverse=True)
# 约束条件:
# ✅ 不设匹配阈值——重排序只要可比分数,排序在代码
# ✅ false 描述里点名快搜索的典型误排(如「只是话题相近」)
# ❌ 别指望重排召回快搜索漏掉的候选——短名单外的一个都进不来✅ Line-by-line search, two questions in one (Choice + Noul version)
Expected effect: for a document within 255 lines, "which line has the answer" and "is there an answer at all" come back in a single request — contract clause locating, policy Q&A, ticket knowledge base retrieval.
from typesafe_sdk import Choice, Noul, NoulCriteria
# 必填项:行 ID 先拼进文档("L052| ..."),选项就是这些 ID、说明留 None
questions = {
"where": Choice(
instructions='Which line of the document contains the answer to: "【大白话问题】"?',
criteria=【{行ID: None for 行ID in 全部行}】,
),
"exists": Noul(
instructions='Does any line of the document address or answer: "【同一问题】"?',
criteria=NoulCriteria(
true="At least one line of the document states or directly implies the answer",
false="No line of the document addresses this",
),
),
}
# 选填项:三档判读——exists ≥ 0.7 算有答案、< 0.35 算没有、中间 partially;阈值用自己的文档调?
# 输出规范:每行一个相关分(Choice probabilities)+ 一个 0–1 的 exists;两题同一请求,文档只发一次
# 约束条件:
# ✅ 文档 ≤ 255 行一段搞定;更长先挑窗口、再在窗口内排行(第二道 Choice)
# ✅ exists 必配——Choice 恒有第一名,没答案时它照样硬指一行
# ❌ 别丢掉行 ID 标注这一步——引用必须指回真实存在的行🔧 Entity alignment, four questions in one (Score + 3 Noul version)
Expected effect: merging catalogs / deduplication / ingestion reconciliation — deciding whether two records are the same thing; especially worth using where a wrong merge is costly.
from typesafe_sdk import Noul, Score
# 必填项:三档从左到右由「敢扔」到「敢并」,中间档写清哪类情况该送人工
LEVELS = [
"【两条记录描述两个不同的东西】",
"【描述紧密相关但不一定是同一款:变体 / 特别版 / 可能指两者任一的名字】",
"【描述同一个东西】",
]
QUESTIONS = {
"link_state": Score(
instructions="How do the two entity descriptions relate?",
criteria=LEVELS,
),
【"same_字段1": Noul(instructions="这两条记录的【字段1】说的是同一个吗?")】,
【"same_字段2": Noul(instructions="……同上,每个文本字段一道")】,
}
# 选填项:数字型字段(价格、度数、数量)不出 Noul——两个数字比较是算术,代码一行?
# 输出规范:score 取最近档定结局(不同/人工/同一),三道 Noul 明细供人工复核
# 约束条件:
# ✅ 全程无阈值常数——切点(0.5 / 1.5)由三档措辞决定,看分数之前就能写好
# ✅ 中间档值得最认真写:它决定哪些对进 curator
# ❌ 别为「不放心」再加一层阈值判断——要么改等级措辞,要么接受送人工的比例Checkpoint ✅
- I can state the task-type combination for each of the three cases: reranking uses 1 Noul per pair, line-by-line search uses 1 Choice + 1 Noul in one request, entity alignment uses 1 Score + 3 Noul in one request
- Official reranking numbers: top-1 hit 5% → 18%, top-5 15% → 35%, top-10 38% → 62%; the correct passage is in the shortlist 100% of the time, yet ranks first only 5% of the time
- I can explain the teaching point of the arbitration query: the highest-ranked line gets a 0.86 relevance score while exists is only 0.14 — most similar does not mean it's the answer
- Entity alignment takes the nearest of the three outcomes (one
routeline); 450 pairs split 360 / 50 / 40 (80.0% / 11.1% / 8.9%), with no threshold constants anywhere in the file - I know the two hard boundaries: reranking cannot rescue recall misses from the fast search; one Choice supports at most 255 options — beyond that, pick a window first, then rank
Pitfall warnings
❌ Making reranking rescue recall misses
Reranking only reranks the candidates already in the shortlist — passages the fast search didn't select are invisible to Jev no matter how accurate it is. In the official data the correct passage made it into the top-30 100% of the time — that's BM25's performance on this dataset; before switching to your corpus, check shortlist recall first, and if it's not enough, tune the fast search's recall or add candidates — that's a first-layer problem, don't fix it in the second layer.
❌ Sending only Choice without asking exists
Choice probabilities always sum to 1, so one line always ranks first — when the document has no answer it will still insist on pointing at a line, and following it is a hallucination. The second number in line-by-line search (the exists Noul) is the real criterion for "is there an answer": its probability doesn't depend on the other options, and drops to near 0 when there's no answer (measured 0.14).
❌ Cramming a 260-line document into one Choice
One Choice supports at most 255 options. The official approach for going over the limit is two-stage: a first Choice picks which window of lines the answer is in, and a second ranks within the window — each stays within 255. Conversely, a 65-line document needs no chunking — just use it directly.
❌ Throwing numeric fields to Jev for comparison too
In the beer catalog, abv (alcohol by volume) 8.10 % vs 9.30 % — the officials explicitly say comparing two numbers is arithmetic; if you want it, do it in code. The three text fields are Noul's job: whether name / brewery / style are "alike" depends on the wording, and 0.35's style score can directly tell the curator where the disagreement lies. It's the same iron rule as value extraction: the model reads text, arithmetic stays in code.
❌ Shipping Chinese documents to production without testing
The official language support statement: English is the primary training language with the best accuracy; other languages (including Chinese) work but with lower accuracy. Before putting Chinese semantic search or catalog alignment into production, run a batch on your own real data first, focusing on the probability distribution and exists interpretation (see the language support section in Models and Pricing).
Advanced Tips (Optional)
One-to-many questioning. In the reranking demo we asked only one question per pair for clarity; the officials' exact words — real applications will ask multiple questions in a single call for the same "query–candidate" pair. Multiple questions in one request adds almost no response time and only costs the tokens of the questions; official testing found 13 questions sent together are 12.2x cheaper and 10.0x faster than sending them separately, with identical answers — see Parallel Questions and Speculative Fan-out for the mechanism.
Swap embeddings for the fast search. The officials make clear that the fast search in the reranking section is just BM25 and nothing else, but that's a side issue — dense embeddings (comparing passages semantically) can also build a shortlist, and the second-layer reranking isn't picky. On Chinese corpora, the combination of embedding retrieval + Jev reranking often beats a pure BM25 starting point.
Generate a Playground link to tune wording. If you want to iteratively adjust question wording in the browser and watch how the probability distribution changes, use cooksafe's make_playground_link (the link carries the full state and questions, so don't put sensitive data in it):
from cooksafe import make_playground_link
link = make_playground_link(
{"entity_a": PAIRS[0]["entity_a"], "entity_b": PAIRS[0]["entity_b"]},
QUESTIONS,
)
print(link) # 打开 console.typesafe.ai/playground,改 middle level 措辞看 score 怎么动That middle-tier wording tested by the official entity alignment cookbook is exactly the kind worth tuning repeatedly: the wording is your decision logic, and tuning words is far more explainable than tuning thresholds.
Hook it into RAG. Retrieved passages raise not only "which one is most relevant" but also "should this go into the context" — that's a separate process; the officials' approach is four Noul scoring questions per passage with threshold routing, see RAG Passage Gating. Combining the two lessons gives you a complete retrieval quality layer.
Lesson Summary
✅ Jev reranking = fast search handles recall + one Noul per pair reranks by probability: on the BM25 shortlist, top-1 hit rate 5% → 18%, top-10 38% → 62%; no thresholds, ranking happens in code.
✅ line-by-line search = row IDs as Choice options (for locating, and only able to point to rows that actually exist) + one exists Noul in the same request (willing to say "the document has no answer"; tested at 0.14 against 0.86's highest relevance score).
✅ entity alignment = one Score question with a three-tier verdict (the nearest tier decides the outcome) + three Noul questions naming the disputed fields for the curator; 450 pairs split 360 / 50 / 40, no threshold constants anywhere — the three-tier wording is the decision logic.
✅ Three hard boundaries: reranking cannot rescue misses from the fast search; Choice allows at most 255 options (beyond that, go two-stage); number comparison is arithmetic, leave it to code.
Key configuration fields recap:
| Field | Value on this page | Notes |
|---|---|---|
shortlist TOP_K | 30 | number of candidates per query handed to reranking; 40 × 30 = 1,200 calls |
exists interpretation line | FOUND 0.7 / ABSENT 0.35 | official note: with an answer usually ≥ 0.9, without ≤ 0.05; tune with your own documents before going live |
LEVELS / OUTCOME | different / related / same ↔ 0 / 1 / 2 | the middle-tier wording decides which pairs go to the curator; the cutoffs 0.5 / 1.5 were not tuned |
| requests per pair | 1 (multiple questions per request) | reranking: 1 Noul per pair; alignment: 1 Score + 3 Noul per pair; line-by-line search: 1 request with 2 questions per query |
MAX_WORKERS | 6 (official) | official note: the public endpoint rate-limits above roughly 8 concurrent requests; over the limit you get 429, and the SDK backs off automatically |
| model | jev-1.12 (official recording) | for live runs, pin your own version ID instead of the drifting jev-latest |
| options limit | Choice 255 | over the limit, go two-stage: first pick a window, then rank rows |
Related tutorials:
- RAG Passage Gating — a same-family retrieval scenario: whether retrieved passages should go into the context, four Noul questions per passage with threshold routing
- Value Extraction: dates, emails, phone numbers, amounts extracted verbatim — the other side of the same "candidate shrinking" idea: regex finds candidates, Jev picks the right one
- Parallel Questions and Speculative Fan-out — the underlying mechanism of "multiple questions in one request" with 12.2x / 10.0x measured data
Next Lesson Preview
Search matching is done — how do you verify field values at the lowest cost? Next lesson: Cascade Extraction: cheap models do the work, Jev verifies, expensive models catch the misses.
You will learn:
- A cheap model does the extraction first, Jev verifies its output field by field, and only verification alarms escalate to the expensive reasoning model — cost goes where it matters
- How to design the "Jev verification" layer: one question per field, alarm logic entirely in code
- A continuation of the same division of labor as this page: the model makes judgments, code does the routing — except this time what's being routed is models