Skip to content

Citation Checking: Blocking Hallucinated Citations ​

💡 One-sentence summary: Use Jev to build an AI citation-checking pipeline—string matching blocks fabricated citations, a single Choice question determines whether the context supports the claim, and low-confidence verdicts are automatically escalated to a human.

LLMs love attaching citations when answering questions: each claim comes with a passage from the source document plus the quote it's based on. But some of these are wrong, or even made up: the quote may not be in the document at all; or it may sit word-for-word in the document while its context says something entirely different.

How tedious is manually checking one citation? The official cookbook lists three steps: find the document, find the quote within it, then read enough context to judge whether it holds up the claim. That's for one citation—what about 8, 80, or 800 of them?

This official cookbook (Double-checking citations) automates the process with two gates: first, plain string matching finds citations "not in the source text"—these are fabricated and never even reach the model; surviving citations then go through one Choice question that reads the context and judges its relationship to the claim.

The previous lesson, LLM Guardrails, blocked "things that shouldn't be said"; this lesson blocks "texts that shouldn't be cited"—the same "probability + threshold + code-based routing" approach, this time guarding "whether every claim has a source." This lesson takes about 12 minutes and walks you through the official data pack end to end: the source document is RFC 7519 (JSON Web Token, 2015-5), and the 8 citations were written by an LLM against it—4 accurate, 4 deliberately tampered with by the officials, all verified against the officially published numbers (jev-1.12, 2026-08-16 sampling).

text
A citation (claim + quote + section)
        │
        ▼
Gate 1: string matching—is the citation in the source document?
        │                    │
     Not found              Found (record its section) / section reference only
        │                    │
        ▼                    ▼
    fabricated        Gate 2: one Choice question
   (fabricated, zero requests)      What's the relationship between context and claim?
                         supports      → verified
                         contradicts   → contradicted
                         says_nothing  → unsupported
                              │
                              ▼
                    confidence ≥ 0.8 → verdict takes effect
                    confidence < 0.8 → human confirmation

What You'll Be Able to Do After This Lesson ​

  • Automatically classify every citation in AI output into one of four verdicts: verified passes, fabricated / contradicted rejected, unsupported escalated to a human
  • Use zero-API-cost string matching to first block hallucinated citations that "don't exist in the source text at all"
  • Use one Choice question to read a citation's context and judge whether it supports / contradicts / says_nothing about the claim
  • Automatically route by confidence threshold: high-confidence verdicts take effect directly, low-confidence ones enter the human queue (official starting point 0.8)

Why Use Jev Instead of Manual Line-by-Line Checking? ​

CapabilityManual Line-by-Line CheckingJev Citation Checking
Time per citation❌ Find the document, find the quote, read enough context—one at a time✅ Measured with the official cache: 0.08–0.25 seconds per verdict
Verdict criteria❌ Varies by person and by day; "not mentioned in the source" and "the source says the opposite" often get lumped together✅ Three relationships defined in criteria, collapsed into four fixed verdicts
Fabricated citations❌ You have to dig through the source to the right spot to spot them✅ String matching blocks them first, without sending a single request
How certainty is expressed❌ "Looks fine"—can't feed into a workflow✅ Calibrated probability + confidence; below 0.8 automatically escalated to a human
Batching and auditing❌ Check records stay in people's heads and chat logs✅ Every verdict comes with probability, confidence, and token counts—archivable and replayable

Your Current Pain Points ​

  • "The citations in AI-written reports look more and more real each time—I can't possibly check every one against the source, right? But last time, one fabricated quote slipped through."
  • "The sneakier kind: a quote that's genuinely in the source word-for-word, but whose context says the opposite—no way to spot it with a quick skim."
  • "I want to check in bulk, but I don't dare let another LLM act as judge—when it gets a verdict wrong, it looks exactly like a citation hallucination."

When to Use This Technique ​

When you need to:

  • Run batch quality checks on "AI output with citations" like RAG Q&A, AI writing assistants, and literature reviews
  • Classify every citation into one of four verdicts, each with its own handling: pass, reject, escalate to a human
  • Block fabricated citations at zero cost, with a human backstop for low-confidence verdicts

And you don't want to:

  • Manually dig through the source for each citation, one by one, reading context
  • End up with check results that are just free text that can't feed into a workflow
  • Let citations that cite real text to make false claims slip through QA and into the final draft

🎒 Prerequisites ​

  • Completed Choice Questions: you know that criteria is an options map and answers come with probabilities and confidence—the entire verdict pipeline on this page relies on a single Choice
  • (Optional but recommended) Read Confidence: of the three usage tiers, "low-confidence goes to a human" is exactly the idea behind the AUTO_ACCEPT threshold on this page
  • Completed Installation and your first request: .venv contains typesafe-sdk, and TYPESAFE_API_KEY is already in the environment variables
  • Know that the official data package ships with a results cache: reruns replay the published numbers and send no requests; only deleting the cache triggers real calls (7 calls — the 8th citation is eliminated at the string-matching step). Billing is per input token, output is free; rates are in Models and pricing

Follow along: ​

Step 1: Get the official data package (RFC 7519 + 8 citations + results cache) ​

Why: checking citations requires real documents and real citations. The official cookbook prepares all three: rfc7519.txt (the RFC 7519 original text, fetched from rfc-editor.org), citations.json (8 citations written by an LLM against the RFC — 4 accurate, 4 tampered with by the officials and doomed to fail), and json_cache.json (the results cache from official runs). All are in the cookbooks/citation_check directory of the typesafe-public-examples repository.

In the directory where you completed the installation lesson (where .venv lives), download the three files and install the two packages used by the official notebook:

bash
curl -sfLO https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks/citation_check/rfc7519.txt
curl -sfLO https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks/citation_check/citations.json
curl -sfLO https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks/citation_check/json_cache.json
.venv/bin/pip install ipython 'cooksafe>=0.2.0,<0.3.0'

Then create this lesson's main script citation_check.py and write the skeleton first (the client and cache come verbatim from the official cookbook):

python
# citation_check.py —— 引用核查:拦住幻觉引文
# 依赖:.venv 里已有 typesafe-sdk(安装课装过),本课另装 ipython 和 cooksafe
import json
import os
import re
from pathlib import Path
from time import perf_counter

from cooksafe import JsonCache
from typesafe_sdk import Choice, TypeSafeClient

# 官方 cookbook 锁定的版本:本页核对的所有数字出自 jev-1.12(2026-08-16 采样)。
# 阈值是对着具体版本调的——jev-latest 别名会随新版本漂移,调过阈值的系统要锁版本 ID。
TYPESAFE_MODEL = "jev-1.12"
AUTO_ACCEPT = 0.8  # 官方注释:起步设高,多转人工;对模型的信任建立起来再往下调

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    timeout=120.0,
)
json_cache = JsonCache(Path("json_cache.json"))

Check that all files are in place:

bash
.venv/bin/python -c "from pathlib import Path; import json; print(Path('rfc7519.txt').stat().st_size, 'bytes'); print(len(json.loads(Path('citations.json').read_text())), 'citations')"

You should see:

text
63039 bytes
8 citations

One thoughtful detail: the officials set the default of api_key to "cache-only" — as long as json_cache.json is present, reruns replay the official numbers end to end without sending a single request, so you can complete this lesson even without an API Key for now; the Key is only required when you delete the cache file to make real calls.

  • The three files downloaded successfully; the command above prints 63039 bytes and 8 citations

Step 2: Load the source document and split it into numbered sections ​

Why: the model shouldn't read the entire 58k-character RFC. The official approach: load_source() first strips the per-page headers and footers (otherwise paragraphs get chopped up), then split_sections() splits the text into 45 sections by numbered section headings; later, each citation sends only its own section into state. "A large state full of irrelevant details" is a known pitfall of Jev — this step keeps irrelevant details out of the door in advance.

Continue writing citation_check.py below (the parsing functions come verbatim from the official cookbook):

python
def load_source() -> str:
    """RFC 7519 verbatim, minus the page headers and footers that interrupt its paragraphs."""
    lines = []
    for line in Path("rfc7519.txt").read_text().splitlines():
        bare = line.lstrip("\f")
        if re.match(r"Jones, et al\.\s.*\[Page \d+\]$", bare):
            continue
        if re.match(r"RFC 7519\s+JSON Web Token \(JWT\)\s+May 2015$", bare):
            continue
        lines.append(bare)
    return re.sub(r"\n{3,}", "\n\n", "\n".join(lines))


def split_sections(source: str) -> dict[str, str]:
    """Map each numbered section ("4.1.3") to its text, split on the RFC's header lines."""
    boundary = re.compile(r"(?m)^(?:(\d+(?:\.\d+)*)\.  .+|Appendix [A-Z]\..*)$")
    marks = list(boundary.finditer(source))
    sections = {}
    for mark, nxt in zip(marks, marks[1:] + [None]):
        if mark.group(1) is None:  # an appendix header only terminates the section before it
            continue
        sections[mark.group(1)] = source[mark.start() : nxt.start() if nxt else len(source)].strip()
    return sections


SOURCE = load_source()
SECTIONS = split_sections(SOURCE)
CITATIONS = json.loads(Path("citations.json").read_text())

Take a look at what the loaded result and the citations look like:

bash
.venv/bin/python -c "from citation_check import SOURCE, SECTIONS, CITATIONS; print(f'{len(SOURCE):,} characters, {len(SECTIONS)} numbered sections, {len(CITATIONS)} citations')"

You should see:

text
58,365 characters, 45 numbered sections, 8 citations

Then look at two of the citations (one with a quote, one that only points to a section):

bash
.venv/bin/python -c "import json; from citation_check import CITATIONS; print(json.dumps(CITATIONS[1], indent=2)); print(json.dumps(next(c for c in CITATIONS if c['quote'] is None), indent=2))"

You should see:

text
{
  "id": "aud_reject",
  "claim": "If a validator does not find itself in a token's audience list, it has to reject the token.",
  "quote": "If the principal processing the claim does not identify itself with a value in the \"aud\" claim when this claim is present, then the JWT MUST be rejected.",
  "section": "4.1.3"
}
{
  "id": "iat_future",
  "claim": "The \"iat\" claim requires validators to reject tokens whose issue time is in the future.",
  "quote": null,
  "section": "4.1.6"
}

Each citation has four fields: id (an identifier you assign), claim (the claim from the AI output), quote (the source passage it cites), section (the section number). quote being null means "only points to a section, no quote given" — there is no string to match, so the named section is sent to the model directly.

  • Both printouts match: 58,365 characters, 45 numbered sections, 8 citations, plus aud_reject (with a quote) and iat_future (quote is null)

Step 3: Use string matching to intercept citations that are "not in the source text" first ​

Why: if a citation isn't in the document, it was made up — no model needed for this kind, plain string matching suffices, without sending a single API call. Normalize before matching: RFC text wraps at about 72 characters per line, so a citation with its original line breaks won't match; collapsing whitespace into single spaces and curly quotes into straight quotes lets citations that span lines still match.

Continue writing below (verbatim from the official cookbook):

python
def normalize(text: str) -> str:
    """Collapse whitespace and fold curly quotes, so a quote matches across line wraps."""
    table = str.maketrans({"“": '"', "”": '"', "‘": "'", "’": "'"})
    return re.sub(r"\s+", " ", text.translate(table)).strip()


def find_quote(sections: dict[str, str], quote: str) -> str | None:
    """The number of the section that contains the quote verbatim, or None."""
    needle = normalize(quote)
    for number in sorted(sections, key=lambda n: [int(p) for p in n.split(".")]):
        if needle in normalize(sections[number]):
            return number
    return None


def locate(sections: dict[str, str], citation: dict) -> tuple[str, str | None]:
    """Step 1 for one citation: a status, plus the section step 2 will read."""
    if citation["quote"] is None:
        return "section-only", sections[citation["section"]]
    number = find_quote(sections, citation["quote"])
    if number is None:
        return "missing", None
    return "found", sections[number]

locate() returns two things: a status (found found / missing missing / section-only section-only), and the section text to send to the model in the next step. Create locate_report.py to see the matching results for all 8 citations:

python
# locate_report.py —— 第 3 步检查:8 条引文的字符串匹配结果
from citation_check import CITATIONS, SECTIONS, locate

for citation in CITATIONS:
    status, section = locate(SECTIONS, citation)
    where = f"section of {len(section):,} chars" if section else "not in the source"
    print(f"{citation['id']:<18}{status:<14}{where}")
bash
.venv/bin/python locate_report.py

You should see (the officially published numbers):

text
epoch_seconds     found         section of 3,122 chars
aud_reject        found         section of 761 chars
sig_reporting     missing       not in the source
clock_skew        found         section of 529 chars
exp_required      found         section of 529 chars
pii_encryption    found         section of 1,653 chars
iat_future        section-only  section of 270 chars
duplicate_names   found         section of 918 chars

sig_reporting is eliminated right here: its "citation" simply doesn't exist in the RFC — the official data license (DATA_LICENSE.md) also states that this quote was written by them, not taken from the RFC. No further requests will be spent on it. iat_future is a different case: it only points to section 4.1.6 without a quote, has no string to match, so that section is taken and it moves on.

  • Of the 8 status rows, one is missing (sig_reporting), one is section-only (iat_future), and the remaining 6 rows are all found

Step 4: Use a Choice question to judge whether the context supports the claim ​

Why: citations that reach this step are all verbatim in the source — but that's not enough: even with an accurate citation, the claim built on it can still be wrong. Judging this requires reading the citation's context (the section found in Step 3). The officials use a single Choice question covering the three possible relations between section and claim; the highest-probability option is the verdict — a verdict with confidence ≥ 0.8 takes effect directly, below 0.8 goes to a human for confirmation.

Continue writing below (the question definition and verdict folding come verbatim from the official cookbook):

python
QUESTIONS = {
    "relation": Choice(
        instructions="How does the section relate to the claim?",
        criteria={
            "supports": "The section states the claim or directly implies that it is true",
            "contradicts": "The section states the opposite of the claim or implies it is false",
            "says_nothing": "The section does not address what the claim asserts, either way",
        },
    ),
}

RELATION_TO_VERDICT = {
    "supports": "verified",
    "contradicts": "contradicted",
    "says_nothing": "unsupported",
}


@json_cache
def ask(claim: str, section: str) -> dict:
    started = perf_counter()
    response = client.system_one(
        state={"claim": claim, "section": section},
        questions=QUESTIONS,
        model=TYPESAFE_MODEL,
    )
    answer = response.answers["relation"]
    return {
        "choice": answer.choice,
        "probabilities": answer.probabilities,
        "confidence": answer.confidence,
        "seconds": round(perf_counter() - started, 2),
        "input_tokens": response.usage.input_tokens or 0,
        "output_tokens": response.usage.output_tokens or 0,
    }


def verdict(status: str, answer: dict | None) -> dict:
    """Fold step 1 and step 2 into one of the four labels, plus an auto-or-review flag."""
    if status == "missing":
        # confidence None: no model was called, so there is no model confidence to report
        return {"verdict": "fabricated", "confidence": None, "auto": True}
    return {
        "verdict": RELATION_TO_VERDICT[answer["choice"]],
        "confidence": answer["confidence"],
        "auto": answer["confidence"] >= AUTO_ACCEPT,
    }


def check_citation(sections: dict[str, str], citation: dict) -> dict:
    status, section = locate(sections, citation)
    answer = ask(citation["claim"], section) if section is not None else None
    return {"id": citation["id"], "status": status, "answer": answer, **verdict(status, answer)}

Three design points deserve attention:

  • The state has only two things: claim and section. The job of "which passage to read" was already done in Step 3, so the model receives clean material for judgment.
  • The @json_cache decorator: cache hits replay the official numbers directly without calling the API; only entries missing from the cache trigger real requests.
  • How the four verdicts are folded: missing → fabricated (confidence recorded as None — no model was called, so there is no model confidence to report); the rest map to verified / contradicted / unsupported via RELATION_TO_VERDICT, with the auto flag decided by AUTO_ACCEPT.

💡 This is the unique value of Jev

"The citation is verbatim in the source" does not mean "the claim holds." exp_required's citation is copied word-for-word from section 4.1.4, and the claim says "every JWT must carry exp, and a token without exp is invalid" — while the same section states "Use of this claim is OPTIONAL." In the probability distribution, contradicts=1.0 and confidence 0.99, intercepting this kind of "citing the real text to say the opposite" in one shot. The verdict is a structured field plus calibrated probability, so code can route it directly; if you instead had a generative model write a "review opinion," you'd still have to parse the text — not to mention it hallucinates too.

💡 A note on Chinese content

The official statement: English is Jev's primary training language; other languages, including Chinese, work but with lower accuracy. Before doing citation checking with Chinese documents and Chinese claims, or translating the question into Chinese, first run a batch on your own real data and watch the probability distribution and confidence before setting thresholds.

You should see: no output at this step — leave the code as is once written; the next step runs the full batch.

Step 5: Run the full AI citation check and see which of the four verdicts catches whom ​

Why: a single item shows the logic, the full run shows the results. The official side went through all 8 quotes: all 4 accurate ones came back verified (confidence 0.93 or higher), and all 4 tampered ones were caught—1 fabricated, 1 contradicted, and 2 unsupported routed to manual review.

To wrap up citation_check.py, add the main program:

python
if __name__ == "__main__":
    print(f"{'citation':<18}{'quote':<14}{'relation':<14}{'conf':>6}  {'verdict':<13}{'action':>7}")
    for citation in CITATIONS:
        result = check_citation(SECTIONS, citation)
        answer = result["answer"]
        relation = answer["choice"] if answer else "-"
        conf = f"{answer['confidence']:.2f}" if answer else "-"
        action = "auto" if result["auto"] else "review"
        print(
            f"{result['id']:<18}{result['status']:<14}{relation:<14}{conf:>6}"
            f"  {result['verdict']:<13}{action:>7}"
        )

Run it (with the cache file present, it replays the official numbers without sending requests; delete the cache to make the 7 real calls):

bash
.venv/bin/python citation_check.py

You should see (the officially published numbers, jev-1.12, 2026-08-16 sampling; real calls once the cache file is deleted):

text
citation          quote         relation        conf  verdict       action
epoch_seconds     found         supports        0.93  verified        auto
aud_reject        found         supports        0.95  verified        auto
sig_reporting     missing       -                  -  fabricated      auto
clock_skew        found         supports        0.99  verified        auto
exp_required      found         contradicts     0.99  contradicted    auto
pii_encryption    found         says_nothing    0.27  unsupported   review
iat_future        section-only  says_nothing    0.56  unsupported   review
duplicate_names   found         supports        0.99  verified        auto

Read this table line by line—four details were deliberately arranged by the official side:

  • 4 accurate quotes all came back verified: epoch_seconds, aud_reject, clock_skew, duplicate_names, with confidence 0.93 or higher, well above the 0.8 effective threshold.
  • sig_reporting didn't spend a single request: the quote isn't in the RFC, so the string-matching step already ruled it fabricated—it never even reached the model.
  • exp_required is a case of "quoting real text to say the opposite": the quote is taken verbatim from 4.1.4, yet the same section says "Use of this claim is OPTIONAL"—contradicts 1.0, confidence 0.99.
  • pii_encryption and iat_future went into the manual-review queue: says_nothing with confidence of only 0.27 and 0.56, both below 0.8. pii_encryption is precisely the official counterexample showing that "string matching alone isn't enough"—the quote appears verbatim in the source, but the section it sits in (Section 3) is noncommittal about the claim that encryption is mandatory.

Take another look at cost: the 7 model calls recorded in the cache took 0.08–0.25 seconds each, totaling 4,420 input tokens and 281 output tokens—output is free; input at the current rate of $0.042/Mtok (see Models & Pricing) works out to about $0.0002 for this full verification run.

  • In the table I ran, all four verdicts—verified / fabricated / contradicted / unsupported—appeared
  • Both unsupported items carry the review flag; the rest are all auto

📋 Magic Prompt ​

The "Prompt" on this page is a question-and-verdict template for the API—copy it, replace the 【】 placeholders (those with ? are optional), and it's ready to use.

📝 Citation-relationship Choice question (three-option verdict) ​

Expected effect: turn "what's the relationship between this context and the claim" into a fixed three-option Choice question, with the verdict criteria hard-coded in the criteria.

python
from typesafe_sdk import Choice

QUESTIONS = {
    "relation": Choice(
        instructions="【一句话问上下文与论断的关系,官方原文:How does the section relate to the claim?】",
        criteria={
            "supports": "【什么算支持:章节陈述了论断,或直接暗示它为真】",
            "contradicts": "【什么算相抵触:章节说的是论断的反面,或暗示它为假】",
            "says_nothing": "【什么算没提到:章节对论断的断言不置可否】",
        },
    ),
}
# 必填项:三个选项的方向要正——supports = 引文真的撑得住论断
# 选填项:需要更多判定档位(如「部分支持」)就加选项;选项数变了 confidence 的读法也跟着变?

# 输出规范:answer.choice → 最高概率选项;answer.probabilities → 每个选项的概率;answer.confidence → 0–1
# 约束条件:
# ✅ 选项定义写成可对照的情境,边界情形放进 criteria
# ❌ 不要只给 supports / not_supports 两档——「原文说了相反的话」和「原文没提」必须分开处理

✅ Four verdicts folded together + manual-review threshold ​

Expected effect: fold the string-matching result and the model's verdict into verified / unsupported / contradicted / fabricated, with low-confidence cases automatically routed to manual review.

python
AUTO_ACCEPT = 【0.8】  # 官方起点:先高后低,低于它就转人工

RELATION_TO_VERDICT = {
    "supports": "verified",
    "contradicts": "contradicted",
    "says_nothing": "unsupported",
}


def verdict(status: str, answer: dict | None) -> dict:
    """status: missing / found / section-only(字符串匹配那步的输出)"""
    if status == "missing":
        return {"verdict": "fabricated", "confidence": None, "auto": True}  # 没调模型,confidence 记 None
    return {
        "verdict": RELATION_TO_VERDICT[answer["choice"]],
        "confidence": answer["confidence"],
        "auto": answer["confidence"] >= AUTO_ACCEPT,
    }
# 必填项:AUTO_ACCEPT 用你自己文档上跑出的分布定,0.8 只是官方起点
# 选填项:想更严可以给 contradicted 单独设一条更低的转人工线?

# 输出规范:{"verdict": "verified|unsupported|contradicted|fabricated", "confidence": float|None, "auto": bool}
# 约束条件:
# ✅ auto=False 的一律进人工队列——官方就是这么处理 pii_encryption(0.27)和 iat_future(0.56)的
# ❌ 不要拿 confidence 改判定本身——它只决定「谁来看」,不决定「是什么」

🔧 citations.json: swap in your own documents and quotes ​

Expected effect: point the verification at your own RAG / writing product: swap the source documents and the quote list—no need to change a single line of the verdict pipeline.

json
[
  {
    "id": "【引文 id,用你产品里的引文主键,判定结果好回写】",
    "claim": "【AI 输出里的那句论断,原文照抄】",
    "quote": "【它引用的原文片段;只指章节不给引文时填 null】",
    "section": "【章节号,如 4.1.4;quote 为 null 时必填,模型读这一节】"
  }
]
  • Required fields: claim + quote (or quote: null together with section)
  • Optional fields: id use your product's primary key, making it easy to write verdicts back to the original document?
  • Output spec: each quote → one of the four verdicts + confidence + auto flag
  • Constraints:
    • ✅ quote must be copied verbatim from the source document—the official matching is a normalized exact match
    • ❌ Don't expect it to tolerate "close enough" quotes—a truncated or reworded quote doesn't count as found and will be ruled fabricated; if production needs to tolerate sloppy citations, you'll have to switch to fuzzy matching (the official wording)

Checkpoint ✅ ​

  • citation_check.py produces an 8-row verdict table, and all four verdicts—verified / fabricated / contradicted / unsupported—appeared
  • I can explain why sig_reporting was ruled fabricated without spending a single request—string matching couldn't find that "quote" in the RFC
  • I know exp_required's quote appears verbatim in 4.1.4, and why the verdict is contradicted—the same section says "Use of this claim is OPTIONAL"
  • I know why pii_encryption (0.27) and iat_future (0.56) went into the manual-review queue—both are below the 0.8 AUTO_ACCEPT
  • I know the official numbers come from jev-1.12 (2026-08-16 sampling): with the cache file present it replays that copy; only deleting it triggers real requests

Pitfall warnings ​

❌ Assuming "the quote is in the source" means "the claim holds" ​

pii_encryption is the official counterexample: its quote appears verbatim in Section 3 of the RFC, and string matching easily returns found—but that section is about JWT's structure (claims encoded into JWS/JWE) and is noncommittal about the claim that "personal data must be encrypted," so the model ruled says_nothing with a confidence of only 0.27. String matching can only catch "fabrication," not "quoting real text to make a false statement"—which is exactly why the Choice question in step 4 exists.

❌ Using exact matching as fuzzy matching ​

The official wording: the matching after normalization is exact—if a quote is truncated, or even a single word is lightly reworded, the result is missing → fabricated. If your system needs to tolerate sloppy citations, you'll need to switch to fuzzy matching. Think it through before going live: how are "quotes" generated in your product, and how common are truncation and rewording.

❌ Splitting into just two buckets, "pass / fail" ​

Each of the four verdicts has its own destination: verified passes straight through; fabricated and contradicted get sent back for rewriting—one is made up, the other is the source saying the opposite; unsupported depends on the case—you may need to add a source, or the claim itself may need to change. Compress them into two buckets and downstream can only apply a one-size-fits-all treatment: exp_required (the source says the opposite) and pii_encryption (the source says nothing) would be treated the same way.

❌ Setting a threshold and never revisiting it ​

The official comments are clear: AUTO_ACCEPT = 0.8 is a "start high, route more to manual review" starting point—run it for a while, see how the model performs on your own documents, then tune it down. Also, the official side pinned TYPESAFE_MODEL to jev-1.12 for a reason—the jev-latest alias drifts with new versions, so a threshold tuned against an old version may behave entirely differently after an upgrade (see Models & Pricing).

Advanced tips (optional) ​

Try it in the browser first (no code). The official cookbook uses cooksafe's make_playground_link() to turn exp_required's claim + section and the question into a share link—opening it runs the same request in the Playground:

Open the exp_required playground example →

The code that generates the link is short (run it in the official notebook; example and example_section are taken from the exp_required citation):

python
from cooksafe import make_playground_link
from IPython.display import Markdown, display

playground_link = make_playground_link(
    {"claim": example["claim"], "section": example_section}, QUESTIONS
)

Point it at your own data. The official wrap-up puts it plainly: just swap out rfc7519.txt and citations.json—but load_source() and split_sections() are written for the RFC format (strip headers and footers, split by numbered section headings), so other document shapes need their own parsing. Web pages, PDF-to-text conversions, and Markdown manuals each split into "sections" differently, but the principle "send only the section a citation belongs to" stays the same.

Cache replay vs. real calls. With json_cache.json in place, reruns replay the official numbers (api_key defaults to "cache-only", so you can reproduce without a Key); delete it, and 7 real calls are billed by input tokens with output free. If you want the same "replay first, then run for real" behavior in your own project, cooksafe's JsonCache decorator is exactly for that.

Lesson Summary ​

✅ AI citation checking has two gates: string matching intercepts citations that are "not in the source text" (fabricated, zero API cost); one Choice question determines whether the context supports the claim.

✅ Four verdicts, each in its place: verified passes, contradicted rejected (the source says the opposite), unsupported goes to human review (the source doesn't mention it), fabricated rejected (the citation is made up)—"citing real text to say false things" and "the source never said that" are handled separately.

✅ Confidence gating: verdicts take effect at ≥ 0.8, below that they go to human review—of the official 8 items, exactly 2 (0.27, 0.56) ended up in the human queue.

✅ Official test results: all 4 accurate citations were verified (0.93 and above), and all 4 problematic citations were intercepted; 7 model calls totaled about 4.4k input tokens, roughly $0.0002 at current prices.

Key configuration fields at a glance:

FieldValue on this pageDescription
TYPESAFE_MODELjev-1.12The official cookbook pins this version; aliases drift, so lock the ID when tuning thresholds
AUTO_ACCEPT0.8The line where a verdict takes effect; below it goes to human review. Start high and adjust downward
QUESTIONS["relation"]Three Choice optionssupports / contradicts / says_nothing
Verdict mappingsupports→verified, etc.missing→fabricated without going through the model, confidence recorded as None
state{"claim": …, "section": …}The model only reads the claim + the section it belongs to; irrelevant details never enter the context
Cachejson_cache.jsonReplays the official numbers; delete it for real calls

Related tutorials:

Next Lesson Preview ​

Citation checking guards "the words quoted out"; the next lesson guards "the passages stuffed into the context": RAG Passage Gating: Only Relevant Content Enters the Context.

You will learn:

  • One request per retrieved passage to score relevance; your code decides which passages make it into the answering model
  • Use the Noul Yes/No Questions to give each passage a 0–1 probability score—complementing this page's Choice verdict perfectly
  • Keep irrelevant passages out of the context, so answer quality no longer drifts with retrieval quality
gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.