Smart Routing: Intent Classification Plus Confidence Gating
The parallel questions lesson from before solved "asking all the questions in one go." This lesson solves what comes next: once you have the answers, who should handle the message — deterministic code, an expert LLM, or a human.
The official docs break this into two patterns: Intent routing classifies first, then dispatches; Confidence-gated routing adds another gate — "the answer tells you what it is; confidence tells you whether to act." Together, the two patterns form this lesson's smart routing. About 12 minutes in total.
In fact, you already did the most basic form of routing in your first request: route_ticket.py used the department Choice to assign tickets to the technical team. This lesson expands that into a complete routing system ready for production.
What You'll Be Able to Do
- Use Choice for Jev intent routing: sort inbound messages into a fixed set of intents (order_status / product_question / return_exchange / complaint), and use Score to assess complexity at the same time
- Write routing code with a fallback branch: confidence below the threshold → human; past the threshold → if/elif dispatch to three kinds of handlers
- Set different thresholds by action risk: low-risk actions (checking a balance) pass at 0.6, while high-risk actions (approving a transfer) require 0.85 or higher to skip confirmation
- Explain when you should not set a confidence threshold, and the official troubleshooting path when routing doesn't behave as expected
- Use probabilities for secondary routing beyond gating, such as "CC the second-ranked intent"
Your Current Struggles
- I use a large model for intent routing, and my prompt hard-codes "output only one of the four options," yet it still occasionally replies with "this is an order inquiry" followed by an explanation. I wrote dozens of lines of regex plus try/except to catch it, and my parsing code ended up longer than the business logic
- I've also tried keyword-based rule routing: "refund" → after-sales, "shipping" → orders. It worked great the first week; the second week a user said "the thing I bought last time still hasn't arrived," and the rule table grew from 20 entries to 200, while the hit rate still wouldn't go up
- The model never tells me when it's "unsure": fifty-two percent certainty and ninety-eight percent certainty look exactly the same when they come back to me, so I just have to push through and execute. If an order lookup is wrong, the user just asks again; but executing a transfer incorrectly means real money lost
When to Use This Technique
When you need to:
- Dispatch inbound requests to different handlers: customer service tickets, voice commands, agent skill selection
- Safely back off when the model is unsure: hand off to a human, ask for confirmation, or rephrase the question
- Spend expensive resources only on requests that truly need them — an expert LLM shouldn't be answering "check my order"
And you don't want to:
- Run every message through a large model and then write parsing code to scrape its text output
- Maintain a keyword mapping table that can never be completed
- Have no uncertainty signal from the model, so you can only fight fires after mistakes happen
🎒 Preparation Before Starting
- Completed Parallel Questions and Speculative Fan-Out: you know that multiple questions are evaluated in parallel and independently within a single request (this lesson's Choice + Score relies on that)
- Have run the first request successfully:
.venvhastypesafe-sdkinstalled, andTYPESAFE_API_KEYis in your environment variables - Roughly remember how Choice and Score are written (it's fine if you forgot — Step 1 will walk you through it)
Core Idea
Classify first, then dispatch. The official definition of Intent routing:
Classify inbound requests and route each one to the optimal handler: deterministic logic, an expert LLM, or a human.
Not every message needs the same kind of handler: some can be answered with a quick database lookup, some need an LLM with domain context, and some must go to a person. Jev sits in front of all handlers as the "fast and cheap" classifier — the official docs say most calls complete in roughly 100 milliseconds (docs/concepts/how-to-build-with-system-one.md:55), and expensive things are only triggered by requests that truly need them.
Add another gate. The official definition of Confidence-gated routing:
Treat confidence as a second axis. The answer tells you "what it is"; confidence tells you "whether to act."
Every Choice and Score answer from Jev comes with its own confidence (0–1, computed from how concentrated the probability distribution is). The official three-tier usage:
| confidence | Behavior |
|---|---|
| High | Execute automatically, no human involvement needed |
| Medium | Proceed with caution: ask the user to confirm, flag for review, or gather more information first |
| Low | Do not act: hand off to a human, ask for clarification, or fall back to another system |
Putting the two patterns together gives you a complete routing pipeline (the structure of the official customer service routing example):
customer message
│ One request: message + 2 questions
▼
┌─────────────────────────────┐
│ intent (Choice) │ ← evaluated in parallel
│ complexity(Score) │
└─────────────────────────────┘
│ One response: 2 answers with confidence
▼
intent confidence ≥ 0.5?──no──▶ human
│yes
▼ Dispatch by intent.choice
├─ order_status → order lookup (deterministic code, no LLM)
├─ product_question → product expert LLM
├─ return_exchange → returns/exchange expert LLM
└─ complaint → complexity > 1 or unsure of itself?──yes──▶ human
│no
▼
complaint handling LLMWhy not just route directly with a big-model prompt?
| Comparison dimension | LLM prompt routing | Jev intent routing |
|---|---|---|
| Return format | Free text; whether it outputs order_status or "order lookup" is a coin flip | Typed answer, always one of the predefined options |
| Parsing code | Regex + try/except + retries | Take the option name straight into if/elif, zero parsing |
| Information beyond intent | Fire another request and parse it again | Add a Score question in parallel within the same request |
| Uncertainty signal | Not available | Every answer carries its own confidence |
| Adding a new intent | Edit the prompt and re-debug | Add an option to the request and a branch to the dispatch code |
So why keep the confidence gate at all? Without it, the top1 probabilities 0.52 and 0.98 would give you two answers that look exactly the same in your hands. If one of the four message types is a message that "both asks about an order and is emotionally intense," the intent splits between order_status and complaint, and the answer still hands you a winner—only the probability is diluted and confidence drops. The confidence gate lets you choose "hold off" at moments like this.
The formula for confidence and the full three-tier interpretation are in the Confidence lesson; this lesson focuses on how to use it for routing.
Follow along
Step 1: Define the two questions of intent classification and complexity
Why: routing can only work if the answers are things code can directly match against. Choice gives you a "pick one of four" intent, and Score gives you a "how complex" ruler—these two pieces of information respectively determine "who handles it" and "whether it can be automated".
Here are the two question definitions given on the official patterns page Playground (extracted from the rendered page; option descriptions are the official original text):
from typesafe_sdk import Choice, Score
ROUTING_QUESTIONS = {
"intent": Choice(
instructions="The primary intent of this customer message",
criteria={
"order_status": "Asking about an existing order",
"product_question": "Asking about a product before buying",
"return_exchange": "Wants to return or exchange something",
"complaint": "Unhappy with experience, wants resolution",
},
),
"complexity": Score(
instructions="How complex is this request to resolve",
criteria=[
"Simple lookup or standard procedure",
"Requires some judgment or multi-step process",
"Unusual situation, edge case, or escalation needed",
],
),
}Three details:
- Option descriptions must distinguish one another. The model sees option names and descriptions, not question IDs. Write "Asking about an existing order" rather than "order-related", so the four intents don't crowd together
- The third level of the complexity scale is "escalation needed". The routing code uses exactly this end as the signal to escalate to a human (Step 4)
- When the list may be incomplete, add
other/none of the aboveoptions so the model can say "none of these"—an official recommendation, see the Choice lesson. In this example the four intents cover all entry points of the sample system, so none were added
You should see: two questions in ROUTING_QUESTIONS—one Choice with four intent options, one Score with a three-level complexity scale.
- Each of my intent options has a description that distinguishes it from the others
Step 2: Get both typed answers in a single request
Why: intent and complexity don't depend on each other—as the previous lesson showed, independent questions should be merged into one request: almost no added response time, just a few extra question tokens.
Create smart_route.py, hook up the question definitions from Step 1, and send one clear message to try it:
from typesafe_sdk import TypeSafeClient
# ROUTING_QUESTIONS 的定义见第 1 步,此处省略
message = "Hi, where is my order? It was supposed to arrive last Tuesday and tracking hasn't updated."
with TypeSafeClient() as client:
response = client.system_one(
state=message,
questions=ROUTING_QUESTIONS,
)
intent = response.answers["intent"]
complexity = response.answers["complexity"]
print(intent.choice, intent.confidence)
print(complexity.score, complexity.confidence)Run it:
.venv/bin/python smart_route.pyYou should see: the first line is order_status plus a confidence close to 1.0—this message has only one clear intent, so the probability concentrates entirely on that option (in the official quickstart example, the ticket that only mentions an API 500 error has a department confidence of 1.0). The second line is the complexity score with its own confidence; grab it now in this step, and use it in Step 4. Actual values depend on your real response and may change with Jev versions.
Swap in a mixed-intent message and run it again:
message = "This is the third time I'm writing and nobody has replied. Where is my order? I'm done waiting — I want my money back."You should see: intent.choice still gives a winner's name (it must—choice is always the highest-probability option), but this time the probability is spread across order_status / return_exchange / complaint several options, and intent.confidence drops noticeably. The official Choice lesson documentation records a similar case: a ticket mentioning both a wrong size and a duplicate charge, returns 0.61 / billing 0.35, with confidence of only 0.42—the message belongs to two teams, and the distribution splits honestly.
This dropped confidence is exactly the signal the gate in Step 3 consumes.
- Clear message: high confidence; mixed message: confidence drops noticeably
Step 3: Write the dispatch code for Jev intent routing
Why: routing is at heart just if/elif—provided the answers are clean enough to need no parsing. Jev guarantees choice is always one of the options in the criteria, so the dispatch code can enumerate exhaustively.
Below is the route_ticket code from the official patterns page, verbatim (docs/patterns/intent-routing.md). The handler functions (handle_order_status, handle_with_llm, etc.) are official placeholder names—replace them with your own business logic:
def route_ticket(ticket_id, response):
intent = response.answers["intent"]
complexity = response.answers["complexity"]
if intent.confidence < 0.5:
# If we don't have enough confidence to classify, route to a human agent
return route_to_human_agent(ticket_id)
if intent.choice == "order_status":
handle_order_status(ticket_id)
elif intent.choice == "product_question":
handle_with_llm(ticket_id, PRODUCT_SPECIALIST)
elif intent.choice == "return_exchange":
handle_with_llm(ticket_id, RETURNS_SPECIALIST)
elif intent.choice == "complaint":
low_confidence = complexity.confidence < 0.5
# A higher complexity.score leans toward the "escalation needed" end of the scale.
if complexity.score > 1 or low_confidence:
# Too complex for safe automation, or we're not sure about the complexity; route to a human.
route_to_human_agent(ticket_id)
else:
handle_with_llm(ticket_id, COMPLAINT_RESOLUTION)Run it against the two messages from Step 2:
route_ticket("T-1024", response)You should see:
- Clear message (order_status, high confidence) → the branch to a human doesn't trigger; handed to
handle_order_status—deterministic code, answered with a single database query, zero LLM involvement - Mixed message (confidence drops below 0.5) → straight to
route_to_human_agent; the error is intercepted before execution
Read through the structure of this code: the four branches correspond exactly to three kinds of handlers:
| intent | handler | cost |
|---|---|---|
order_status | deterministic code (order lookup) | cheapest, no LLM |
product_question / return_exchange | each with its own expert LLM | each loaded with a different domain context |
complaint (complex or uncertain) | human | the most expensive, reserved only for cases that truly need it |
The official summary of this code: classification is all done in one quick call, and expensive resources are only triggered by requests that truly need them.
- The first path in my routing code is "confidence not high enough → human", not direct dispatch
Step 4: add a second complexity gate to the complaint branch
Why: the gate in Step 3 only looked at the confidence of the intent itself. But some intents naturally require one more layer of scrutiny: a complaint might be "grumbling that shipping is slow", or it might be "threatening to expose this on social media" — the same intent label, with vastly different handling costs. This layer of judgment is handed off to the complexity answer already obtained in the same request, with no second call needed.
Looking back at the complaint branch in the Step 3 code:
elif intent.choice == "complaint":
low_confidence = complexity.confidence < 0.5
# A higher complexity.score leans toward the "escalation needed" end of the scale.
if complexity.score > 1 or low_confidence:
# Too complex for safe automation, or we're not sure about the complexity; route to a human.
route_to_human_agent(ticket_id)
else:
handle_with_llm(ticket_id, COMPLAINT_RESOLUTION)Two gate conditions, each intercepting one kind of case:
complexity.score > 1: the score crossed the middle level, leaning toward the "escalation needed" end of the scale — could be automated, but not safe to automate, route to a humancomplexity.confidence < 0.5: even the complexity itself wasn't assessed clearly — insufficient information, also route to a human
After this code, the official docs give a special reminder: the Score answer should also undergo a confidence check, and low-confidence meanings should be read in the context of the system and the cost of the decision.
You should see: for the Step 1 message "this is my third letter and nobody is responding, I want my money back", the intent most likely lands around complaint; when complexity is high or uncertain, it falls into the human branch instead of the complaint LLM — spending one extra human slot in exchange for a safe handling that doesn't escalate the customer complaint.
- I can state what the two route-to-human conditions in the complaint branch each intercept: "too complex" and "uncertain"
Step 5: scale confidence thresholds by risk (voice banking example)
Why: the earlier 0.5 threshold is a safety net for "classification itself". But even when the classification is correct, different actions have different error tolerance: playing the wrong balance once is fixed by the user just listening again; approving the wrong transfer means financial loss. The official second patterns page turns this logic into a voice banking example.
Its intent question (also extracted from the official Playground):
BANK_INTENT = Choice(
instructions="What action is the user requesting?",
criteria={
"check_balance": "Check the balance of an account",
"approve_transfer": "Approve the pending transfer request",
"other": "Something else",
},
)The gating code (verbatim from the official docs, docs/patterns/confidence-routing.md):
action = response.answers["intent"]
# Below 0.6 confidence on any action, route to a human
if action.confidence < 0.6:
route_to_support_agent(account_id)
elif action.choice == "check_balance":
# Low stakes. 0.6 confidence is sufficient.
show_balance(account_id)
elif action.choice == "approve_transfer":
if action.confidence > 0.85:
# High stakes, but high confidence. Safe to act automatically.
approve_transfer(account_id)
else:
# High stakes, moderate confidence. Verify intent first.
ask_user_to_confirm("Just to confirm: you would like to approve this transfer, is that correct?")
else:
route_to_support_agent(account_id)The same set of multiple-choice questions, four exits, with the threshold following the risk:
| Case | Threshold | Exit |
|---|---|---|
| Any action, confidence < 0.6 | safety baseline | route to a human agent |
check_balance, ≥ 0.6 | low risk | announce the balance directly |
approve_transfer, 0.6–0.85 | high risk, medium confidence | verbally confirm with the user before executing |
approve_transfer, > 0.85 | high risk, high confidence | auto-approve |
other (intent outside the list) | — | route to a human agent |
The official explanation of these numbers: the 0.6 safety baseline catches all cases where the model is genuinely uncertain; above the baseline, each action has its own threshold based on "the cost of executing it wrongly" — checking the balance only needs 0.6, because the worst case is just the user hearing the announcement one more time; approving a transfer requires above 0.85, otherwise the system should confirm first.
Try these three commands (replacing the state and question definitions from Step 2, sending one request at a time):
for message in [
"What's my account balance?", # → check_balance
"Yes, go ahead and approve the transfer.", # → approve_transfer
"Can you help me with something else?", # → other
]:
response = client.system_one(state=message, questions={"intent": BANK_INTENT})
# …接下面的门控代码You should see: the first automatically announces the balance; the second most likely lands in either the > 0.85 auto-approve branch or the 0.6–0.85 confirm-first branch — both branches are reasonable exits, and that is exactly the point of gating; the third goes to the else branch and routes to a human.
Don't confuse the two sets of numbers: another official example for the same scenario (the "Thresholds scale with risk" section of docs/confidence.md, also covered in the Confidence lesson) uses 0.5 fallback / transfer > 0.9. Both are official examples, with different numbers but the same pattern—the difference itself is exactly what the officials want to convey, and the NOTE in the same section explicitly states:
The correct thresholds depend on your domain and how the model performs on your use case. Start with conservative thresholds, test with your own data, then adjust as you observe.
💡 Routing Chinese-language messages
The official docs state that English is the primary training language with the best accuracy, and other languages including Chinese work but with lower accuracy. Before putting a routing system for Chinese messages into production, first run a batch with your own data and focus on the confidence distribution — whether the gate wrongly blocks or wrongly passes matters more than checking individual answers for correctness.
Checkpoint ✅
- I can describe the two steps of intelligent routing: Choice/Score classifies intent and complexity, and confidence decides whether to let it through
- I know the three kinds of handlers: deterministic code (no LLM), expert LLM, and human — and which intents each suits
- The first path in my routing code is a human fallback (e.g.
intent.confidence < 0.5→ human) - I know thresholds scale with risk: low risk only needs 0.6, high risk (transfer approval) requires above 0.85
- I know the official troubleshooting order when routing misbehaves: check the thresholds and the question first, not the model
Pitfall warnings
❌ Setting confidence thresholds everywhere
Stuffing the "below X, route to a human" check into every code path is the most common misuse. The official agent-skill docs warn:
If you only care about picking the best option, just pick the one with the highest confidence (rather than setting a confidence threshold). If you have specific statistical algorithm requirements, you should most likely use probabilities rather than confidence.
When to use gating: when the answer triggers real actions and those actions differ in cost. For scenarios that only involve ranking or only need the top1, just take choice directly.
❌ When routing misbehaves, going straight to rewording the question
Symptom: recall is off — cases that should go to a human don't (false negatives), and cases that shouldn't do (false positives). The official troubleshooting order is check the thresholds first, then the question:
- Threshold set too high → false negatives (many cases that could be automated get routed to humans)
- Threshold set too low → false positives (many uncertain cases get executed automatically)
- Only if the thresholds really are fine, then make the question more specific (e.g. the option descriptions don't differentiate enough)
❌ Using a single threshold for the whole system
0.5 serves as the fallback floor for classification, 0.6 for low-risk actions, and 0.85 for high-risk actions — all three numbers coexist in the voice banking example above. The official docs state plainly that "confidence threshold is not one number": within the same system, different actions get their own thresholds based on "the cost of getting it wrong," and your code embodies your risk tolerance. A system with only a single threshold either blocks too much (human review volume explodes) or lets too much through (high-risk actions run unguarded).
❌ The intent list has no "none of these" escape hatch
When the option list doesn't cover all inputs, the model is forced to pick the least-bad one from the wrong set — and it will pick one confidently (the official function calling cookbook's exact words on parameter selection: "with no stated issue, choice must commit to a window, and it will commit to one confidently"). The right approach: if the list may be incomplete, add other / none of the above options, and give other an explicit exit path in your code (the else → 转人工 branch, i.e., the else branch that routes to a human, shown above). The docs also caution that none of the above performs mediocrely in many scenarios; an alternative is to add a companion Noul in the same request asking "does this message mention any of our products," and have the code read the Choice answer only when the Noul is yes.
Advanced techniques (optional)
Use probabilities for second-intent carbon copies. Beyond gating, probabilities can also serve as a secondary routing signal. In the official Choice docs' five-question triage example (a ticket involving both a return/exchange and a duplicate charge), there's this passage:
# A second team with a real share of the probability gets a copy
for team, probability in answer.probabilities.items():
if team != answer.choice and probability > 0.25:
notify(ticket, team=team)The primary route goes to the returns team, while billing also receives a carbon copy because it got a probability share of 0.35 (above the 0.25 threshold) — a third path more fine-grained than "low confidence means route everything to humans": the primary answer executes as usual, and the runner-up answer gets flagged to a human. The official record for that ticket: it was routed to returns (issue=wrong_size), billing received a carbon copy, and the customer was asked what they wanted (the confidence for resolution came in at 0.20, below 0.5) — one request, five answers, all gated.
Two alternative gating metrics. If the confidence formula doesn't suit your scenario, the docs recommend two metrics likewise computed from probabilities: first, the top probability p_max (read as "how likely the selected option is," with the threshold set by the number of options — 0.5 is weak with two options and strong with ten); second, the top-to-second ratio p_max / p_second (measuring the leader's edge over the runner-up, purpose-built for routing scenarios where "two strong contenders fight").
Other forms of the gating idea. This "answer + fall back one level when confidence is insufficient" pattern has more than one way to fall back:
- 75-industry classification: SEC annual reports split into 75 industry groups; when confidence falls below 0.9, escalate to the broader division it belongs to — in the official test of 60 documents, the confident half hit 90% while the uncertain half managed only 40%; after escalating to divisions, that rose to 70%
- Agent decisions: picking a skill for an agent from 194 skills — two requests + two thresholds cut wrong loads in half
- The official how-to guide also has a minimal version of gating (route to human review when
confidence < 0.8), four lines of code, a good fit for your first version
Lesson summary
✅ Smart routing = Jev intent routing + confidence gating: Choice classifies the intent, Score rates the complexity, both returned in parallel from one request, and confidence decides whether to let it through.
✅ Three categories of handlers each find their place: deterministic code handles mechanical intents (zero LLM), the expert LLM handles domain questions, and humans handle low-confidence and high-risk cases.
✅ The gating structure is "a fallback floor + risk thresholds": every action first passes the fallback floor (example: 0.5 / 0.6), and once past the floor, each action gets its own threshold based on its cost (checking a balance 0.6, transferring money 0.85).
✅ When the intent list may be incomplete, add other / none of the above and give it an explicit exit path; a model forced to commit will "confidently pick wrong."
✅ Picking only the best option needs no threshold; use probabilities for needs like ranking and carbon-copying.
Key facts at a glance:
| Item | Value | Source |
|---|---|---|
| Intent question instructions | The primary intent of this customer message | Official Playground (patterns/intent-routing) |
| Intent options | order_status / product_question / return_exchange / complaint | Same as above |
| Classification fallback | intent.confidence < 0.5 → human | docs/patterns/intent-routing.md |
| complaint second gate | complexity.score > 1 or complexity.confidence < 0.5 → human | Same as above |
| Banking fallback floor | confidence < 0.6 → human | docs/patterns/confidence-routing.md |
| Transfer threshold | > 0.85 executes automatically, 0.6–0.85 confirms first | Same as above |
A routing system is a classic case of "running Jev on the critical path"; before going live, walk through the nine failure modes together with Jev 1.13 pitfall handbook (especially the literal-interpretation and the large-state-irrelevant-details ones).
Next lesson preview
Step 4 used a single Score to rate complexity and was already able to trim the human workload. The next lesson takes this idea all the way — composite scoring: break complex judgments into atomic scores and weight them.
You will learn:
- Breaking complex judgments like "how severe is this complaint" into multiple atomic Score questions
- Normalization (dividing by the highest level number) to pull scores from different scales onto the same scale
- Weighting and merging into a total score in code, where the weights are your statement of risk
- When rankings don't match expectations, change only the weight constants — never touch any prompt