Agent decision-making: picking skills, converting to function calls
💡 One-sentence summary: use Jev to govern Agent decision-making—one Choice ranks 194 skills, two gates decide whether to say anything, and natural-language requests are converted into typed function calls whose arguments are always valid.
Questions like which skill the agent should load or which function it should call, when thrown at an LLM, yield a piece of text that needs further parsing; thrown at Jev, they yield a typed answer + probabilities, with the decision-making power staying in your code.
The officials broke down this Jev Agent decision-making recipe with two cookbooks: Skill suggestion gates skills for an agent loaded with 194 skills, and Function calling converts natural-language trading requests into calls to ordinary typed functions. Both share the same philosophy: Jev makes the judgments, the code makes the decisions.
The user's request this turn
│
▼ TypeSafe request 1 (ranking + gates, same state evaluated in parallel)
┌──────────────────────────────────────────────┐
│ One Choice: which of the 194 skills? │
│ → probabilities are the ranking, code takes the top three │
│ Three Nouls: should we act (three different phrasings) │
│ One Noul: is a specific tool / template / method named? │
└──────────────────────────────────────────────┘
│
▼ Code: gate ≥ 0.30 or named ≥ 0.50?
No ──┤── Suggest loading nothing (and also send a line saying "no relevant skills")
│ Yes
▼ TypeSafe request 2 (top three only, with full descriptions + SKILL.md openings)
┌──────────────────────────────────────────────┐
│ One Choice: which of the three is it really? │
│ Three Nouls: does this skill actually do this thing? (each answers on its own) │
└──────────────────────────────────────────────┘
│
▼ Code: best fits ≥ 0.30?
No ──┤── Suggest loading nothing
│ Yes
▼
<skill_relevance> block appended after the roster → the agent keeps its own judgmentWhy put another judgment in front of the agent's own judgment? The officials get straight to the point: an agent with a large skill roster is essentially choosing with almost no information—the roster arrives as an index, one skill per line, with descriptions truncated so they don't crowd out the conversation; truncated to 60 characters, the Office skill for editing .pptx and the finance skill for building decks both read as "PowerPoint tools." On turns where no skill fits at all, it may still force-load one, because a list of names by itself invites guessing.
This lesson takes about 20 minutes. All measured numbers on this page come from the official cookbooks' recorded data: skill suggestion uses jev-1.14.0 (tested agent claude-haiku-4-5-20251001, 2026-10-30 rendering), function calling uses jev-1.12; the recording files ship with the official repository, so you can replay and verify them with zero keys.
What you'll be able to do after this lesson
- Add a "which skill should be loaded" gate to a skill-based agent: two requests (first a Choice ranks 194 skills, then a re-check of the top three), officially measured wrong loads 15.7% → 7.9%, useless loads 14.0% → 5.2%
- Write
suggest(): returns at most one skill name, wrapped withsuggestion_block()into a<skill_relevance>block appended to the system prompt—the agent keeps the full index and its own judgment, and the suggestion only says "look at this one first" - Convert natural-language requests into typed function calls:
closed_setsreads function signatures,Literal→ Choice,list[Literal[...]]→ one Noul per member,bool→ Noul, 54 questions asked in one request, and what enters the function is always a legal value from the signature - Understand the two uses of confidence: a function call's confidence = the least-confident judgment in the whole call (one wrong argument ruins the pot); when picking skills and only "picking the best" matters, just take the highest-probability option—no need to set confidence thresholds everywhere
Why use Jev instead of letting the agent pick on its own?
| Capability | LLM picks from the index on its own / generates JSON | Jev Structured decision-making |
|---|---|---|
| Can it tell apart "building a deck" from "editing a deck" | ❌ Index descriptions truncated to 60 characters, both .pptx skills read as PowerPoint tools, officially measured 15.7% wrong loads | ✅ The re-check request carries full descriptions + the first 700 characters of SKILL.md, and deck requests flip from pptx-author to microsoft-office |
| Does it dare stay silent when no skill fits | ❌ A list of names invites blind guessing, officially measured 14.0% forced loads | ✅ Four Nouls stand guard; if the re-check finds everyone unqualified, it suggests "load nothing" (5.2%) |
| Are function arguments legal | ❌ LLM generates JSON that then gets parsed; field values may be fabricated and the parsing code keeps growing | ✅ Closed-set arguments map to Choice / Noul, in the officials' own words: what enters the function is always a value the function accepts |
| How much certainty you need before executing | ❌ An LLM's self-reported confidence is text that has to be parsed yet again | ✅ RLCD calibration training (events marked 0.2 actually occur about 20% of the time in long runs), confidence = the weakest judgment, and anything below the threshold escalates to confirmation |
| Changing the decision policy | ❌ Editing prompts and rerunning everything makes review an uphill battle | ✅ Two thresholds plus the question wording live together as code constants, and the official recommendation is to keep them in a single file for easy human review |
Your current predicament
- "The agent has 194 skills attached, and each description in the index is only 60 characters long—ask it to modify an existing deck and it loads the skill for building a new deck; on turns where no skill fits at all, it still forces one to load"
- "Function calling relies on the LLM generating JSON that then gets parsed; parameter values are occasionally fabricated, field names don't match, and the parsing code keeps growing thicker"
- "You want the model itself to decide 'what to do next,' but the answer is a blob of text, and thresholds, retries, and fallbacks are all scattered across prompts—impossible to review"
When to use this technique
When you need:
- An agent whose skill/tool catalog is so large that descriptions get truncated, and wrong loads and useless loads start eating into reliability
- To turn a user's natural-language request into a call to an existing typed function, with parameters drawn from a fixed list
- The thresholds for "should this be done" plainly in code, so changing policy doesn't mean rewriting prompts
And when you don't want:
- To stuff the entire catalog verbatim into the system prompt—in the official words: that degrades skill selection and brings context rot to the rest of the conversation
- To pay for an extra full LLM generation call every turn just to act as a judge
- To parse JSON spat out by an LLM
🎒 Prerequisites
- Completed Choice questions: you know about the
criteriaoptions map andprobabilities—the first request on this page is a Choice with 194 options (limit 255) - Completed Parallel questions and speculative fan-out: you know that "the code must have the previous answer before it can construct the next request" is what makes a real dependency—the two requests on this page are one of the official three real dependent-request examples
- Got Installation and the first request working:
.venvcontainstypesafe-sdk, andTYPESAFE_API_KEYis in the environment variables - (Optional, for a zero-key reproduction) Download the official data: typesafe-public-examples repository, the
cookbooks/skill_suggestiondirectory'shermes_roster.json,requests.json,json_cache.json, and thecookbooks/function_callingdirectory'strader.py,dispatch.py,spec.json,json_cache.json—run it exactly as the official code does, without a single key - You only need
ANTHROPIC_API_KEYif you want to run the skill-suggestion full live test (step 4): the agent under test is an Anthropic model; if you're only doing steps 2 and 3 withsuggest(),TYPESAFE_API_KEYalone is enough - Know where this page's numbers come from: the skill-suggestion recording used
jev-1.14.0(2026-10-30 rendering) and the function-calling recording usedjev-1.12; your live runs now will usejev-latest(currently pointing tojev-1.14.0), so probabilities will differ—tune thresholds against the version you've pinned
Follow along
Step 1: Meet the skill catalog and the bare-agent baseline (the failure mode of a 60-character index)
Why: the officials first put the failure mode of "the agent picks for itself" on the table: Hermes's 194 skills enter the system prompt as an index, one skill per line, with descriptions truncated to 60 characters.
First pull both cookbooks' data into your working directory (dependencies installed in two lines: neither side's official pip line originally included typesafe-sdk, so it's added here):
mkdir agent-decisions && cd agent-decisions
base=https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks
curl -sLO $base/skill_suggestion/hermes_roster.json
curl -sLO $base/skill_suggestion/requests.json
curl -sLO $base/skill_suggestion/json_cache.json
curl -sLO $base/function_calling/trader.py
curl -sLO $base/function_calling/dispatch.py
curl -sLO $base/function_calling/spec.json
curl -sLO $base/function_calling/json_cache.json
pip install typesafe-sdk anthropic matplotlib ipython polars numpy 'cooksafe>=0.2.0,<0.3.0'hermes_roster.json is NousResearch/hermes-agent's 194 skills at one pinned commit: each record has name, category, a truncated description as it appears in the index, the full description, and the beginning of the SKILL.md. Load it and look at the official statistics output (excerpt from the official Step 1 unit):
import json
from collections import defaultdict
from pathlib import Path
ROSTER = json.loads(Path("hermes_roster.json").read_text(encoding="utf-8"))
BY_NAME = {skill["name"]: skill for skill in ROSTER}
def render_index() -> str:
"""The body of <available_skills>: skills grouped by category, both sorted by name."""
by_category = defaultdict(list)
for skill in ROSTER:
by_category[skill["category"]].append(skill)
lines = []
for category in sorted(by_category):
skills = sorted(by_category[category], key=lambda s: s["name"])
if len(skills) == 1: # a one-skill category is a single top-level row
lines.append(f" - {skills[0]['name']}: {skills[0]['description']}")
continue
lines.append(f" {category}:")
lines.extend(f" - {skill['name']}: {skill['description']}" for skill in skills)
return "\n".join(lines)
widths = [len(skill["description"]) for skill in ROSTER]
print(f"{len(ROSTER)} skills in {len({s['category'] for s in ROSTER})} categories")
print(f"index description: {sum(widths) / len(widths):.0f} characters on average, "
f"{max(widths)} at most")You should see (official release):
194 skills in 32 categories
index description: 55 characters on average, 60 at mostrequests.json is 452 single-turn requests: 280 are covered by exactly one skill (145 distinct skills), and 172 are covered by no skill at all. The covered requests were written by Claude Sonnet 5 from each skill's own SKILL.md; all 172 uncovered ones were written to "punish blind guessing"—84 everyday requests, 42 technical questions no skill serves (explain what a monad is), and 46 asking for things absent from the catalog (post this to Mastodon when the catalog only covers X).
Bare-agent baseline (running the 452 requests one by one, with the model under test pinned to claude-haiku-4-5-20251001):
wrong loads 15.7% (280 covered requests)
needless loads 14.0% (172 uncovered requests)
of 26 wrong first picks, 11 came from the right skill's own categoryBoth metrics are error rates—lower is better: wrong load means that among covered requests, the first skill_view call was not the covering skill (never loading at all also counts); needless load means that among uncovered requests, skill_view was called anyway. Of the 26 wrong loads, 11 fell in the correct skill's own category—the agent roughly looked in the right place; the hard part is telling apart several look-alikes.
💡 This is what "asking at the wrong layer" looks like
The official Agent Skill SKILL.md puts it bluntly: questions that ask Jev to make the decision directly ("Should this ticket be escalated?" "What should the agent do next?") usually indicate the question is being asked at the wrong layer—decompose the decision into the judgments it depends on, and put the decision itself into code, visible and testable. For the full philosophy see Developing with Claude Code; this page follows that throughout: Jev ranks and gates, while the code holds the thresholds and the authority to say "no."
- I can state what each of the two baseline error rates measures, and what it means that 11 of the 26 wrong loads came from the correct skill's own category
Step 2: The first request—one Choice ranking 194 skills, four Noul gates
Why: ranking and "does this turn need a skill" share the same state, so the officials packed them into a single request—the Choice's probabilities are the ranking, and four Nouls handle the gating.
A single Choice easily holds 194 options (limit 255); the officials also left a note: if it grows several times larger, rank in chunks and re-rank the chunk winners. The options' criteria are just the index descriptions the agent itself sees (the same text), and the question's instructions reads as follows:
CHOICE_INSTRUCTIONS = (
"Which of these skills, if any, is the right one to load to help with the "
"user's latest request?"
)The gates are four Nouls: three ask "does this turn require action" from different angles, plus one asking "was a specific method named"—there are only two reasons a turn needs a skill: either it must act on the user's files/account/device, or it names a method a skill owns (a meme template, a Suno prompt).
GATE_QUESTIONS = {
"acts_on_user_system": (
"Is the assistant being asked to act on the user's files, accounts, devices, "
"or online services, rather than only to explain or advise?"
),
"would_follow_documented_procedure": (
"Would a careful expert answering this consult a specific documented procedure "
"or set of commands, rather than answering from general understanding?"
),
"prose_suffices": (
"Could a knowledgeable generalist fully satisfy this request in prose, with "
"no tools, no documentation, and no access to the user's files or accounts?"
),
}
INVERTED = {"prose_suffices"} # a yes here points away from needing a skill
NAMED_QUESTION = (
"Does the request ask for a specific named tool, format, template, or method to be "
"used, rather than for general help?"
)
GATE_THRESHOLD = 0.30 # mean of the three action nouls; at or above it, a skill is wanted
NAMED_THRESHOLD = 0.50 # the "names a method" noul; at or above it, a skill is wanted too
def build_state(request: str) -> dict:
return {"request": request, "recent_context": ""}
@json_cache
def rank_wide(request: str) -> dict:
"""Request 1: rank all 194 skills, and score the request for whether a skill applies."""
questions = {
"which": Choice(
instructions=CHOICE_INSTRUCTIONS,
criteria={skill["name"]: skill["description"] for skill in ROSTER},
)
}
for key, text in GATE_QUESTIONS.items():
questions[f"gate::{key}"] = Noul(instructions=text)
questions["named"] = Noul(instructions=NAMED_QUESTION)
response = client.system_one(
state=build_state(request), questions=questions, model=TYPESAFE_MODEL
)
ranked = sorted(
response.answers["which"].probabilities.items(), key=lambda kv: -kv[1]
)
values = {
key.removeprefix("gate::"): answer.noul
for key, answer in response.answers.items()
if key.startswith("gate::")
}
oriented = [(1.0 - v) if k in INVERTED else v for k, v in values.items()]
return {
"ranked": ranked[:12],
"gate": sum(oriented) / len(oriented),
"named": response.answers["named"].noul,
}
def needs_skill(wide: dict) -> bool:
"""Either reason is enough: an action is wanted, or a named method is asked for."""
return wide["gate"] >= GATE_THRESHOLD or wide["named"] >= NAMED_THRESHOLDThree design points are worth keeping an eye on:
- The action question asks "should we act," not "what is the topic?" A topic question can't separate explain what a monad is from requests that need a skill—both are software.
prose_sufficesis a reverse question: a yes answer means precisely that no skill is needed, so take1.0 - vbefore entering the average.nameddoes not enter the average; it opens its own gate. In the official's own words: the three action answers will outvote it; on turns where a method is named but no action is taken (making a meme), it is the only way to pull the request through.
needs_skill provides the two inputs of the truth table: action-gate mean ≥ 0.30 or named ≥ 0.50—only if either passes the line do we continue; if both are low, the recommendation is to load nothing.
Run the official's four DEMO requests through it (outputs are the officially published values):
"Can you save this recipe as a new note in my 'Recipes' folder in Notes.app so "
action 0.83, names a method 0.88 -> suggest (0.14s)
0.990 apple-notes Manage Apple Notes via memo CLI: create, search, edit.
0.010 computer-use Drive the desktop background-first; escalate on signal.
0.000 apple-reminders Apple Reminders via remindctl: add, list, complete.
"I've got a client pitch deck as a .pptx template - I need to duplicate slide 4"
action 0.73, names a method 0.64 -> suggest (0.25s)
0.730 pptx-author Build PowerPoint decks headless with python-pptx.
0.270 microsoft-office Create, read, and edit Word, PowerPoint, and Excel files.
0.000 apple-notes Manage Apple Notes via memo CLI: create, search, edit.
"Post this announcement to my Mastodon account."
action 0.86, names a method 0.61 -> suggest (0.1s)
0.280 xurl X/Twitter via xurl CLI: raw post search, posting, DM, media.
0.240 social-media-content-calendar Plan multi-platform social campaigns: briefs to posting.
0.190 computer-use Drive the desktop background-first; escalate on signal.
"Can you make me a Gru's Plan meme about a developer's plan to refactor the cod"
action 0.12, names a method 0.87 -> suggest (0.12s)
1.000 meme-generation Create meme PNGs from templates with Pillow text overlay.
0.000 apple-notes Manage Apple Notes via memo CLI: create, search, edit.
0.000 apple-reminders Apple Reminders via remindctl: add, list, complete.Each of the four illustrates one thing: the Notes request is unambiguous, and the top-ranked skill is the right one; the deck request has two .pptx skills leading, but under the 60-character description, the deck-building pptx-author comes first—for a request that wants to modify an existing deck, that is exactly the sign of a wrong answer; the Mastodon request passes the action gate at 0.86 as expected (posting is an action), but the top-ranked xurl is a skill for posting to X—ranking cannot rescue something that is not in the catalog; the meme request has a very low action gate at 0.12, relying entirely on named 0.87 to open the gate, and the top-ranked skill is indeed the meme skill.
💡 This is the unique value of Jev
Ranking (one Choice) and gating (four Noul questions) evaluate the same state in parallel and independently within the same request—adding questions barely increases response time, only the tokens for a few extra questions. In the official recording, the first request took 0.1–0.25 seconds. Compare this with "having an LLM judge on every turn": that requires paying for a full generation call each for ranking and for gating, plus parsing the text the judge emits. For the mechanism of multiple questions in one request, see Parallel Questions and Speculative Fan-out.
💡 A note on Chinese-language content
The official statement: English is Jev's primary training language; other languages, including Chinese, work but with lower accuracy. For all the questions above, keeping the original English wording is recommended (the official phrasing was validated through testing); to make skill suggestions for Chinese-language requests, first run a batch on your own data and watch the probability distributions before setting thresholds.
You should see: the gates / rankings of the four DEMO requests match the table above (bit-for-bit identical on cache replay; live runs will differ in probabilities—follow the version you have pinned).
Step 3: The second request—re-check the top three, allow everyone to fail
Why: this is one of the official's three "true two-request" cases: the code must obtain the first request's ranking before it can construct the second request's options—each of the top three carries its full description and the beginning of its SKILL.md.
The principle of Parallel Questions is "ask together whenever you can"—so why fire a second request here? Because the second request's options depend on the first request's answer: each candidate must carry description_full and the first 700 characters of body (the roster file stores 1600 characters per entry), and these long texts can only be assembled into the criteria once the top three are known. The official primitives documentation names this cookbook as a true two-request case: rank all skills in one request, then take the top three's full texts and judge once more against better evidence.
SHORTLIST = 3 # candidates carried from the first request into the second
EXCERPT_CHARS = 700 # SKILL.md characters each candidate brings; the roster file stores 1600
FITS_THRESHOLD = 0.30 # a shortlist whose best "does this fit" noul is under this is dropped
RERANK_INSTRUCTIONS = (
"Exactly one of these skills is the right one to load for the user's latest "
"request. Which one? Read what each actually does, not just its name."
)
def rerank_questions(names, excerpt):
questions = {
"which": Choice(
instructions=RERANK_INSTRUCTIONS,
criteria={
name: f"{BY_NAME[name]['description_full']} — {BY_NAME[name]['body'][:excerpt]}"
for name in names
},
)
}
for name in names:
questions[f"fits::{name}"] = Noul(
instructions=(
f"Does the skill '{name}' do the specific thing the user's request asks "
f"for? It is described as: {BY_NAME[name]['description_full']}"
)
)
return questions
@json_cache
def rerank(request, names, excerpt):
"""Request 2: the same Choice over a shortlist, plus one absolute noul per candidate."""
response = client.system_one(
state=build_state(request),
questions=rerank_questions(names, excerpt),
model=TYPESAFE_MODEL,
)
return {
"winner": response.answers["which"].choice,
"fits": {
key.removeprefix("fits::"): answer.noul
for key, answer in response.answers.items()
if key.startswith("fits::")
},
}fits—the three Noul questions each answer independently and can all be low—they and which decide different things: which determines "which of the three," fits determines "whether to say anything at all." Running the four DEMO requests through the full pipeline (officially published values):
"Can you save this recipe as a new note in my 'Recipes' folder in Notes.app so "
was apple-notes -> apple-notes (0.11s)
fits 0.81 apple-notes
fits 0.50 computer-use
fits 0.02 apple-reminders
"I've got a client pitch deck as a .pptx template - I need to duplicate slide 4"
was pptx-author -> microsoft-office (0.11s)
fits 0.77 pptx-author
fits 0.76 microsoft-office
fits 0.01 apple-notes
"Post this announcement to my Mastodon account."
was xurl -> nothing fits (0.07s)
fits 0.10 xurl
fits 0.27 social-media-content-calendar
fits 0.28 computer-use
"Can you make me a Gru's Plan meme about a developer's plan to refactor the cod"
was meme-generation -> meme-generation (0.16s)
fits 0.84 meme-generation
fits 0.03 apple-notes
fits 0.01 apple-reminders.pptx: the highlight scene flips in the second request—once the two skills are separated and each carries its full text, the deck request flips from pptx-author to the file-editing microsoft-office. Note that its fits values are both not low (0.77 / 0.76—both can do something with a deck), but this question does not decide "which one"; the flip is owed to the Choice.
The Mastodon request stops at the second stage: none of the three similar skills actually does this, and the best fits 0.28 falls below 0.30, so the whole group is abandoned—this request passed the gate both times, yet the final recommendation was still to load nothing; that is what "allowing all to fail" means.
Fold the two requests and two thresholds into one function—that is the entire recipe:
def suggest(request: str) -> tuple[str, ...]:
"""At most one skill name for a request, or () for "nothing here applies"."""
wide = rank_wide(request)
if not needs_skill(wide):
return ()
shortlist = tuple(name for name, _ in wide["ranked"][:SHORTLIST])
result = rerank(request, shortlist, EXCERPT_CHARS)
if max(result["fits"].values()) < FITS_THRESHOLD:
return ()
return (result["winner"],)
def suggestion_block(names: tuple[str, ...]) -> str:
"""What gets appended after the roster, in the suggestion."""
body = (
f"Relevant to the current request: {', '.join(names)}. Ignore this if it does not "
"fit what the user actually asked for."
if names
else "No skill in the roster appears relevant to this request."
)
return f"\n\n<skill_relevance>\n{body}\n</skill_relevance>"
print(suggestion_block(suggest(DEMO[1])))
print(suggestion_block(suggest(DEMO[2])))You should see:
<skill_relevance>
Relevant to the current request: microsoft-office. Ignore this if it does not fit what the user actually asked for.
</skill_relevance>
<skill_relevance>
No skill in the roster appears relevant to this request.
</skill_relevance>Only the winner's name goes into a single line of the system prompt. The agent keeps the full index and its own judgment; that line only tells it "which one to look at first." Both wording details come from real testing (in the official's own words):
- The suggestion says "may be ignored": forcing the suggestion harder also raises the compliance rate for wrong suggestions—and a wrong suggestion is worse than none.
- Even with no suggestion, send a line saying "no relevant skills": if nothing is sent at all, the roster's own "when in doubt, load more" instruction has no counterweight.
Also note the placement: the suggestion block is added after the roster rather than inside it—the roster text stays identical turn after turn, so prefix caching can take advantage of it.
- I can explain what each of the two requests does and why two are required (the second request's options are determined by the first request's answer)
- The Mastodon request passed the gate twice yet ended with a recommendation to load nothing—I can point to which question stopped it
Step 4: Reconciliation—a three-arm live test on 452 requests
Why: the official ran each of the 452 requests three times; the three arms differ in only one thing: what else was stuffed into the system prompt besides the roster.
| What was stuffed into the system prompt | |
|---|---|
| agent alone (baseline) | Nothing |
| agent with a suggestion | The suggestion block returned by suggest() |
| agent given the answer | The correct skill name, or "nothing applies" when there is no covering skill |
The third arm is the floor: no real system knows the covering skill in advance, so it draws a lower bound for the other two arms—even the best selection method cannot beat it.
run wrong loads needless loads
baseline 15.7% 14.0%
TypeSafe 7.9% 5.2%
oracle 3.2% 4.1%
baseline -> TypeSafe: 2.0x fewer wrong loads, 2.7x fewer needless onesThe official summary: wrong loads were cut by more than half (15.7% → 7.9%), closing more than half of the gap between "guessing from a truncated index" and "being handed the answer"; useless loads were cut by nearly two-thirds (14.0% → 5.2%), almost closing that segment of the gap entirely.
The cost is laid out too: of 280 covering requests, the suggestion fixed 29 and broke 7. Some requests the agent would have gotten right on its own were pulled away after a confident wrong suggestion was attached—the official says this is the cost of placing a suggestion before the turn.
You should see: the three-row, two-column error-rate table matches the above (bit-for-bit identical on cache replay; live runs follow the version you have pinned).
- I can explain how each of the three arms differs and why the oracle is a floor rather than a contestant
- I know 7.9% does not come for free: 29 fixed, 7 broken—which is why the suggestion wording includes "may be ignored"
Step 5: Turning it into function calls—closed_sets reads signatures, 54 questions in one request
Why: the second application of the same philosophy: the functions stay untouched, and a spec is added—one question per parameter, one line of description per option, one paragraph of description per function, plus one question to pick the function.
The official's analogy is direct: when you order a "large iced oat latte, no sugar," the barista does not write down your exact words but checks four options on the cup. This cookbook does the same for a trading API—one sentence in, function name and parameter values out, each with a confidence level:
"plot rolling correlation between nvda and spy for the past month"
rolling_correlation(symbol='NVDA', benchmark='SPY', window='1mo') confidence 0.91
"compare nvda amd and msft over the past three months"
compare_returns(symbols=['NVDA', 'AMD', 'MSFT'], window='3mo') confidence 0.94
"show me apple daily with volume"
plot_price(symbol='AAPL', resolution='1d', include_volume=True) confidence 0.75
"what tickers do you have"
list_symbols() confidence 1.00trader.py is ten ordinary functions. Parameters whose values come from fixed lists are already Literal in the type annotations—closed_sets reads the signatures and sorts the fillable parameters into three shapes:
def plot_price(
symbol: Literal["SPY", "NVDA", "AMD", "AAPL", "MSFT", "TSLA"],
style: Literal["line", "candles"] = "line",
resolution: Literal["1m", "5m", "15m", "1h", "1d"] = "15m",
window: Literal["1d", "1w", "1mo", "3mo"] = "1w",
include_volume: bool = False,
moving_average: Literal["9", "20", "50"] | None = None,
log_scale: bool = False,
): ...| Shape | Type annotation | What question it becomes |
|---|---|---|
| choice | Literal[...] | a single Choice question, with the options being the legal values |
| set | list[Literal[...]] | one Noul per member (the {} placeholder is replaced with the member name) |
| flag | bool | a single Noul, ≥ 0.5 means on |
Other types—free text, numbers, dates—generate no questions, and the function's default stands: the third parameter limit: int = 3 of top_movers never generates a question and always keeps the default value 3.
The spec lives in spec.json, stating in one sentence "what the user means by what they said"—Literal only provides the two strings "1mo" and "3mo", and cannot say that "this quarter" refers to the latter. The officials also left one remark: the spec lets the LLM write it for you from the signature. The role questions for rolling_correlation's two same-source parameters are worth reading word for word:
"symbol": {
"question": "Which stock is the user asking about - the one being measured, named first?",
"options": { "SPY": "SPY, the S&P 500 index ETF, the market as a whole", "…": "…" }
},
"benchmark": {
"question": "Which stock is it being compared against - the second one named, the yardstick?",
"stated": "Does the user name a second symbol to compare against?",
"options": { "SPY": "SPY, the S&P 500 index ETF, the market as a whole", "…": "…" }
}symbol and benchmark draw from the same pool of six options, and the role questions put each in its place—"the one being measured, named first" versus "the second one named, the benchmark yardstick." The option keys are exactly the strings the function receives, eliminating a mapping layer.
stated makes a parameter optional: it is a second yes-or-no question, asking "did the command mention this parameter." If not mentioned (below 0.5), the whole parameter is left unfilled and the function's own default takes effect.
Dispatcher builds all the questions from the spec in one go. After that, each command is a single request: the function choice and all parameters of the ten functions are sent together, and the code only reads the answers for the selected function—the questions are all asked before "which function was chosen," which is the ultimate form of speculative fan-out:
assistant = Dispatcher(SPEC, TOOLS, client) # SPEC 来自 spec.json,TOOLS 来自 trader.py
print(f"{len(assistant.questions)} questions per command, among them:")
for qid in ("__tool__", "plot_price.style", "plot_price.style?", "compare_returns.symbols.NVDA"):
question = assistant.questions[qid]
print(f" {qid:<30}{question['type']:<8}{str(question['instructions'])[:64]}")You should see (official release):
54 questions per command, among them:
__tool__ choice What is the user asking the trading assistant to do?
plot_price.style choice Does the user want a plain line or candles?
plot_price.style? noul Does the user say how the chart should be drawn, such as a line,
compare_returns.symbols.NVDA noul Does the user want NVDA in the comparison?The ten functions have 28 fillable parameters in total; adding the function choice and the various stated questions, each command's 54 questions are asked in one request. The questions are written about "ideas" rather than "words the user might say": what is matched is meaning—is amd tracking nvidia lately hits rolling_correlation, even though the words tracking and lately never appear in spec.json at all.
Run the official 14 commands (output is the officially published values):
"show nvda 1h"
plot_price(symbol='NVDA', resolution='1h') confidence 0.78 tool 1.00
"plot rolling correlation between nvda and spy for the past month"
rolling_correlation(symbol='NVDA', benchmark='SPY', window='1mo') confidence 0.91 tool 1.00
"when during the day does nvda trade the most"
intraday_pattern(symbol='NVDA') confidence 0.53 tool 1.00
"what moved today"
top_movers(window='1d', direction='gainers') confidence 0.90 tool 0.90
"what tickers do you have"
list_symbols() confidence 1.00 tool 1.00
"how did the market do this week"
market_summary(window='1w') confidence 0.96 tool 0.99
"candles for tesla with a 20 period moving average"
plot_price(symbol='TSLA', style='candles', moving_average='20') confidence 0.69 tool 0.97
"compare nvda amd and msft over the past three months"
compare_returns(symbols=['NVDA', 'AMD', 'MSFT'], window='3mo') confidence 0.94 tool 1.00
"how volatile is tsla"
volatility(symbol='TSLA') confidence 0.96 tool 1.00
"biggest losers today"
top_movers(window='1d', direction='losers') confidence 0.98 tool 0.98
"worst drawdown for nvda this quarter, and chart it please"
drawdown(symbol='NVDA', window='3mo', plot=True) confidence 0.84 tool 0.84
"spy stats for the last month"
summary_stats(symbol='SPY', window='1mo') confidence 0.88 tool 0.88
"show me apple daily with volume"
plot_price(symbol='AAPL', resolution='1d', include_volume=True) confidence 0.75 tool 0.85
"is amd tracking nvidia lately"
rolling_correlation(symbol='AMD', benchmark='NVDA') confidence 0.82 tool 0.82The two long commands show the value best: rolling correlation filled four parameters in one sentence, with symbol and benchmark each in its place; the compare command put three tickers into the set and left three others outside.
- I can state what question each of the three closed-set shapes maps to, and why
limit: intdoes not appear among the 54 questions - I know why the spec's option keys directly use the strings the function receives
Step 6: read confidence—the weakest judgment determines the certainty of the whole call
Why: a function call's confidence reports the least certain judgment in the entire call, not the product of all judgments—one wrong parameter is enough to ruin the result.
The official wording: the product answers a different question ("is every part correct"), and it gets lower the more parameters there are, regardless of which judgment is actually uncertain. min lets you see the weak spot directly. A parameter-by-parameter accounting for is amd tracking nvidia lately (officially published values):
"is amd tracking nvidia lately" -> rolling_correlation(symbol='AMD', benchmark='NVDA') confidence 0.82
symbol 'AMD' p 0.87 AMD 0.87 NVDA 0.13 AAPL 0.00
benchmark 'NVDA' p 0.78 NVDA 0.92 AMD 0.08 AAPL 0.00
window omitted, default stands p 0.96
resolution omitted, default stands p 0.99
weakest argument: benchmarkwindow and resolution are both omitted—lately says neither how far back to look nor what bar, so rolling_correlation runs with its own defaults: one month, hourly bars. This is exactly what stated is for: without it, the Choice would be forced to specify a window, and it would specify one with full confidence.
The confidence of 0.82 comes from the weakest benchmark (p 0.78). Whether to execute, whether to confirm with the user "is comparing against NVDA okay"—this information is directly usable, with the threshold set in your code.
Finally, tie the two scenarios back to the same principle. On the skill-picking side, the official Agent Skill FAQ says exactly the opposite: if you only care about "picking the best," just take the highest-probability option—no need to set confidence thresholds everywhere; if you have statistical algorithm needs, use probabilities. The two requests gate on "to say or not to say"—that gate is set with the Noul probability itself, and Noul never had a separate confidence to begin with.
You should see: the parameter-by-parameter breakdown matches the table above; weakest argument: benchmark points out the parameter most deserving of suspicion in the whole call.
- I can explain why confidence takes the min rather than the product, and where the default value comes from when stated is omitted
📋 Magic Prompt
The "Prompt" on this page is a template of questions and decision code for the API—copy it, replace the 【】 placeholders (those with ? are optional) and it's ready to use.
📝 Skill-ranking question set (one Choice + four Nouls)
Expected effect: splits "which skill should be loaded" into two groups of judgments—ranking and gating—and gets all the answers in one request.
QUESTIONS = {
"which": Choice(
instructions="【这些技能里哪个适合用户的这一轮请求?】",
criteria={【技能名】: 【一行索引描述】}, # 194 个也放得下,上限 255
),
"gate::acts": Noul(
instructions="【用户是要对文件 / 账户 / 设备 / 线上服务动手,还是只要解释或建议?】",
),
"gate::procedure": Noul(
instructions="【认真回答这个问题的专家要不要查一份具体流程或命令集?】",
),
"gate::prose": Noul(
instructions="【一个什么都接触不到的通才,纯靠文字能把这请求答满意吗?】", # 反向题:yes = 不需要技能
),
"named": Noul(
instructions="【请求点名了具体工具、格式、模板或方法吗?】",
),
}
# 必填项:state = {"request": 【用户这轮的原话】, "recent_context": 【近期对话,可为空串】}
# 选填项:闸门题按你的领域改问法,但保持「问动作,不问主题」?
# 输出规范:probabilities 降序取前三;gate = 三道动作题的均值(prose 取 1-p 再平均);named 单独读
# 约束条件:
# ✅ 问「要不要动手」,别问「主题是什么」——主题题分不开 explain what a monad is
# ✅ 官方英文原文经过了 452 请求实测,中文化前先用自己的数据验证
# ❌ 不要问「该不该建议技能」——那是代码的活,改政策不该改问题✅ Suggested decision for the two thresholds (wrapped up with a truth table)
Expected effect: the two sets of probabilities become "at most one skill name," with all the policy in visible constants.
GATE_THRESHOLD = 【0.30】 # 三道动作 Noul 的均值,官方给这套请求挑的起点
NAMED_THRESHOLD = 【0.50】 # 「点名了方法」单独开门,不进平均
FITS_THRESHOLD = 【0.30】 # 复查里最好的 fits 低于它就整组放弃
SHORTLIST = 【3】 # 第一道带进第二道的候选数
def suggest(request):
wide = rank_wide(request)
if not (wide["gate"] >= GATE_THRESHOLD or wide["named"] >= NAMED_THRESHOLD):
return () # 两个理由都不成立:什么都不建议
shortlist = top(wide["ranked"], SHORTLIST)
result = rerank(request, shortlist) # 第二道:选项 = 前三的全文,真二请求
if max(result["fits"].values()) < FITS_THRESHOLD:
return () # 允许全员落选
return (result["winner"],)
# 必填项:三个阈值——官方说明它们是起点,不是默认值
# 选填项:建议块措辞保留「可以忽略」,并给「没有建议」也发一句话?
# 输出规范:至多一个技能名,或 ();包进 <skill_relevance> 块加在 roster 之后
# 约束条件:
# ✅ 建议块加在 roster 之后,别改 roster 文本——prefix caching 靠它逐回合一致
# ✅ 阈值、问题措辞集中在一个文件里,方便人审(官方 Agent Skill 的建议)
# ❌ 不要把建议写死成命令——自信的错误建议比没有更糟(实测弄坏 7 个本来对的)🔧 Function-calling spec template (one entry per function)
Expected effect: each closed-set parameter in the signature gets its own "ask about the idea" question, so the values entering the function are always legal.
{
"question": "【用户想让这个助手做什么?(挑函数的那道题)】",
"functions": {
"【函数名】": {
"description": "【用户想要这个函数做什么,一段话】",
"arguments": {
"【参数名】": {
"question": "【这个参数用户要什么——按想法问,不按参数名问】",
"stated": "【用户说没说这个参数?】",
"options": {
"【合法值,即 Literal 成员】": "【这个值是什么意思,一行】"
}
}
}
}
}
}
# 必填项:每个 Literal 参数一条;list[Literal] 的 question 里放 {} 占位(每成员一道 Noul)
# 选填项:可选参数配 stated;bool 参数不需要 options(一道 Noul ≥ 0.5 为开)?
# 输出规范:Dispatcher 一次建 54 道问题(本页示例规模),每条命令一次请求
# 约束条件:
# ✅ 选项键 = 函数收的字符串,省掉映射层
# ✅ spec 可以让 LLM 从签名帮你写(官方原话),写完人审一遍
# ❌ 不要拿参数名当问题("Which resolution?")——没给命令任何可匹配的东西Checkpoint ✅
- I can state what each of the two requests does: the first has one Choice ranking 194 skills + four Nouls gating; the second looks only at the top three, rechecks with full descriptions, and allows everyone to lose
- I know why the second request must be sent: the second request's options (the full text of the top three) can only be constructed after the first request's ranking—one of the officials' three true two-request cases
- I can recite the three-arm real test: wrong-load 15.7% → 7.9% (more than halved), useless-load 14.0% → 5.2% (nearly two-thirds); oracle floor 3.2% / 4.1%
- I can name the three closed-set shapes in function calling:
Literal→ Choice,list[Literal[...]]→ one Noul per member,bool→ Noul; numbers and free text get no questions and keep their defaults - I know that confidence in function calling takes the weakest judgment (min) rather than the product, and that when picking skills "just pick the best" there's no need to set thresholds everywhere
Pitfall reminders
❌ Making Jev directly answer "what should the agent do next"
SKILL.md's own words: this kind of question usually means you're asking at the wrong layer. The right approach is to break the decision into the judgments it depends on—ranking, action gate, naming gate, fits—give the judgments to Jev, and write the decision (two thresholds + a truth table) into code, visible and testable. This page's two requests and four constants are the complete demonstration.
❌ Setting confidence thresholds everywhere
This one is right on the official Agent Skill FAQ list: if you only care about "picking the best," just take the highest-probability option. If you have statistical algorithm needs, use probabilities. This page's gates use the Noul probability itself (Noul has no separate confidence); only the function-calling side's confidence is used as a "certainty level," and it takes only the weakest judgment.
❌ Using a topic question as a gate
"Is this software-related" cannot separate explain what a monad is from a software request that needs a skill—both are software. The action question asks "should we act on the user's files, accounts, devices, or online services," and that does separate them. For reverse questions (prose_suffices), remember to take 1-p before averaging.
❌ Using the parameter name as the question
"Which resolution?" is called out by name by the officials: it gives the command nothing to match against. Ask about the idea: "How far back is the user asking about - today, the past week, the past month, or the past three months?"—that way this quarter lands on 3mo, and lately honestly fills in nothing.
Advanced tips (optional)
** 🔧 Point to your own skill directory. ** The official wording: just swap out hermes_roster.json—every question above reads only the four fields name, description, description_full, body; there is nothing else Hermes-specific in suggest().
📚 What about a larger catalog. The official word: one Choice is more than enough at this scale; if it grows several times larger, rank in chunks and re-rank the chunk winners just like in step 3. The Rerank cookbook (Search and Matching) uses the same two-step approach of "shortlist first, then re-score," swapping the cheap first pass for BM25.
⏳ Replay all the numbers with zero keys. Both cookbooks' calls are recorded in their respective json_cache.json, keyed by input, so replay sends no requests at all; delete the cache file and everything runs live. The skill-suggested published recording uses jev-1.14.0 + claude-haiku-4-5-20251001 (2026-10-30 rendering), and function calling uses jev-1.12.
🚀 Let a coding agent write these for you. Install the TypeSafe Agent Skill and hand this page to it to follow along:
claude plugin marketplace add typesafe-ai/skills
claude plugin install typesafe@typesafe-aiThe official advice is to keep questions and threshold constants in a single file for easy human review—the GATE_THRESHOLD / NAMED_THRESHOLD / FITS_THRESHOLD and the four gate questions on this page are exactly the kind of thing that belongs in that one file. See Developing with Claude Code.
Lesson Recap
✅ Skill suggestion = two requests + two thresholds: the first pass uses one Choice to rank all 194 skills, with four Noul gates (three action questions asking "should we act" and one asking "was it named"); the second pass looks only at the top three, each with its full description + the first 700 characters of SKILL.md, and if fits is all low the whole group is abandoned. The output is at most one skill name, with the <skill_relevance> block appended after the roster.
✅ Measured gains and costs: wrong-load 15.7% → 7.9%, useless-load 14.0% → 5.2%; the oracle floor of 3.2% / 4.1% shows that no matter how good, you can't beat "just give the answer." Of 280 coverage requests, the suggestion fixed 29 and broke 7—the suggestion wording including "can be ignored" is not just politeness.
✅ Function calling = one spec + one request: closed_sets reads signatures and sorts closed-set parameters into three shapes—choice / set / flag; 54 questions are asked in a single request, and the code reads only the answers of the selected function. What enters the function is always a legal value from Literal; numbers and free text are not asked and keep their defaults.
✅ Confidence takes the weakest: the confidence of the whole call is determined by the least confident judgment (min, not a product); stated lets unstated parameters be omitted entirely so function defaults take effect; weakest() points out which parameter to doubt.
Key configuration fields recap:
| Field | Value on this page | Description |
|---|---|---|
SHORTLIST | 3 | Number of candidates carried from the first pass into the second |
EXCERPT_CHARS | 700 | Number of leading SKILL.md characters included per candidate (each roster entry stores 1600) |
GATE_THRESHOLD | 0.30 | If the mean of the three action Noul crosses the line, it counts as "skill wanted" |
NAMED_THRESHOLD | 0.50 | "Named the method" opens its own gate separately, not included in the average |
FITS_THRESHOLD | 0.30 | If the best fits in the review falls below it, the whole group is abandoned |
MEMBER / STATED | 0.5 | Threshold for a set member entering the set / an optional parameter being filled |
TYPESAFE_MODEL | jev-1.14.0 (skills) / jev-1.12 (function calling) | Official recording versions; running live locks your own version IDs |
AGENT_MODEL | claude-haiku-4-5-20251001 | The agent under test; official note: pinned to guarantee stable scores |
Related tutorials:
- RAG Passage Gating: Only Let Relevant Content into the Context—previous lesson: the same "probabilities + thresholds in your code" finish, applied to gating the passages that enter the context
- A Basic Memory System: Letting Jev Decide What to Remember—next lesson: the officially defined harness (code surrounding the model that gives it extra capabilities)—this page gave tool calls, that page gives memory
- Developing with Claude Code: TypeSafe Agent Skills—the full version of the "judgment to Jev, decisions stay in code" truth-table philosophy, plus installing and using Agent Skills
Next Lesson Preview
This lesson's harness decides "which tool to call, which skill to pick"; the next lesson adds memory to the harness: A Basic Memory System: Letting Jev Decide What to Remember.
You will learn:
- After a call, a single Noul gate decides "does this request state a customer fact worth remembering," storing only if P ≥ 0.5
- Memory is stored in an ordinary Python dictionary and merged into state before the call—
ask_with_memory()has the same parameters and return values asask(), so swapping it in requires no changes to other code - Official measurements: subsequent ticket routing went from 3/6 to 6/6, with only one extra request per call
- The officially defined harness concept: code surrounding the model that gives it extra capabilities (tool calls, running in a loop)—the
Dispatcheron this page is exactly the kind that adds tool calls to the model