Score Questions: Rate Against a Scale You Define

The Choice from the previous lesson picks one from fixed options. But many judgments are not "which one to pick" but "to what degree": how severe a bug is, how angry a customer is, how proficient a candidate is in Python.

Use Score for these questions: you provide an ordered scale, and Jev places the content at a position on the scale, returning it along with the probability of each level and the confidence.

In fact, you have already seen it in the first request—the frustration in the ticket example is a three-level Score question that returned 0.67. This lesson covers it thoroughly, taking 10 minutes in total.

What You'll Be Able to Do After This Lesson

• Use criteria to define an ordered scale (at least two levels recommended, up to 10 levels), and have Jev Score rate any content against your scale
• Read the five returned fields: score, legend, probabilities, confidence, type
• Explain clearly how a score like 1.43 that "falls between two levels" is calculated
• Use score to pass thresholds, and use confidence to decide whether to escalate to a human
• Split multi-dimensional judgments into multiple Score questions, normalize and combine them with weights in code

Your Current Predicament

• Having GPT rate tickets on a 1–10 scale—the same ticket gets 6 this time and 8 next time, so the threshold for "escalate to a human above what score" can never be settled
• The rating criteria are left to the model's free interpretation: my "moderately severe" doesn't match what it understands, and changing one version of the prompt means all historical scores have to be re-rated
• Getting a 7 and calling it done—with no idea whether it's a rock-solid 7, or a 7 pieced together from a fifty-fifty split between 6 and 8, so which ones need human review is pure guesswork

When to Use This Technique

When you need to:

• Position content on a spectrum: severity, frustration level, skill level, report quality
• Have scores go directly into code: threshold checks, sorting, weighted sums
• Use each level's probability and the overall confidence as a signal for "should a human take a look"

And you don't want to:

• Let a chat model freely "give a 1–10 score" and then write parsing code to scrape its text output
• Mix rating criteria into the prompt, so the criteria drift whenever the model version changes
• Let it run automatically even when the model isn't confident

🎒 Preparation Before Starting

• Completed State: know that the content to be scored goes in state (a string, JSON object, or array of texts all work)
• Ran the first request successfully: .venv has typesafe-sdk installed, and TYPESAFE_API_KEY is in the environment variables

Core Idea

Score answers "to what degree." Official applicable scenarios: how severe a bug is, how satisfied a customer is, how much Python experience a candidate has. The answer lies on a spectrum, and you can describe that spectrum level by level—this is the prerequisite for using Score.

The scale is criteria. A Score question has only three fields:

Field	What to Write	Example
type	Fixed as "score"	—
instructions	One sentence stating what is being rated	How severe is the reported issue?
criteria	Ordered list of levels, from low to high, one description per level	See the three-level scale in Step 1

The level numbers are just array indices, starting from 0: a three-level scale is 0, 1, 2. All the model sees is the text description of each level—it cannot see the level numbers or the adjacent levels.

Five fields are returned.

Field	What It Is
score	Position on the scale, 0 to the highest level number; can fall between two levels, e.g. 1.43
legend	Mapping of level number → level description, to help you interpret the score
probabilities	Probability of each level, summing to approximately 1
confidence	0–1; the more concentrated the probability is on a single level, the higher it is
type	"score"

score is calculated, not guessed. Each level number is multiplied by its probability and then summed: 0×0.0 + 1×0.57 + 2×0.43 = 1.43.

How to choose among the three question types.

Question type	Answer form	Usage in code
Choice	Pick one from fixed options, with no ordering among options	Maps to N code paths
Score (this lesson)	A position on a spectrum, where you can describe each level	Thresholding, sorting
Noul	A clean yes/no, where the probability itself is the signal	A single if

Official criteria: if the answer lies on a spectrum and each level can be described → Score; if the options are fixed with no ordering → Choice; if it's a clean yes/no → Noul. When it resembles both, pick the one whose answer can be acted on directly by code. And don't use Noul to ask "is this candidate strong at Python"—when "strong" is undefined, a noul of 0.5 just means half right and half wrong, not "intermediate level"; to measure level, use Score with defined levels.

Follow along with me

Step 1: Define your Jev Score scale (ordered levels in criteria)

Why: When the model judges, it only has access to the level descriptions you write—it cannot see the level numbers or the neighboring levels. Write the descriptions as "situations" so the model can match them against the state; if you write numbers or degree words like "moderately severe," it has nothing to compare against (see Pitfall Reminder item 1 for the consequences).

This lesson uses the bug severity scale from the official documentation, with three levels:

0: Cosmetic; no impact to functionality
1: Broken or degraded feature, but workaround exists
2: Blocking issue; no workaround exists

From low to high: level 0 "cosmetic issue, does not affect functionality," level 1 "functionality is broken but there is a workaround," level 2 "completely blocking, no workaround."

Official guidelines for writing a scale:

• Describe situations, not degrees: "functionality is broken but there is a workaround" can be matched against a setting, "moderately severe" cannot
• Write as many levels as you can describe clearly, up to 10 levels; three levels is very common
• One question tests only one dimension—"punctual, smart, experienced" is three questions (see Pitfall Reminder item 2)
• If the top of the scale has rare cases requiring special handling, give them their own level (see Pitfall Reminder item 4)

You should see: three level descriptions, each one a concrete situation you can directly compare against a bug report.

• Each of my level descriptions is a situation, with no "mild/moderate/severe" degree words and no bare numbers

Step 2: Send your first Score request with typesafe-sdk

Why: First get the official single-question example running as-is, then swap in your own scale and logic—when debugging, you always have a "known-correct" baseline.

Create score_bug.py and paste in the official example:

from typesafe_sdk import Score, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="The export button crashes the settings page in Safari. It works in Chrome, but a few of our customers only use Safari.",
        questions={
            "bug_severity": Score(
                instructions="How severe is the reported issue?",
                criteria=[
                    "Cosmetic; no impact to functionality",
                    "Broken or degraded feature, but workaround exists",
                    "Blocking issue; no workaround exists",
                ],
            ),
        },
    )

    print(response.answers["bug_severity"].score)

Run it:

.venv/bin/python score_bug.py

You should see:

1.43

1.43 falls between level 1 and level 2. The official interpretation of this example: the export crashes in Safari, and most customers can get by with Chrome (level 1), but customers who only use Safari cannot (level 2)—the model splits between the two levels, leaning toward level 1. The example values may change with Jev versions, so don't worry about differences after the decimal point.

• I see a number falling between two levels (the example value is 1.43)

💡 Can Chinese content be scored

Yes. The official note: English is the primary training language and gives the best accuracy; other languages, including Chinese, work but with lower accuracy. Before putting a Chinese-language scenario into production, first run a batch with your own data to see how it performs, focusing on the confidence distribution.

Step 3: Read the returned score, legend, and probability distribution

Why: In real projects a score is never used on its own—the legend helps you interpret it, and probabilities and confidence tell you how "solid" the score is.

The full return of the official example:

{
  "model": "jev-1.13.0",
  "answers": {
    "bug_severity": {
      "type": "score",
      "score": 1.43,
      "confidence": 0.35,
      "legend": {
        "0": "Cosmetic; no impact to functionality",
        "1": "Broken or degraded feature, but workaround exists",
        "2": "Blocking issue; no workaround exists"
      },
      "probabilities": {
        "0": 0.0,
        "1": 0.57,
        "2": 0.43
      }
    }
  },
  "usage": {
    "input_tokens": 332,
    "output_tokens": 18
  }
}

You should see: The answer object has five fields; score matches the 1.43 you printed, and legend maps 0/1/2 back to the three level descriptions you wrote. The "model": "jev-1.13.0" in the example is the version used when the official example was recorded; when you run it now with jev-latest, this field shows the current model version.

Break it down one by one:

Field	Value in this example	How to read it
score	1.43	0×0.0 + 1×0.57 + 2×0.43: the probability-weighted position
legend	0 → cosmetic, 1 → has a workaround, 2 → blocking	The level numbers map back to the level descriptions
probabilities	level 1 0.57, level 2 0.43	the model splits between the two levels
confidence	0.35	The probability is spread across two levels, hence the low value

In the HTTP response, the keys of legend and probabilities are the strings "0"/"1"/"2"; the Python SDK has already converted them to integer keys (answer.legend[1], answer.probabilities[1]), and all fields of ScoreAnswer are typed.

The same question, five different tickets (a comparison example from the official documentation):

Ticket content	score	confidence	Probabilities (level 0/1/2)
The export button is misaligned by a few pixels	0.0	1.0	1.0 / 0.0 / 0.0
Clicking PDF export does nothing; you can still export CSV and convert it yourself, but it takes a lot of time	1.0	1.0	0.0 / 1.0 / 0.0
The export keeps spinning and never finishes; some colleagues' CSV still works, some are broken too	1.11	0.84	0.0 / 0.89 / 0.11
Safari crashes, Chrome works fine, and some customers use only Safari	1.43	0.35	0.0 / 0.57 / 0.43
The whole team can't log into the system since this morning; every attempt returns a 500	2.0	1.0	0.0 / 0.0 / 1.0

Two key insights:

• The same score can come from different distributions. 1.0 could mean "all-in on level 1", or it could mean "half level 0 and half level 2"—same score, completely different meaning. So read it together with probabilities and confidence.
• confidence 1.0 describes the distribution, not a guarantee that "the answer is correct". When the distribution is all-in on one level it is 1.0; when the model can't distinguish between levels, it drops.

A decimal score is just a position: you can use it directly for sorting (queuing by severity); when code needs an explicit tier, round to the nearest level and act on that—the Search and Matching scenario page has a practical example of making decisions by "rounding to the nearest level".

Step 4: Set thresholds by score and confidence—handle automatically or escalate to a human

Why: A score only counts as being used once it turns into action. The official pattern: if score passes the threshold, run the automated flow; low confidence means the model isn't sure, so escalate to a human.

Building on the response from step 2, write:

answer = response.answers["bug_severity"]

if answer.score >= 1.5:
    action = "P1：立即处理"
elif answer.confidence < 0.5:
    action = "转人工复核"
else:
    action = "进正常队列"

print(action)

You should see (for the Safari ticket):

转人工复核

score 1.43 doesn't pass 1.5, and confidence 0.35 is below 0.5—the model is split between two levels, exactly the signal that a human should take a look.

1.5 and 0.5 are illustrative values. The official three-tier usage: high confidence executes automatically; medium means confirm, flag, or ask for more info; low means do nothing—escalate to a human or fall back.

Thresholds scale with risk: in the official example, 0.5 is the fallback baseline, and high-risk operations like approve_transfer require > 0.9 for unattended execution. The full formulas and examples are in the Confidence lesson.

Step 5: Split into multiple atomic Score questions and combine them with weights in code

Why: "Should this ticket be prioritized" is a multi-dimensional judgment—severity, customer frustration, and report quality each contribute a bit. The official approach: one Score question per dimension, sent in parallel in a single request, combined with weights in code. Changing weights means changing code—no prompt retuning needed.

from typesafe_sdk import Score, TypeSafeClient

TRIAGE_QUESTIONS = {
    "severity": Score(
        instructions="How severe is the reported issue?",
        criteria=[
            "Cosmetic; no impact to functionality",
            "Broken or degraded feature, but workaround exists",
            "Blocking issue; no workaround exists",
        ],
    ),
    "frustration": Score(
        instructions="How frustrated is the customer?",
        criteria=[
            "Calm, just stating facts",
            "Frustrated but civil",
            "Very angry, strong language or threatening to leave",
        ],
    ),
    "report_quality": Score(
        instructions="How much does the report give an engineer to work with?",
        criteria=[
            "No detail; just says something is broken",
            "Names the feature but no steps or environment",
            "Steps to reproduce or environment, but not both",
            "Steps to reproduce and environment",
        ],
    ),
}


def normalized(answers, question_id: str) -> float:
    """Put a score on 0 to 1 by dividing by its top level number."""
    top_level = len(TRIAGE_QUESTIONS[question_id].criteria) - 1
    return answers[question_id].score / top_level


def priority(ticket: str) -> float:
    with TypeSafeClient() as client:
        response = client.system_one(
            state=ticket,
            questions=TRIAGE_QUESTIONS,
        )
    answers = response.answers

    severity = normalized(answers, "severity")
    frustration = normalized(answers, "frustration")
    report_quality = normalized(answers, "report_quality")

    # A detailed report helps an engineer investigate, so it raises priority a little.
    return 0.6 * severity + 0.3 * frustration + 0.1 * report_quality

Two details:

• Normalize before combining. The max score on a three-level scale is 2, on a four-level scale it's 3—without normalization, four-level questions are inherently heavier. Divide each question by its own highest level number len(criteria) - 1 so everything falls into 0–1, and only then do the weights mean something: severity with weight 0.6 is exactly twice the frustration with 0.3.
• Add questions, not requests. The official example (the same "spinning" ticket from step 3 with more context added) returns: severity 1.24 (confidence 0.64), frustration 1.28 (0.58), report_quality 3.0 (1.0); after normalization 0.62 / 0.64 / 1.0, priority = 0.6×0.62 + 0.3×0.64 + 0.1×1.0 = 0.66. Multiple questions evaluated in parallel add almost no response time, only a few extra question tokens.

You should see: priority() returns a number in 0–1; the official example ticket computes to 0.66. Later, if the ordering feels wrong, just adjust the weights and rerun; to add a new dimension, add one question to TRIAGE_QUESTIONS—the number of requests stays the same.

This "split into atomic scores, weight in code" approach is the official Composite scoring pattern, fully covered in the Composite Scoring lesson.

Checkpoint ✅

• I know the level number is just the array index of criteria, starting from 0
• I got 1.43 and can verify it by hand: 0×0.0 + 1×0.57 + 2×0.43
• I know the same score can come from different distributions, so I read probabilities and confidence together
• Every level in my scale is a scenario description, and each question tests only one dimension
• I know multi-dimensional judgments need normalization (÷ highest level number) before weighted combination

Pitfall reminders

❌ Only numbers in the scale

instructions: "Rate severity from 0 to 2, where 2 is worst"
criteria: ["0", "1", "2"]
→ score 0.55, confidence 0.33, probabilities 0: 0.45, 1: 0.55, 2: 0.0

The official team ran this experiment: the same "misaligned button" ticket scored 0.55 with confidence 0.33 on a pure-number scale—probabilities scattered between levels 0 and 1; after switching to a three-level scenario description, score 0.0, confidence 1.0.

The model only matches descriptions against the state; a question like "give it a 1–10 score" doesn't even work well for Jev itself. The right approach: write a matchable scenario for each level, and don't count on the numbers in instructions doing anything.

❌ One question testing several things

If a level description reads "punctual, smart, experienced", the question is testing three things at once—inputs where one is high and another is low have nowhere to land, confidence drops, and the meaning of the score gets muddled. The right approach: one Score per thing, combine the results in code (the approach in step 5).

❌ Using the score without looking at the distribution

score 1.0 could be all-in on level 1 (high confidence), or half level 0 and half level 2 (low confidence). The official docs also give a comparison of how the split position affects confidence: on a three-level scale, probabilities (0, 0.5, 0.5) give confidence 0.25, while (0.5, 0, 0.5) gives 0—a split between adjacent levels is milder, a split across the two ends of the spectrum is worse.

The right approach: read score together with probabilities and confidence. A low confidence usually means the levels overlap, the question tests more than one thing, or the state information is insufficient — hand off to a human or add information instead of forcing automatic execution.

❌ Rare extreme cases don't have their own level

The top of an emotion scale is "very angry," and abusive or threatening messages also cluster near the top — score alone can't separate them, yet these are precisely the messages that most need special handling. The official approach: give rare extreme cases their own level, e.g. add "abusive or threatening" above "very angry."

For more failure modes (literal interpretation, unreliable counting, date comparisons should be done in code), see the Jev 1.13 pitfalls handbook.

Advanced techniques (optional)

Levels can be written as objects. If the model keeps landing between two levels on inputs you consider clear, change each level from a string to an object — one field says what the level covers, and another gives a few example cases (keep the field names consistent across levels):

"1": {
  "what": "Broken or degraded feature, but workaround exists",
  "examples": ["export fails in one browser but works in another"]
}

The official team ran a comparison using the same Safari ticket: with plain strings, score 1.43 and confidence 0.35; after adding a fitting example ("export fails in one browser but works in another"), they became 1.03 and 0.96; when the added example was unrelated to browsers, they stayed at 1.43 and 0.35.

Examples only help if they resemble your real inputs; a higher confidence doesn't mean the answer is correct — after adjusting level descriptions, test again with other inputs before finalizing.

Try the scale without writing code. Paste the state and add questions in the console Playground (https://console.typesafe.ai/playground) for a code-free experience.

Bypass the SDK and call the API directly. POST https://api.typesafe.ai/v1/systemone; the raw JSON for a score question looks like this (official example):

{
  "model": "jev-latest",
  "state": "I was charged twice for my subscription this month. Please refund the extra charge.",
  "questions": {
    "urgency": {
      "type": "score",
      "instructions": "How urgent is this message?",
      "criteria": ["Can wait", "Needs attention this week", "Needs attention today"]
    }
  }
}

For the complete definition of field constraints (the item limit for the criteria array, the five answer fields), see the API quick reference.

Lesson summary

✅ How to use Jev Score: define an ordered scale in criteria (at least two levels recommended, up to 10 levels), Jev places content on the scale and returns five fields.

✅ The five fields: score (the probability-weighted position, which can fall between two levels), legend, probabilities, confidence, type.

✅ Getting scores into code: apply thresholds, sort; hand off to a human when confidence is low instead of forcing automatic execution.

✅ Break complex judgments into single-dimension Scores: normalize (÷ by the highest level number) and combine with weights — this is the composite scoring pattern.

Key configuration fields recap:

Field	Recommended way to write it	Description
type	"score"	Fixed value
instructions	One sentence stating clearly what is being assessed	e.g. How severe is the reported issue?
criteria	Level descriptions from low to high; ≥ 2 levels recommended, up to 10 levels	Level number = array index, starting from 0; one dimension per question
Answer access	response.answers["id"].score	In the SDK, legend / probabilities are keyed by integer level number

To see Score used in real scenarios: Search and matching (using Score for re-ranking, rounding to the nearest level); for the idea of using question types in combination to gate what content enters the context, see RAG passage gating.

Next lesson preview

At the other end of the spectrum is a clean yes/no: Noul yes/no questions: get a true probability from 0 to 1.

You will learn:

• noul is the probability of 0–1; the answer has no separate confidence
• 0.5 means fifty-fifty, not "moderate degree" — degree should be measured with this lesson's Score
• Starting thresholds: yes ≥ 0.7, no ≤ 0.3
• When to replace Score with Noul, and when to ask both questions together