Skip to content

Performance breakdown: how the 7.073 seconds were measured ​

In the previous lesson you watched the agent finish a flight search in 7 seconds with your own eyes. This lesson takes the stopwatch apart — 7.073 seconds is not a marketing line, it is a recording with complete measurement boundaries: the officials not only provided a report, they also put the raw measurement JSON into the repository, complete with the source code hash and browser version. Once you can read the Jev Ultrafast performance data properly, you can not only reproduce it, but also copy this methodology to measure your own agent.

The numbers in this lesson mainly come from three files in the jev-ultrafast repository you cloned: docs/performance.md (the report body), docs/flights-measurement.json (recording data), and docs/full-speed-measurement.json (comparison experiment data).

What you'll be able to do after this lesson ​

  • State the timing methodology for the 7.073 seconds: the clock starts at the first prediction and stops when DONE is accepted; what the five on-clock items and three off-clock items are
  • Break down the recording bill: 17 decision requests (median 178 ms, total 3,720 ms), 2 text generations, 11 actions, and at which second each action executed
  • Understand the design of the 9.450 → 7.092 second comparison experiment, and explain why the officials themselves said "3 pairs is too few, not a general benchmark"
  • Explain clearly what the optimization cut out: browser protocol calls 1,092 → 101, of which 828 DOM.resolveNode calls went straight to zero
  • Use the trace from examples/flights.py to align to the same methodology and measure your own seconds

Your current predicament ​

  • "The officials say 7 seconds, I get 12 seconds — isn't this number probably cherry-picked from the best run?" Without knowing the methodology, any comparison is meaningless
  • You want to cite this set of ultrafast speed numbers in a report, but falter the moment someone asks "how was it measured"
  • You want to run a performance comparison for the browser agent you built, but don't know which variables to control: the model? the network? the browser? You can't even find a template to copy from

When to use this technique ​

When you need to:

  • Relay or cite Jev Ultrafast's performance numbers, and withstand the follow-up question "how was it measured"
  • Run your own performance comparison for a browser agent and need a ready-made measurement template (methodology, independent verification, sample size statement) to copy
  • Judge whether this set of numbers fits your scenario: it is single-task, on a live network, with a small sample

And you don't want to:

  • Treat seconds from a single run as a stable metric and cite it everywhere
  • Compare numbers measured under different methodologies (inspector with screenshots vs. script without screenshots; jev-1.13 vs. jev-1.14)

🎒 Preparation before starting ​

  • Completed Running Google Flights successfully: have run examples/flights.py at least once and seen "passed": true in the verification JSON and artifacts/flights/latest/
  • Remember the "two decisions per round trip" from Jev Ultrafast principles — each of the 17 requests in this lesson follows this structure
  • (Optional) Keep the jev-ultrafast repository open and read along with docs/performance.md and the two measurement JSON files; every number in this lesson can be found there
  • A glass of water — this lesson is mostly reading; hands-on work comes in "Advanced tips"

Core idea ​

The officials' attitude toward performance numbers can be compressed into three sentences:

  1. Every number comes with its methodology — from which millisecond the 7.073 seconds start and stop, and what counts as on the clock versus off the clock, the documentation spells it all out
  2. Comparisons must control variables — old and new versions alternately run 3 times each, same task, same model, same budget, same independent verification
  3. Proactively declare limitations — sample size, live network, single task: the officials say it themselves first, waiting for no one to call them out

The four sets of official numbers this lesson will break down:

NumberOfficial methodologySource
7.073 sfirst prediction → accepted DONE; includes model calls, text generation, browser operations, stale decisions, loadingdocs/performance.md, flights-measurement.json
9.450 → 7.092 s6 alternating runs, median taken; both 3/3 verified independentlydocs/full-speed-measurement.json
1,092 → 101median of browser protocol calls from the same comparison experimentdocs/full-speed-measurement.json
$0.00006272OpenRouter billing for only two text helper calls, excluding TypeSafedocs/performance.md

One more thing to keep in mind: all of these numbers come from the same combination—decision model jev-1.13.0 + text model inception/mercury-2.5 (via OpenRouter, reasoning disabled). In the measurement JSON, TYPESAFE_MODEL is null, meaning the code default jev-latest was used (jev_ultrafast/model.py line 108), which at the time resolved to 1.13.0; today jev-latest already points to jev-1.14.0. If you reproduce it today, the model will be different, so the seconds will naturally differ.

💡 Why this lesson is worth a careful read

More useful than memorizing the number 7.073 is the paradigm of "methodology + billing + sample size declaration." You can copy it wholesale when measuring the performance of any agent yourself—whether Ultrafast or anything else.

Follow along with me ​

Step 1: Align the timing methodology—from which millisecond does the official clock start ​

Why: Align the methodology before comparing any two timings, otherwise every comparison is invalid. Whether you run 9 seconds or 12 seconds yourself, the first step is to figure out how the official clock runs.

The official methodology, laid out in one table:

ItemContent
Clock startthe first prediction after the initial homepage observation
Clock stopthe moment DONE is accepted
On the clockmodel calls, text generation, browser operations, stale decisions (predictions discarded and redone), loading waits
Off the clockbrowser startup, initial navigation, the standalone verify() check after the run completes

This methodology matches the code (jev_ultrafast/agent.py):

  • started_at is set only on the first predict—so the initial page observation is off the clock
  • DONE updates elapsed_ms when accepted in act—that's the clock stop point
  • An expired prediction (StalePage) discards the decision, re-observes, and re-predicts—the discarded time stays on the clock

The video uses the same methodology: 1× original playback speed, no opening pause, only a 0.5-second pause at the end; 186 continuous screen-recording frames plus 1 initial screenshot (source_frames: 187 in the JSON), with only the top 64 px account/navigation bar cropped out—no trimming of the head or tail.

You should see: the timing section in docs/performance.md corresponds to the table above item by item—"begins at the first prediction after initial homepage observation and ends at the accepted DONE choice", matching what's on and off the clock word for word.

  • Be able to name the five on-the-clock items and the three off-the-clock items

Step 2: Break down the bill for 7.073 seconds—where did 17 requests go ​

Why: The officials wrote every number from this recorded run into docs/flights-measurement.json, so the bill can be copied directly. The totals:

MetricValue
Total time7,073 ms
Decision requests17 requests
Decision latencymedian 178 ms, total 3,720 ms
Actions executed11 (10 interactions + 1 explicit WAIT)
Text generation2 times (Zurich 581 ms, London 346 ms)
token90,558 input + 6,325 output
Decision modeljev-1.13.0

Where the time went (the first two rows are raw fields from the official JSON; the third row is this page's calculation: 7073 − 3720 − 927):

DestinationTimeShare
17 decision requests3,720 ms≈ 53%
2 text generations927 ms≈ 13%
Browser execution, observation, page loading≈ 2,426 ms≈ 34%

17 requests don't equal 17 actions: 11 actions plus 1 accepted DONE make 12 predictions "spent where it counts"; the remaining 5 were intercepted by the freshness check before execution—the page fingerprint changed, so decisions based on the old snapshot were discarded and redone. These are the stale decisions in the methodology, and their time is likewise counted in the total seconds. (5 = 17 − 11 − 1; this page derives it from agent.py's loop logic of "each prediction is consumed exactly once, and a discarded one triggers a re-measure"—the officials did not publish the stale count separately.)

Positions of the 11 actions on the timeline (executed_ms, taken from the actions array in the measurement JSON):

TimeAction
0.336 sCLICK Change ticket type. Round trip
0.640 sCLICK One way
1.595 sTYPE_TEXT Where from? → Zurich (text generation 581 ms)
1.930 sCLICK Zürich, Switzerland (selecting the autocomplete suggestion)
2.748 sTYPE_TEXT Where to? → London (text generation 346 ms)
3.053 sCLICK London, United Kingdom
3.452 sCLICK Open Departure
3.769 sCLICK Sunday, September 20, 2026
4.097 sCLICK Done. Search for one-way flights…
5.217 sCLICK Search (official wording: the search was executed at second 5.217)
5.587 sWAIT for the page to update
7.073 sDONE accepted, stop the clock

The first 5.2 seconds were spent filling in and submitting the form; the last 1.9 seconds waiting for results and confirming completion. WAIT executed at 5.587 seconds, DONE accepted at 7.073 seconds — for this final roughly 1.49 seconds, the official explanation is results-page loading, state changes, and the completion decision. The two text generations each caused a "typing pause" of about 0.3–0.6 seconds, which is why in the video the input field seems to "think for a moment before text appears".

You should see: flights-measurement.json the four top-level fields elapsed_ms, decision_requests, decision_median_ms, decision_total_ms, and the executed_ms of the 11 entries in the actions array — the timeline above was copied from this array.

  • Be able to say how many of the 17 requests became actions, how many were stale, and how many were DONE

Step 3: Understand the Jev Ultrafast performance comparison experiment: 9.450 → 7.092 s ​

Why: a single 7.073-second run is just a recording; the conclusion "25% faster" comes from a separate controlled experiment. Its design determines how strong the conclusion can be.

Experiment design (the "Matched runtime comparison" section of performance.md):

  • 6 runs, alternating: one baseline run, one optimized run, 3 pairs in total; the same task, the same existing Chrome profile
  • The two arms used exactly the same: natural-language goal, independent result checker, 1120×780 viewport, jev-1.13.0, mercury-2.5 (reasoning off), action and request budgets
  • Initial navigation was untimed on both sides; each run opened and closed its own tab; all 6 runs counted, none discarded, no vendor or verification failures
  • The baseline arm was frozen at commit 68c077bf…; both sides used Mercury, avoiding mixing the two variables of "swapping the text model" and "changing the code"

Results:

PairBaselineOptimizedVerification
111.214 s6.964 sBoth passed
28.984 s7.913 sBoth passed
39.450 s7.092 sBoth passed
Median9.450 s7.092 seach 3/3

Three median changes: task time −25.0%, TypeSafe requests 22 → 17, browser protocol calls 1,092 → 101. The optimized version was faster in all three pairs.

But the officials immediately gave three disclaimers themselves:

  1. Too few pairs (3); a two-sided sign test gives p = 0.25, which cannot support a strong statistical conclusion
  2. This is a controlled small-sample comparison, not a broad agent benchmark
  3. Google, network responses, routing, and browser cache were all live and not controlled

⚠️ The correct way to cite these numbers

"In 3 pairs of alternating runs on the same task, same model, same browser, the optimized version was faster every time, with a median speedup of 25%" — that is a supported claim. "Jev Ultrafast is 25% faster than some agent" — not supported; the officials never ran a cross-agent comparison.

You should see: the comparison table in performance.md is just the 3 pairs of numbers above plus a bolded median row; "p = 0.25" and "not a broad agent benchmark" are both in the original text.

  • Be able to say which variables the experiment controlled and which it did not

Step 4: Find where the time went — what the 1,092 → 101 protocol calls cut out ​

Why: the cause of the "25% faster" result lies entirely in the browser protocol calls. The officials stored the complete CDP (Chrome DevTools Protocol) call details for every run in the measurement JSON — the hardest set of evidence in the whole course.

The three burdens of the old loop (performance.md, "Where the time went"):

  1. Every DOM change — including animations — invalidated the decision and triggered repeated re-prediction
  2. Repeatedly reading the entire accessibility tree (Accessibility.getFullAXTree)
  3. Parsing hundreds of DOM nodes (DOM.resolveNode)

The new loop's approach (performance.md "Where the time went" + README):

  • One browser call reads common HTML/ARIA controls — no more repeatedly walking the tree and parsing nodes one by one
  • The click guard only compares the target, nearby context, and document/form state; geometry and hit testing are still done before input
  • combobox event waiting: autocomplete suggestions must arrive first (README: cap of 200 ms; other interactions take at most two animation frames or 50 ms) before asking Jev
  • Native text replacement explicitly issues the browser's select-all command

CDP call breakdown (listed one by one across three runs, from the cdp object of each run in full-speed-measurement.json):

Protocol methodBaseline (3 runs)Optimized (3 runs)
DOM.resolveNode828 / 828 / 8280 (key no longer appears)
Accessibility.getFullAXTree30 / 30 / 300
Runtime.callFunctionOn40 / 40 / 400
DOM.getBoxModel + DOM.getNodeForLocation10 + 10 / 10 + 10 / 10 + 100
Runtime.evaluate158 / 146 / 14875 / 77 / 71
Input.dispatchMouseEvent / KeyEvent / insertText20 / 4 / 2 (identical each time)20 / 4 / 2 (unchanged)
Total1102 / 1090 / 1092 (median 1,092)101 / 103 / 97 (median 101)

Two insights from reading the table:

  • The way of "looking at the page" changed, but "clicking the page" did not — the three input call types (mouse, keyboard, text insertion) are completely identical on both sides; everything cut was observation-type calls
  • Fewer requests too (median 22 → 17): decisions are no longer frequently invalidated by animations, so re-predictions naturally decrease

💡 This is the quantitative evidence for "one snapshot, one browser call"

The principles lesson covered the README design principle "One browser call per snapshot". In numbers: Runtime.evaluate dropped from about 150 calls to about 75, and DOM.resolveNode went from 828 calls to zero. The architectural claim and the measured data line up.

You should see: in full-speed-measurement.json, the cdp breakdown of the baseline's three runs shows DOM.resolveNode at 828 calls each; the cdp breakdown of the optimized version's three runs doesn't contain this method at all.

  • Be able to name the three types of observation calls that were cut and why input-type calls are unchanged

Step 5: Work out the cost — tokens, the text helper, and the price the officials didn't report ​

Why: a performance read can't stop at seconds; you also need to know how much this run cost. The officials reported half of it, and the other half can be derived yourself.

What the officials reported:

  • All requests in the recording totaled 90,558 input tokens + 6,325 output tokens
  • OpenRouter billed the two text calls at $0.00006272 (Zurich $0.0000293 + London $0.0000334)
  • The officials explicitly state: this is only the cost of the text helper, not the total task cost — the TypeSafe response contains only token counts, no dollar bill, and browser costs are not included

What the officials didn't report but you can derive yourself (derived on this page, not official figures):

At TypeSafe official pricing ($42 / B input tokens, output free), 90,558 input tokens ≈ $0.0038. Derivation: 90558 × 42 ÷ 10⁹ ≈ 0.0038 dollars. Adding the text helper's $0.00006272, this 7-second task's total model cost is roughly 0.4 cents.

One example of variance (all in full-speed-measurement.json): for the same London generation, the recorded run took 346 ms and 94 completion tokens ($0.0000334); the optimized run of pair 2 in the comparison experiment took 1,137 ms and 511 tokens ($0.0000961) — a single text call's latency can differ by more than 3x, and all such variance goes into the total seconds.

You should see: the "$0.00006272" section in performance.md says "text-helper charge, not total task cost"; the unit prices of the two calls are in flights-measurement.json's text_calls (2.93e-05 + 3.342e-05).

  • Be able to explain clearly what $0.00006272 includes and what it doesn't

Checkpoint ✅ ​

  • Be able to state the full timing methodology: start point, stop point, the five in-table items, the three out-of-table items
  • Be able to recite the four key figures of the recorded bill: 17 requests, median 178 ms, 11 actions, search executed at second 5.217
  • Be able to explain the 9.450 → 7.092 second experiment design, and state the officials' own three disclaimers
  • Be able to name the three types of calls that went to zero among protocol calls 1,092 → 101
  • Be able to explain clearly that $0.00006272 is only the text helper cost, and the TypeSafe bill must be calculated separately from the price list

Pitfall warning ​

❌ Using your own seconds to directly challenge the official numbers ​

Symptom: you run 9 seconds, 12 seconds, and suspect the 7.073 seconds are inflated.

Why: at least four measurement discrepancies — ① Different models: the measurement used jev-1.13.0, while today jev-latest already points to jev-1.14.0; ② The network is live: the official optimized version itself spread across 6.964–7.913 seconds over three runs; ③ The inspector was taking screenshots, which is not a timed run to begin with (as covered in the previous lesson); ④ Your run may have hit more stale decisions.

The right approach: run examples/flights.py (no screenshots) and read artifacts/flights/latest/state.json's top-level elapsed_ms — that is the number on the same basis as the official one; align the model, URL, and goal before comparing.

❌ Presenting "25% faster" as a cross-agent conclusion ​

The claim: "Jev Ultrafast is 25% faster than some browser agent."

Why: the official comparison is between its own old and new versions — the baseline is the old implementation frozen at commit 68c077bf…, not any other agent; and 3 samples with p = 0.25, which the officials themselves call a controlled small sample.

The right approach: include qualifiers when citing: "In 3 interleaved runs on the same task and the same model, the new version's median is 25% faster than the old version."

❌ Assuming the officials tuned it to 7 seconds in one shot ​

The claim: seeing 7.073 seconds and assuming the performance came out naturally.

Why: performance.md preserves the complete development record: the old version passed once at 9.302 s; two accessibility-tree approaches at 9.395 s and 10.157 s; the first direct DOM approach at 8.697 s, but independent verification failed (incomplete name/value extraction); after the fix it climbed through 8.051, 8.631, 8.395, 8.385, 7.741 s, with Mercury diagnosing 7.559 s. Even earlier there was a recording at 11.387 s and a strategy regression at 12.898 s.

The right approach: treat "8.697 s but verify failed" as a reminder — no matter how good the seconds look, it doesn't count unless independent verification passes. The officials dared to write every failed attempt into the documentation, which precisely shows that reproducibility matters more than looking good.

❌ Applying 7 seconds to all websites ​

The claim: "Flights takes 7 seconds, so my complex admin panel should also take 7 seconds."

Why: the officials have only published timings for three tasks: Flights 7.073 s, opening Gödel's incompleteness theorem on Wikipedia 2.798 s, and a local hotel task 1.896 s — the officials explicitly called the latter two "smoke checks, not matched comparisons." The DOM reader only supports common HTML/ARIA controls; shadow DOM, canvas, uploads, and so on are not supported.

The right approach: measure on your own site with your own goals; first check the unsupported list in Ultrafast troubleshooting and rule out inapplicable pages.

Advanced tips (optional) ​

Measure with your own methodology. After the previous lesson's uv run --env-file .env python examples/flights.py --keep-open run finishes, open artifacts/flights/latest/state.json: the top-level elapsed_ms is this round's official-basis seconds; in history, each action carries executed_ms, latency_ms, and text_latency_ms, where decisions is each prediction and text_calls is the text helper. The official measurement JSON is a report distilled from these fields. Run three rounds and take the median — more convincing than a single run.

Verify this lesson's numbers field by field. docs/flights-measurement.json (the recording) and docs/full-speed-measurement.json (the comparison experiment) carry all the raw data: source code hashes, token counts, and the cost of every text call; the comparison experiment's 6 runs each also carry the browser version (Chrome/152.0.7977.83) and full CDP details. To verify any number in this article, go to these two files.

Watch for code changes after the measurement. performance.md records: after timing and recording were completed, the interrupt handling for native SELECTs was tightened (stop when the result is uncertain, no longer treating it as a retryable stale read); Flights doesn't trigger native SELECTs, so the timing and recording hashes remain unchanged. The code you clone will be slightly newer than at measurement time — that's normal.

When choosing a text helper, see how the officials picked theirs. The officials used two real flight fields as probes and tried Gemini 2.5 Flash Lite, Gemini 3.1 Flash Lite, and Mercury 2.5; all returned correct values. Earlier probes rejected one model that swapped origin/destination and one model that didn't output valid JSON but output commentary. Lesson: when switching TEXT_MODEL, don't just look at latency — probe correctness with real fields first.

Lesson summary ​

✅ Jev Ultrafast's performance numbers fall into three groups: the recording at 7.073 s (basis: first prediction → accepted DONE), the comparison experiment at 9.450 → 7.092 s (6 interleaved runs, median taken, 3/3 for both sides), and protocol calls at 1,092 → 101 (same experiment).

✅ Recording bill: 17 decision requests (median 178 ms, totaling 3,720 ms, about 53% of total time), 2 text generations (927 ms), 11 actions, 90,558 + 6,325 tokens; the search executed at second 5.217.

✅ 17 requests = 11 actions + 1 accepted DONE + 5 stale (a prediction invalidated by a page change before execution; the time still counts).

✅ The speedup comes from a changed way of "looking at the page": DOM.resolveNode 828 → 0, full accessibility tree 30 → 0, Runtime.evaluate about 150 → about 75; input-related calls are unchanged on both sides.

✅ Include qualifiers when citing: single task, 3 interleaved pairs, live network, p = 0.25 — the officials wrote the limitations right next to the numbers, and you should relay them the same way.

Key numbers at a glance:

MetricOfficial numberBasis
Recording duration7.073 sFirst prediction → accepted DONE
Decision requests / median latency17 requests / 178 msRecording
Search execution time5.217 sRecording, one of 11 actions
Median elapsed time comparison9.450 → 7.092 s (−25.0%)3 pairs of alternating runs
Median protocol calls1,092 → 101Same comparison experiment
Text helper cost$0.00006272Only 2 text calls (OpenRouter)
TypeSafe token90,558 input / 6,325 outputFull log of all requests

Next lesson preview ​

Now that we've gone through the numbers, the next lesson Write your own browser task: Agent library in practice turns Ultrafast into your own tool. You will learn:

  • Complete usage of Agent(url, goals): goals can be a string or a list, run() to get the state step by step
  • How to use each field in state: history, decisions, text_calls, status, elapsed_ms — the very data source behind this lesson's bill
  • A new URL plus a one-line goal is a new task, and when you should set up your own independent verification
  • How to swap the text model: configuration methods for DeepSeek, GLM and other OpenAI-compatible endpoints

Appendix: source code reference ​

Click to expand and view source code locations

Last updated: 2026-11-04 (repository github.com/browser-use/jev-ultrafast, local source of information sources/jev/repo-ultrafast/)

FeatureFile pathKey locations
Performance report bodydocs/performance.mdTiming methodology (Timing begins…); full bill (17 requests / 178 ms / 5.217 s / 7.073 s / 90,558+6,325 token / $0.00006272); comparison experiment table and p=0.25; Where the time went; Other checks (2.798 s / 1.896 s); Development attempts; Limits
Measured datadocs/flights-measurement.jsonelapsed_ms 7073; decision_requests 17; decision_median_ms 178; decision_total_ms 3720; actions array of 11 entries' executed_ms timeline; text_calls (581/346 ms and $2.93e-05/$3.342e-05); tokens 90558/6325; video (187 frames / 1× / 0 opening pause / 500 ms ending pause / trimmed 64 px); timing field contains the original methodology wording
Comparison experiment datadocs/full-speed-measurement.jsonsummary (9450/7092, 1092/101); runs 6 entries (elapsed_ms, decision_requests, cdp details, source code hash, tokens, text_calls for each pair); scope (3 pairs, 1120×780, live uncontrolled); baseline_commit 68c077bf…
README summaryREADME.md"Evidence and limits": 7,073 ms methodology, 3/3 passed, 25%, protocol calls 1,092 → 101, 2.798 s / 1.896 s, DONE requires independent verification
Timing start and stale handlingjev_ultrafast/agent.pyL68–69 started_at set on first predict; L93–99 updated when DONE is accepted elapsed_ms; L55–64 StalePage → discard decision, re-observe; L75–76 model call budget MAX_STEPS×2; L102–104 action budget
Default modeljev_ultrafast/model.pyL108 os.environ.get("TYPESAFE_MODEL", "jev-latest")
Trace saved fieldsexamples/flights.pyL50–52 prints elapsed_ms/status; L54–56 state.json = agent.snapshot() + verification (including elapsed_ms, history, decisions, text_calls)
Original design principlesREADME.mdL98 "One browser call per snapshot"; L100 combobox suggested wait cap of 200 ms, other interactions two animation frames or 50 ms
Jev pricing (basis for cost estimation)TypeSafe official documentation docs/models.md (local sources/jev/docs/models.md)$42 / B input token, output free; alias jev-latest currently points to jev-1.14.0
gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.