Performance breakdown: how the 7.073 seconds were measured
In the previous lesson you watched the agent finish a flight search in 7 seconds with your own eyes. This lesson takes the stopwatch apart — 7.073 seconds is not a marketing line, it is a recording with complete measurement boundaries: the officials not only provided a report, they also put the raw measurement JSON into the repository, complete with the source code hash and browser version. Once you can read the Jev Ultrafast performance data properly, you can not only reproduce it, but also copy this methodology to measure your own agent.
The numbers in this lesson mainly come from three files in the jev-ultrafast repository you cloned: docs/performance.md (the report body), docs/flights-measurement.json (recording data), and docs/full-speed-measurement.json (comparison experiment data).
What you'll be able to do after this lesson
- State the timing methodology for the 7.073 seconds: the clock starts at the first prediction and stops when DONE is accepted; what the five on-clock items and three off-clock items are
- Break down the recording bill: 17 decision requests (median 178 ms, total 3,720 ms), 2 text generations, 11 actions, and at which second each action executed
- Understand the design of the 9.450 → 7.092 second comparison experiment, and explain why the officials themselves said "3 pairs is too few, not a general benchmark"
- Explain clearly what the optimization cut out: browser protocol calls 1,092 → 101, of which 828 DOM.resolveNode calls went straight to zero
- Use the trace from
examples/flights.pyto align to the same methodology and measure your own seconds
Your current predicament
- "The officials say 7 seconds, I get 12 seconds — isn't this number probably cherry-picked from the best run?" Without knowing the methodology, any comparison is meaningless
- You want to cite this set of ultrafast speed numbers in a report, but falter the moment someone asks "how was it measured"
- You want to run a performance comparison for the browser agent you built, but don't know which variables to control: the model? the network? the browser? You can't even find a template to copy from
When to use this technique
When you need to:
- Relay or cite Jev Ultrafast's performance numbers, and withstand the follow-up question "how was it measured"
- Run your own performance comparison for a browser agent and need a ready-made measurement template (methodology, independent verification, sample size statement) to copy
- Judge whether this set of numbers fits your scenario: it is single-task, on a live network, with a small sample
And you don't want to:
- Treat seconds from a single run as a stable metric and cite it everywhere
- Compare numbers measured under different methodologies (inspector with screenshots vs. script without screenshots; jev-1.13 vs. jev-1.14)
🎒 Preparation before starting
- Completed Running Google Flights successfully: have run
examples/flights.pyat least once and seen"passed": truein the verification JSON andartifacts/flights/latest/ - Remember the "two decisions per round trip" from Jev Ultrafast principles — each of the 17 requests in this lesson follows this structure
- (Optional) Keep the jev-ultrafast repository open and read along with
docs/performance.mdand the two measurement JSON files; every number in this lesson can be found there - A glass of water — this lesson is mostly reading; hands-on work comes in "Advanced tips"
Core idea
The officials' attitude toward performance numbers can be compressed into three sentences:
- Every number comes with its methodology — from which millisecond the 7.073 seconds start and stop, and what counts as on the clock versus off the clock, the documentation spells it all out
- Comparisons must control variables — old and new versions alternately run 3 times each, same task, same model, same budget, same independent verification
- Proactively declare limitations — sample size, live network, single task: the officials say it themselves first, waiting for no one to call them out
The four sets of official numbers this lesson will break down:
| Number | Official methodology | Source |
|---|---|---|
| 7.073 s | first prediction → accepted DONE; includes model calls, text generation, browser operations, stale decisions, loading | docs/performance.md, flights-measurement.json |
| 9.450 → 7.092 s | 6 alternating runs, median taken; both 3/3 verified independently | docs/full-speed-measurement.json |
| 1,092 → 101 | median of browser protocol calls from the same comparison experiment | docs/full-speed-measurement.json |
| $0.00006272 | OpenRouter billing for only two text helper calls, excluding TypeSafe | docs/performance.md |
One more thing to keep in mind: all of these numbers come from the same combination—decision model jev-1.13.0 + text model inception/mercury-2.5 (via OpenRouter, reasoning disabled). In the measurement JSON, TYPESAFE_MODEL is null, meaning the code default jev-latest was used (jev_ultrafast/model.py line 108), which at the time resolved to 1.13.0; today jev-latest already points to jev-1.14.0. If you reproduce it today, the model will be different, so the seconds will naturally differ.
💡 Why this lesson is worth a careful read
More useful than memorizing the number 7.073 is the paradigm of "methodology + billing + sample size declaration." You can copy it wholesale when measuring the performance of any agent yourself—whether Ultrafast or anything else.
Follow along with me
Step 1: Align the timing methodology—from which millisecond does the official clock start
Why: Align the methodology before comparing any two timings, otherwise every comparison is invalid. Whether you run 9 seconds or 12 seconds yourself, the first step is to figure out how the official clock runs.
The official methodology, laid out in one table:
| Item | Content |
|---|---|
| Clock start | the first prediction after the initial homepage observation |
| Clock stop | the moment DONE is accepted |
| On the clock | model calls, text generation, browser operations, stale decisions (predictions discarded and redone), loading waits |
| Off the clock | browser startup, initial navigation, the standalone verify() check after the run completes |
This methodology matches the code (jev_ultrafast/agent.py):
started_atis set only on the firstpredict—so the initial page observation is off the clock- DONE updates
elapsed_mswhen accepted inact—that's the clock stop point - An expired prediction (
StalePage) discards the decision, re-observes, and re-predicts—the discarded time stays on the clock
The video uses the same methodology: 1× original playback speed, no opening pause, only a 0.5-second pause at the end; 186 continuous screen-recording frames plus 1 initial screenshot (source_frames: 187 in the JSON), with only the top 64 px account/navigation bar cropped out—no trimming of the head or tail.
You should see: the timing section in docs/performance.md corresponds to the table above item by item—"begins at the first prediction after initial homepage observation and ends at the accepted DONE choice", matching what's on and off the clock word for word.
- Be able to name the five on-the-clock items and the three off-the-clock items
Step 2: Break down the bill for 7.073 seconds—where did 17 requests go
Why: The officials wrote every number from this recorded run into docs/flights-measurement.json, so the bill can be copied directly. The totals:
| Metric | Value |
|---|---|
| Total time | 7,073 ms |
| Decision requests | 17 requests |
| Decision latency | median 178 ms, total 3,720 ms |
| Actions executed | 11 (10 interactions + 1 explicit WAIT) |
| Text generation | 2 times (Zurich 581 ms, London 346 ms) |
| token | 90,558 input + 6,325 output |
| Decision model | jev-1.13.0 |
Where the time went (the first two rows are raw fields from the official JSON; the third row is this page's calculation: 7073 − 3720 − 927):
| Destination | Time | Share |
|---|---|---|
| 17 decision requests | 3,720 ms | ≈ 53% |
| 2 text generations | 927 ms | ≈ 13% |
| Browser execution, observation, page loading | ≈ 2,426 ms | ≈ 34% |
17 requests don't equal 17 actions: 11 actions plus 1 accepted DONE make 12 predictions "spent where it counts"; the remaining 5 were intercepted by the freshness check before execution—the page fingerprint changed, so decisions based on the old snapshot were discarded and redone. These are the stale decisions in the methodology, and their time is likewise counted in the total seconds. (5 = 17 − 11 − 1; this page derives it from agent.py's loop logic of "each prediction is consumed exactly once, and a discarded one triggers a re-measure"—the officials did not publish the stale count separately.)
Positions of the 11 actions on the timeline (executed_ms, taken from the actions array in the measurement JSON):
| Time | Action |
|---|---|
| 0.336 s | CLICK Change ticket type. Round trip |
| 0.640 s | CLICK One way |
| 1.595 s | TYPE_TEXT Where from? → Zurich (text generation 581 ms) |
| 1.930 s | CLICK Zürich, Switzerland (selecting the autocomplete suggestion) |
| 2.748 s | TYPE_TEXT Where to? → London (text generation 346 ms) |
| 3.053 s | CLICK London, United Kingdom |
| 3.452 s | CLICK Open Departure |
| 3.769 s | CLICK Sunday, September 20, 2026 |
| 4.097 s | CLICK Done. Search for one-way flights… |
| 5.217 s | CLICK Search (official wording: the search was executed at second 5.217) |
| 5.587 s | WAIT for the page to update |
| 7.073 s | DONE accepted, stop the clock |
The first 5.2 seconds were spent filling in and submitting the form; the last 1.9 seconds waiting for results and confirming completion. WAIT executed at 5.587 seconds, DONE accepted at 7.073 seconds — for this final roughly 1.49 seconds, the official explanation is results-page loading, state changes, and the completion decision. The two text generations each caused a "typing pause" of about 0.3–0.6 seconds, which is why in the video the input field seems to "think for a moment before text appears".
You should see: flights-measurement.json the four top-level fields elapsed_ms, decision_requests, decision_median_ms, decision_total_ms, and the executed_ms of the 11 entries in the actions array — the timeline above was copied from this array.
- Be able to say how many of the 17 requests became actions, how many were stale, and how many were DONE
Step 3: Understand the Jev Ultrafast performance comparison experiment: 9.450 → 7.092 s
Why: a single 7.073-second run is just a recording; the conclusion "25% faster" comes from a separate controlled experiment. Its design determines how strong the conclusion can be.
Experiment design (the "Matched runtime comparison" section of performance.md):
- 6 runs, alternating: one baseline run, one optimized run, 3 pairs in total; the same task, the same existing Chrome profile
- The two arms used exactly the same: natural-language goal, independent result checker, 1120×780 viewport, jev-1.13.0, mercury-2.5 (reasoning off), action and request budgets
- Initial navigation was untimed on both sides; each run opened and closed its own tab; all 6 runs counted, none discarded, no vendor or verification failures
- The baseline arm was frozen at commit
68c077bf…; both sides used Mercury, avoiding mixing the two variables of "swapping the text model" and "changing the code"
Results:
| Pair | Baseline | Optimized | Verification |
|---|---|---|---|
| 1 | 11.214 s | 6.964 s | Both passed |
| 2 | 8.984 s | 7.913 s | Both passed |
| 3 | 9.450 s | 7.092 s | Both passed |
| Median | 9.450 s | 7.092 s | each 3/3 |
Three median changes: task time −25.0%, TypeSafe requests 22 → 17, browser protocol calls 1,092 → 101. The optimized version was faster in all three pairs.
But the officials immediately gave three disclaimers themselves:
- Too few pairs (3); a two-sided sign test gives p = 0.25, which cannot support a strong statistical conclusion
- This is a controlled small-sample comparison, not a broad agent benchmark
- Google, network responses, routing, and browser cache were all live and not controlled
⚠️ The correct way to cite these numbers
"In 3 pairs of alternating runs on the same task, same model, same browser, the optimized version was faster every time, with a median speedup of 25%" — that is a supported claim. "Jev Ultrafast is 25% faster than some agent" — not supported; the officials never ran a cross-agent comparison.
You should see: the comparison table in performance.md is just the 3 pairs of numbers above plus a bolded median row; "p = 0.25" and "not a broad agent benchmark" are both in the original text.
- Be able to say which variables the experiment controlled and which it did not
Step 4: Find where the time went — what the 1,092 → 101 protocol calls cut out
Why: the cause of the "25% faster" result lies entirely in the browser protocol calls. The officials stored the complete CDP (Chrome DevTools Protocol) call details for every run in the measurement JSON — the hardest set of evidence in the whole course.
The three burdens of the old loop (performance.md, "Where the time went"):
- Every DOM change — including animations — invalidated the decision and triggered repeated re-prediction
- Repeatedly reading the entire accessibility tree (
Accessibility.getFullAXTree) - Parsing hundreds of DOM nodes (
DOM.resolveNode)
The new loop's approach (performance.md "Where the time went" + README):
- One browser call reads common HTML/ARIA controls — no more repeatedly walking the tree and parsing nodes one by one
- The click guard only compares the target, nearby context, and document/form state; geometry and hit testing are still done before input
- combobox event waiting: autocomplete suggestions must arrive first (README: cap of 200 ms; other interactions take at most two animation frames or 50 ms) before asking Jev
- Native text replacement explicitly issues the browser's select-all command
CDP call breakdown (listed one by one across three runs, from the cdp object of each run in full-speed-measurement.json):
| Protocol method | Baseline (3 runs) | Optimized (3 runs) |
|---|---|---|
| DOM.resolveNode | 828 / 828 / 828 | 0 (key no longer appears) |
| Accessibility.getFullAXTree | 30 / 30 / 30 | 0 |
| Runtime.callFunctionOn | 40 / 40 / 40 | 0 |
| DOM.getBoxModel + DOM.getNodeForLocation | 10 + 10 / 10 + 10 / 10 + 10 | 0 |
| Runtime.evaluate | 158 / 146 / 148 | 75 / 77 / 71 |
| Input.dispatchMouseEvent / KeyEvent / insertText | 20 / 4 / 2 (identical each time) | 20 / 4 / 2 (unchanged) |
| Total | 1102 / 1090 / 1092 (median 1,092) | 101 / 103 / 97 (median 101) |
Two insights from reading the table:
- The way of "looking at the page" changed, but "clicking the page" did not — the three input call types (mouse, keyboard, text insertion) are completely identical on both sides; everything cut was observation-type calls
- Fewer requests too (median 22 → 17): decisions are no longer frequently invalidated by animations, so re-predictions naturally decrease
💡 This is the quantitative evidence for "one snapshot, one browser call"
The principles lesson covered the README design principle "One browser call per snapshot". In numbers: Runtime.evaluate dropped from about 150 calls to about 75, and DOM.resolveNode went from 828 calls to zero. The architectural claim and the measured data line up.
You should see: in full-speed-measurement.json, the cdp breakdown of the baseline's three runs shows DOM.resolveNode at 828 calls each; the cdp breakdown of the optimized version's three runs doesn't contain this method at all.
- Be able to name the three types of observation calls that were cut and why input-type calls are unchanged
Step 5: Work out the cost — tokens, the text helper, and the price the officials didn't report
Why: a performance read can't stop at seconds; you also need to know how much this run cost. The officials reported half of it, and the other half can be derived yourself.
What the officials reported:
- All requests in the recording totaled 90,558 input tokens + 6,325 output tokens
- OpenRouter billed the two text calls at $0.00006272 (Zurich $0.0000293 + London $0.0000334)
- The officials explicitly state: this is only the cost of the text helper, not the total task cost — the TypeSafe response contains only token counts, no dollar bill, and browser costs are not included
What the officials didn't report but you can derive yourself (derived on this page, not official figures):
At TypeSafe official pricing ($42 / B input tokens, output free), 90,558 input tokens ≈ $0.0038. Derivation: 90558 × 42 ÷ 10⁹ ≈ 0.0038 dollars. Adding the text helper's $0.00006272, this 7-second task's total model cost is roughly 0.4 cents.
One example of variance (all in full-speed-measurement.json): for the same London generation, the recorded run took 346 ms and 94 completion tokens ($0.0000334); the optimized run of pair 2 in the comparison experiment took 1,137 ms and 511 tokens ($0.0000961) — a single text call's latency can differ by more than 3x, and all such variance goes into the total seconds.
You should see: the "$0.00006272" section in performance.md says "text-helper charge, not total task cost"; the unit prices of the two calls are in flights-measurement.json's text_calls (2.93e-05 + 3.342e-05).
- Be able to explain clearly what $0.00006272 includes and what it doesn't
Checkpoint ✅
- Be able to state the full timing methodology: start point, stop point, the five in-table items, the three out-of-table items
- Be able to recite the four key figures of the recorded bill: 17 requests, median 178 ms, 11 actions, search executed at second 5.217
- Be able to explain the 9.450 → 7.092 second experiment design, and state the officials' own three disclaimers
- Be able to name the three types of calls that went to zero among protocol calls 1,092 → 101
- Be able to explain clearly that $0.00006272 is only the text helper cost, and the TypeSafe bill must be calculated separately from the price list
Pitfall warning
❌ Using your own seconds to directly challenge the official numbers
Symptom: you run 9 seconds, 12 seconds, and suspect the 7.073 seconds are inflated.
Why: at least four measurement discrepancies — ① Different models: the measurement used jev-1.13.0, while today jev-latest already points to jev-1.14.0; ② The network is live: the official optimized version itself spread across 6.964–7.913 seconds over three runs; ③ The inspector was taking screenshots, which is not a timed run to begin with (as covered in the previous lesson); ④ Your run may have hit more stale decisions.
The right approach: run examples/flights.py (no screenshots) and read artifacts/flights/latest/state.json's top-level elapsed_ms — that is the number on the same basis as the official one; align the model, URL, and goal before comparing.
❌ Presenting "25% faster" as a cross-agent conclusion
The claim: "Jev Ultrafast is 25% faster than some browser agent."
Why: the official comparison is between its own old and new versions — the baseline is the old implementation frozen at commit 68c077bf…, not any other agent; and 3 samples with p = 0.25, which the officials themselves call a controlled small sample.
The right approach: include qualifiers when citing: "In 3 interleaved runs on the same task and the same model, the new version's median is 25% faster than the old version."
❌ Assuming the officials tuned it to 7 seconds in one shot
The claim: seeing 7.073 seconds and assuming the performance came out naturally.
Why: performance.md preserves the complete development record: the old version passed once at 9.302 s; two accessibility-tree approaches at 9.395 s and 10.157 s; the first direct DOM approach at 8.697 s, but independent verification failed (incomplete name/value extraction); after the fix it climbed through 8.051, 8.631, 8.395, 8.385, 7.741 s, with Mercury diagnosing 7.559 s. Even earlier there was a recording at 11.387 s and a strategy regression at 12.898 s.
The right approach: treat "8.697 s but verify failed" as a reminder — no matter how good the seconds look, it doesn't count unless independent verification passes. The officials dared to write every failed attempt into the documentation, which precisely shows that reproducibility matters more than looking good.
❌ Applying 7 seconds to all websites
The claim: "Flights takes 7 seconds, so my complex admin panel should also take 7 seconds."
Why: the officials have only published timings for three tasks: Flights 7.073 s, opening Gödel's incompleteness theorem on Wikipedia 2.798 s, and a local hotel task 1.896 s — the officials explicitly called the latter two "smoke checks, not matched comparisons." The DOM reader only supports common HTML/ARIA controls; shadow DOM, canvas, uploads, and so on are not supported.
The right approach: measure on your own site with your own goals; first check the unsupported list in Ultrafast troubleshooting and rule out inapplicable pages.
Advanced tips (optional)
Measure with your own methodology. After the previous lesson's uv run --env-file .env python examples/flights.py --keep-open run finishes, open artifacts/flights/latest/state.json: the top-level elapsed_ms is this round's official-basis seconds; in history, each action carries executed_ms, latency_ms, and text_latency_ms, where decisions is each prediction and text_calls is the text helper. The official measurement JSON is a report distilled from these fields. Run three rounds and take the median — more convincing than a single run.
Verify this lesson's numbers field by field. docs/flights-measurement.json (the recording) and docs/full-speed-measurement.json (the comparison experiment) carry all the raw data: source code hashes, token counts, and the cost of every text call; the comparison experiment's 6 runs each also carry the browser version (Chrome/152.0.7977.83) and full CDP details. To verify any number in this article, go to these two files.
Watch for code changes after the measurement. performance.md records: after timing and recording were completed, the interrupt handling for native SELECTs was tightened (stop when the result is uncertain, no longer treating it as a retryable stale read); Flights doesn't trigger native SELECTs, so the timing and recording hashes remain unchanged. The code you clone will be slightly newer than at measurement time — that's normal.
When choosing a text helper, see how the officials picked theirs. The officials used two real flight fields as probes and tried Gemini 2.5 Flash Lite, Gemini 3.1 Flash Lite, and Mercury 2.5; all returned correct values. Earlier probes rejected one model that swapped origin/destination and one model that didn't output valid JSON but output commentary. Lesson: when switching TEXT_MODEL, don't just look at latency — probe correctness with real fields first.
Lesson summary
✅ Jev Ultrafast's performance numbers fall into three groups: the recording at 7.073 s (basis: first prediction → accepted DONE), the comparison experiment at 9.450 → 7.092 s (6 interleaved runs, median taken, 3/3 for both sides), and protocol calls at 1,092 → 101 (same experiment).
✅ Recording bill: 17 decision requests (median 178 ms, totaling 3,720 ms, about 53% of total time), 2 text generations (927 ms), 11 actions, 90,558 + 6,325 tokens; the search executed at second 5.217.
✅ 17 requests = 11 actions + 1 accepted DONE + 5 stale (a prediction invalidated by a page change before execution; the time still counts).
✅ The speedup comes from a changed way of "looking at the page": DOM.resolveNode 828 → 0, full accessibility tree 30 → 0, Runtime.evaluate about 150 → about 75; input-related calls are unchanged on both sides.
✅ Include qualifiers when citing: single task, 3 interleaved pairs, live network, p = 0.25 — the officials wrote the limitations right next to the numbers, and you should relay them the same way.
Key numbers at a glance:
| Metric | Official number | Basis |
|---|---|---|
| Recording duration | 7.073 s | First prediction → accepted DONE |
| Decision requests / median latency | 17 requests / 178 ms | Recording |
| Search execution time | 5.217 s | Recording, one of 11 actions |
| Median elapsed time comparison | 9.450 → 7.092 s (−25.0%) | 3 pairs of alternating runs |
| Median protocol calls | 1,092 → 101 | Same comparison experiment |
| Text helper cost | $0.00006272 | Only 2 text calls (OpenRouter) |
| TypeSafe token | 90,558 input / 6,325 output | Full log of all requests |
Next lesson preview
Now that we've gone through the numbers, the next lesson Write your own browser task: Agent library in practice turns Ultrafast into your own tool. You will learn:
- Complete usage of
Agent(url, goals): goals can be a string or a list,run()to get the state step by step - How to use each field in state: history, decisions, text_calls, status, elapsed_ms — the very data source behind this lesson's bill
- A new URL plus a one-line goal is a new task, and when you should set up your own independent verification
- How to swap the text model: configuration methods for DeepSeek, GLM and other OpenAI-compatible endpoints
Appendix: source code reference
Click to expand and view source code locations
Last updated: 2026-11-04 (repository github.com/browser-use/jev-ultrafast, local source of information sources/jev/repo-ultrafast/)
| Feature | File path | Key locations |
|---|---|---|
| Performance report body | docs/performance.md | Timing methodology (Timing begins…); full bill (17 requests / 178 ms / 5.217 s / 7.073 s / 90,558+6,325 token / $0.00006272); comparison experiment table and p=0.25; Where the time went; Other checks (2.798 s / 1.896 s); Development attempts; Limits |
| Measured data | docs/flights-measurement.json | elapsed_ms 7073; decision_requests 17; decision_median_ms 178; decision_total_ms 3720; actions array of 11 entries' executed_ms timeline; text_calls (581/346 ms and $2.93e-05/$3.342e-05); tokens 90558/6325; video (187 frames / 1× / 0 opening pause / 500 ms ending pause / trimmed 64 px); timing field contains the original methodology wording |
| Comparison experiment data | docs/full-speed-measurement.json | summary (9450/7092, 1092/101); runs 6 entries (elapsed_ms, decision_requests, cdp details, source code hash, tokens, text_calls for each pair); scope (3 pairs, 1120×780, live uncontrolled); baseline_commit 68c077bf… |
| README summary | README.md | "Evidence and limits": 7,073 ms methodology, 3/3 passed, 25%, protocol calls 1,092 → 101, 2.798 s / 1.896 s, DONE requires independent verification |
| Timing start and stale handling | jev_ultrafast/agent.py | L68–69 started_at set on first predict; L93–99 updated when DONE is accepted elapsed_ms; L55–64 StalePage → discard decision, re-observe; L75–76 model call budget MAX_STEPS×2; L102–104 action budget |
| Default model | jev_ultrafast/model.py | L108 os.environ.get("TYPESAFE_MODEL", "jev-latest") |
| Trace saved fields | examples/flights.py | L50–52 prints elapsed_ms/status; L54–56 state.json = agent.snapshot() + verification (including elapsed_ms, history, decisions, text_calls) |
| Original design principles | README.md | L98 "One browser call per snapshot"; L100 combobox suggested wait cap of 200 ms, other interactions two animation frames or 50 ms |
| Jev pricing (basis for cost estimation) | TypeSafe official documentation docs/models.md (local sources/jev/docs/models.md) | $42 / B input token, output free; alias jev-latest currently points to jev-1.14.0 |