Performance investigation: targeted troubleshooting with evidence
💡 One-sentence summary: Use Grok Bot to build your AI performance troubleshooting pipeline — connect observability, analytics, and incident tools, let the Bot look at dashboards and traces itself, pinpoint the highest-confidence hotspot, and produce a short report with screenshots and direct links, with facts and assumptions kept separate, never modifying alerts or production settings throughout.
This is the second lesson of Route B (programmers), coming from the same page of official roles as Bug reproduction and using the same official six-step method. This lesson corresponds to the official use-cases page, specifically the Product performance (performance investigator) role among 8 roles. The official positioning for it is: targeted performance investigation with evidence — not "helping you guess why things got slow," but an investigation that, every time, points to one hotspot backed by traceable evidence. The three types of connections the official docs name are observability, analytics, and incident tools.
The whole lesson takes about 12 minutes. The official docs have one special requirement for this role: routines are for periodic health reports, not unsupervised production changes — step 6 of this lesson is dedicated to how to hold that line.
What you'll be able to do after this lesson
- Build a "performance investigator" Bot: connect observability, analytics, and incident tools, and turn "production got slow" into a targeted investigation with evidence
- Let the Bot look at dashboards and traces itself, pinpoint the highest-confidence hotspot, and return a short report: screenshots + direct links, with facts and assumptions kept separate
- Hold the official two red lines: never modify alerts, never touch production settings — change recommendations go into the report, execution stays behind approval
- Save the investigation workflow as a skill and rerun it by name; hand the weekly health report to a routine (optional)
Why use Grok Bot instead of web-based AI?
| Capability | Web-based AI | Grok Bot |
|---|---|---|
| Can see your monitoring data | ❌ Relies on your screenshots and verbal descriptions; the AI can't see the dashboards | ✅ Connects to observability/analytics/incident tools and looks at dashboards and traces itself |
| Traceable evidence | ❌ Conclusions can't be verified; you can only choose to trust them | ✅ Every conclusion comes with direct links and screenshots; you open and verify them one by one |
| Facts and assumptions kept separate | ❌ All in one muddle; "I think" and "the data shows" are written mixed together | ✅ The official starter prompt itself requires keeping facts and assumptions separate |
| Remembers the investigation criteria | ❌ You re-explain which panels to look at for every incident | ✅ The investigation workflow is saved as a skill, with consistent criteria, rerun by name |
| Periodic health reports | ❌ Manually export data and piece together a weekly report every week | ✅ A scheduled routine runs on its own when the time comes, and the report is posted in the conversation awaiting your review |
Your current predicament
- "Checkout got slow after yesterday's release" — you keep switching between five or six dashboards and can't pinpoint the hotspot after half a day
- Everyone in the incident channel is guessing: is it the database? The cache? The new version? No evidence, just the loudest voices
- The investigation report is scattered as soon as it's written: screenshots not saved, links not kept, no timestamps — the numbers don't match up in the post-mortem
When to use this technique
When you need to:
- Turn "XX got slow" into a targeted AI performance investigation with evidence: look at dashboards and traces, pinpoint the highest-confidence hotspot
- Get the "take the incident → check the panels → narrow down the hotspot → write the report" pipeline running, with you only reviewing the report
- Make every conclusion in the report traceable: direct links, screenshots, timestamps, with facts and assumptions kept separate
And you don't want to:
- Open dashboards one by one, take screenshots, and piece together a report yourself
- Locate the problem by guessing in the incident channel
- Hear an AI say "it might be a slow database" without offering any evidence
🎒 Preparation before you start
- Have read 3.2 Skills and demonstration — step 5 of this lesson saves the investigation workflow as a skill, and this is also a common prerequisite for most scenario pages
- Have a Bot you can assign work to: if you don't have one yet, first go to 1.5 Create your first Bot
- Access to observability tools (dashboards and a traces system — either a connector or the web version works)
- Access to analytics and incident tools (the Sentry and PagerDuty types named in the official incident trigger sources, or your own incident channels)
- A real performance incident as practice input (e.g., "latency on an endpoint rose after yesterday's release")
- About 12 minutes, plus one round of human feedback after the first report comes out
Follow along with me
Step 1: Write the performance investigator's responsibilities and boundaries into the Bot's description
Why: The first step of the official six-step method is to write responsibilities, source systems, output format, and standing boundaries into the Bot's description — this role's two red lines (never modify alerts, never touch production settings) must also be hard-coded here.
How to do it: Create or open a Bot (sidebar + → Create a Bot), and write the description like this:
You are my performance investigator, responsible for targeted performance investigations with evidence.
Data sources: observability tools (dashboards, traces), analytics tools, incident tools.
Output format: a short report — the highest-confidence hotspot, supporting evidence (screenshots + direct links),
and suggested next steps; write facts and assumptions separately, and list anything unverifiable on its own.
Standing boundaries: the investigation is read-only — never modify alerts, never touch production settings;
any changes are written as suggestions only, and execution is approved by me.You should see: the description is saved on this Bot's details page. Writing the boundaries here means every future investigation follows the same red line.
Step 2: Connect observability, analytics, and incident tools
Why: facts in an investigation can only come from the systems you connect—the three categories of connections the official docs name for this role are observability, analytics, and incident tools.
How: handle the three tool categories separately; for details of the 5-step connection method see 2.5 Connect apps:
- Tools with a connector: search in Connect apps first; if you want the Bot to install it for you, just say something like "Add the Linear app" (the official docs use Linear as the example—the Bot searches the directory, installs the app, and gives you a Connect card when login is needed)
- Tools without a connector: have the Bot open the web version in its cloud computer browser, with login credentials stored in Vault—the Bot fills them in but cannot see the values
- Tools that provide an MCP server: have the Bot add them by URL
- Dashboards that only open on the company intranet: open Settings → Computer → Route traffic through this computer so the Bot's browser uses your desktop's network and IP (details in 5.3 Traffic routing and local execution)
- If authorization breaks, click Reconnect (on the app) or Reopen (on the connection card in the chat) to get back to the login page
You should see: the Bot can open all three tool categories. Note that Added ≠ logged in—when Needs auth or Disconnected shows, click Authenticate to re-authorize; once intranet routing takes effect, internal dashboards load normally in the Bot's browser (sites resolving to private/VPN addresses are blocked by default and require organization approval—when blocked, this setting shows the number of blocks this time).
Step 3: Run your first AI performance investigation with the official starter prompt
Why: the official starter prompt deliberately stops at a read-only investigation review point—validate report quality first, then talk about automation.
How: send the incident context (which metrics, since when, what changes happened during that period) to the Bot along with your message, then send this Chinese-adapted version of the official Product performance starter prompt:
Investigate the rise in checkout latency after yesterday's release.
Look at dashboards and traces, and pinpoint the hotspot with the highest confidence,
then reply with a short investigation report with screenshots and direct links.
Keep facts and assumptions separate. Do not change alerts, do not touch production settings.You should see: the Bot opens dashboards one by one and goes through traces in the computer preview on the right—click the preview to watch its clicks and navigation. Finally it replies with a short report, with screenshots attached as cards in the reply that you can click to preview. A qualified report includes:
- The hotspot with the highest confidence (one sentence explaining "where it's slow")
- Supporting evidence: screenshots + direct links (clicking takes you back to the source panel)
- Facts and assumptions kept separate ("the data shows" and "I infer" are not mixed)
- Suggested next steps (suggestions only, not executed)
- Anything that could not be verified (per the official evidence-list requirement: list it explicitly, don't make things up)
💡 This is the unique value of the Grok Bot
A web-based AI can only rely on your screenshots and descriptions, with no way to verify its conclusions. The Grok Bot logs into monitoring tools itself to look at dashboards and traces—the evidence is what it saw with its own eyes, with direct links you can trace back—so you can verify item by item by clicking through, and in incident channels you no longer have to locate the problem by shouting.
Step 4: Revise the report until it's reviewable—verify the evidence item by item
Why: the official six-step method requires revising results until they're "reviewable without you watching"; the official evidence list also specifically warns: rapidly changing data cannot rely on screenshots alone.
How: verify the Bot's report item by item—
- Open every direct link; it should return to the source panel and match the numbers in the report
- Screenshots include relevant context (time range, service name, the anomalous segment visible)
- Timestamps and time zones are clearly stated (postmortems that don't add up usually fail here)
- Fast-changing metrics (latency curves, QPS, etc.) have source links or exports, not just a screenshot—in the official words: "Do not rely on a screenshot alone for rapidly changing data"
- Are facts and assumptions separated? If not up to standard, send it back and have the Bot revise the existing report rather than write a new one
You should see: after a few rounds of revision, the report can be reviewed at a glance—every conclusion carries evidence, and facts and assumptions are clearly separated.
Step 5: Save it as a "performance investigation" skill and test it with a different incident
Why: step four of the official six-step method: save the workflow that ran successfully as a skill, and test it with a second input.
How: send the following:
Save the complete performance investigation workflow we just ran as a skill named "performance investigation".
Include: source systems (dashboards, traces, incident tools), investigation steps,
output format (hotspot + evidence + fact/assumption separation + suggested next steps),
and the three rules: "don't change alerts, don't touch production settings, changes are written as suggestions only".The official docs say a useful skill must make six things clear: when to use it, what inputs and permissions it needs, the order of work, how to validate results, what it returns, and what requires approval. Once saved, ask for it by name in any conversation—just say "run the performance investigation skill" to invoke it (@ is for Bots, groups, routines, and apps; it doesn't trigger skills).
You should see: the Bot confirms the skill is saved. The official docs require testing with a second input: run it again with a different performance incident (another latency rise, a spike in error rate) and check whether the report is still solid. The private skill library is shared across all Bots; for saving details see 3.2 Skills and teaching.
Step 6: The routine only produces periodic health reports, not production changes
Why: the official docs' specific requirement for this role, verbatim: "Use a routine for a recurring health report, not for unsupervised production changes"—the correct use of a routine is to produce a report on schedule, not to change production while you sleep.
How: send the following:
Every Monday at 9 a.m., run the "performance investigation" skill and produce a health report:
trends of core metrics over the past week, anomalous segments, and early signals worth attention.
Post the report in this conversation and wait for my review. If data is missing, state what's missing—don't guess.After sending it, go through the six items of the official confirmation checklist with the Bot: which Bot it belongs to, time and time zone, input sources, expected results, approval boundaries, and what to do about missing data—correct it if anything doesn't match.
You should see: the routine appears in this Bot's routines. A new routine doesn't run immediately; it waits for the next scheduled time—if you want to verify right away, use Test. Routines run in the cloud, even with your computer closed; each run consumes usage, and a weekly health report keeps usage under control.
Hold the approval line at this step too: even if one day you want the routine to run automatically in response to incidents, its output can only be a report—changing alerts or touching production settings are external actions that always stay behind approval. An approval card triggered by a routine expires after 30 minutes with no response, so don't configure actions that require approval in a routine. For details on approval rules see 3.1 Approvals and Auto Review.
📋 Magic Prompts
📝 Targeted Performance Investigation (official starter prompt structure version)
Expected outcome: The Bot returns a short report — the hotspot is located by it looking at dashboards and traces, the evidence includes direct links, facts and hypotheses are separated, and you can verify each item one by one.
## Role
You are my performance investigator, responsible for targeted performance investigations with evidence.
## Task
Investigate this performance incident, locate the highest-confidence hotspot, and produce a short report.
## Input Information
### Required Items
- Incident description: [which metrics, what symptoms, since when]
- Time range: [the time period the investigation covers]
- Changes during the period: [what versions were deployed, what configurations were changed during this time]
### Optional Items
- Services of primary suspicion: [services or panels to check first?]
- Anomaly criteria: [the standard for judging an anomaly, e.g., P95 exceeding how many milliseconds?]
## Output Specification
Short report: the highest-confidence hotspot (one sentence), supporting evidence (screenshots + direct links),
facts and hypotheses in separate sections, suggested next steps, and a list of items that cannot be verified.
## Constraints
- ✅ Every conclusion carries a direct link that opens back to the source panel
- ✅ Facts and hypotheses are written separately; anything unverifiable is explicitly listed
- ❌ Do not change alerts or production settings
- ❌ For rapidly changing data, don't give only screenshots — keep the source link or an export📝 Weekly Health Report (routine version)
Expected outcome: a health report that automatically appears in the conversation every Monday — trends, anomaly segments, early signals — one glance tells you whether there's anything to watch this week.
## Role
You are my performance investigation specialist, producing a health report on a weekly routine basis.
## Task
Summarize the past week's core metrics and produce a health report posted in the conversation.
## Input Information
### Required Items
- Core panel list: [the dashboards and traces that must be reviewed every week]
- Time range: [the past week, mind the time zone]
### Optional Items
- Carry-over items from last week: [entries marked for continued observation in last week's report?]
## Output Specification
Health report: weekly trends of core metrics, anomaly time segments (with direct links),
early signals worth attention, and comparison with last week's report.
## Constraints
- ✅ If data is missing, state what's missing — don't guess
- ✅ Only produce reports; all changes are left for my approval
- ❌ Do not change alerts
- ❌ Do not touch production settings🔧 Investigation Report Quality Feedback (write the criteria into the skill)
Expected outcome: you point out what this report is missing, the Bot writes the requirements into the "Performance Investigation" skill, and the next one is noticeably more solid.
## Role
You are my performance investigation specialist, and you have just delivered an investigation report.
## Task
Update the reporting criteria of the "Performance Investigation" skill according to my feedback.
## Input Information
### Required Items
- Problems with this report: [links won't open / screenshots missing time range / facts and hypotheses mixed together...]
- The correct criteria: [what you want every future report to achieve]
### Optional Items
- Additional requirements for specific metrics: [e.g., latency metrics must include P95 values and time zone?]
## Output Specification
First restate the new criteria you've summarized; update the skill only after I confirm.
## Constraints
- ✅ Criteria are written into the skill and take effect from the next investigation
- ✅ Only supplement the criteria; do not change the approval boundaries
- ❌ Do not cut evidence items for convenienceCheckpoint ✅
- The Bot can open all three types of tools: connectors show Installed (not Needs auth / Disconnected), and the intranet dashboard routing is working
- The first report is complete: hotspot, screenshots, direct links, facts and hypotheses separated, suggested next steps, list of unverifiable items
- Every direct link opens and traces back; rapidly changing metrics have source links or exports, not just screenshots
- Read-only throughout: no alerts changed, no production settings touched; changes appear only in "suggestions"
- The investigation workflow is saved as a skill and can be re-run by name for another incident; the routine only produces health reports
Pitfall Reminders
❌ Letting the Bot "casually" adjust an alert threshold
The official starter prompt's boundary, verbatim: "Do not change alerts or production settings"; the official positioning of routines, verbatim: "not for unsupervised production changes". The right approach: write change suggestions into the report and leave execution to approval — you act only after reviewing, so there's a traceable record if something goes wrong.
❌ Treating a single dashboard screenshot as all the evidence
The official evidence checklist explicitly says: "Do not rely on a screenshot alone for rapidly changing data". Performance data changes fast, and a screenshot is only a snapshot of one particular second. The right approach: attach a direct link to every conclusion, and for rapidly changing metrics have the Bot include a link to the source system or an export.
❌ Mixing facts and hypotheses together
The official starter prompt specifically requires "Separate facts from hypotheses". "The data shows P95 rose by 300ms" is a fact; "it's most likely a database query in the new version" is a hypothesis — mixed together, readers will take the hypothesis as a conclusion. The right approach: following the official five-part split, have the Bot write in sections — facts, hypotheses, completed, pending approval, unresolved.
❌ Giving up when internal monitoring tools won't open
The Bot's cloud computer is in the cloud, so dashboards that only open inside the company intranet are invisible by default. The official solution: Settings → Computer → Route traffic through this computer, which makes the Bot's browser use your desktop's network and IP. Note two things: sites resolving to private/VPN addresses are blocked by default and need organizational approval; only one computer can route at a time. For details, see 5.3 Traffic Routing and Local Execution.
Advanced Tips (Optional)
- Auto-draft investigations on incidents: officially supported incident trigger sources include Sentry and PagerDuty — configure an incident routine for the "Performance Investigation" skill, and as soon as an incident comes in, an investigation report is automatically produced and posted in the conversation. The boundary is unchanged: reports can be generated automatically, but changes always go through approval. See 3.3 Routines.
- Archive health reports as Pages: have the Bot organize the weekly health reports into Pages (official positioning: readable pages like reports) and store them in the Bot's Library — comparing with last week's trends is faster than scrolling through chat history. See 2.2 Files and Reviewable Artifacts.
- Hand off to Bug reproduction: once the investigation pinpoints a specific version or feature, hand the report along with the evidence to B1 Bug Reproduction — the workflow there turns it into a reproduction package developers can directly locate.
- Make asking for evidence a conversation habit: the official efficient-conversation checklist includes "ask for evidence" — keep insisting that every conclusion carries a link in your everyday conversations with the Bot, and report quality will be much more stable.
Lesson Summary
- ✅ One "performance investigation specialist" Bot + three types of data sources (observability, analytics, incident tools) turns performance incidents into evidence-backed, targeted AI performance troubleshooting
- ✅ Start with the official starter prompt: look at dashboards and traces, locate the highest-confidence hotspot, short report with screenshots and direct links, facts and hypotheses separated
- ✅ Hold the two official red lines: don't change alerts, don't touch production settings; write changes only as suggestions and leave execution to approval
- ✅ Once refined and reviewable, save as a skill and test with the next incident; routines only do periodic health reports
| Key Item | Value of This Lesson |
|---|---|
| Bot Role | Product performance (performance investigation specialist) — evidence-backed targeted performance investigation |
| Connected Apps | Observability (dashboards/traces), analytics, incident tools; intranet tools via local routing |
| Starter Prompt | Official Product performance starter prompt (localized version) |
| skill | "Performance Investigation" (official six elements, invoked by name, shared across all Bots) |
| routine | Weekly health report (optional); event triggers only produce reports, no production changes |
| Approval line | No alert modifications, no production setting changes; change execution always stays behind approval |
Next lesson preview
Next lesson B3 Using a cloud PC as a development environment: Once investigation and reproduction run smoothly, get to know the machine the Bot works on. You will learn:
/workspacePersistent workspace: where files go and how to share them across Bots- Have the Bot run scripts, install dependencies, and verify the environment on the cloud PC
- Directories like node_modules and venv are not within the sync scope; reinstall after updates
- How the Bot handles a nearly full disk (report the largest files first, delete only with consent)