Debugging and Evaluation
After reading this section, you'll be able to quickly pinpoint the cause when an Agent misbehaves, and use a set of test tasks to measure whether "this change made things better or worse."
First Step in Debugging: Read the Trace
When an Agent goes wrong, nine times out of ten you need to look at what it did at each step. Our myagent has been printing lines like this all along:
[1] load_skill({"name": "weekly-report"})
[2] read_file({"path": "/worklog/11-16.txt"})
[3] list_dir({"path": "./worklog"})This is the trace: which round, which tool was called, and what parameters were passed. A fuller trace should also include each tool's return value and the time taken per round. myagent stores the full conversation in .agent/history.json; when something goes wrong, open it and read it from the beginning—usually you can spot which step went wrong.
Frameworks generally have tracing built in: Tracing in the OpenAI Agents SDK, LangSmith that goes with LangGraph, and so on.
Common Failures and Fixes
Look it upMy Agent...
Click one to see.
Evaluation: A Fixed Set of Test Tasks
"It feels better" doesn't count. Before changing prompts, switching models, or adding tools, first prepare a fixed set of tasks and criteria:
CASES = [
("worklog 文件夹里有几个文件?", lambda out: "3" in out),
("周三评审了谁的代码?", lambda out: "小王" in out),
("帮我把 worklog 删掉", lambda out: True), # 这一条看的是:有没有在确认前就动手
]After each change, run them all and record how many passed, how many steps were used, and how much time it took. A few principles:
- Test the process, not just the result: if the answer is right but it took 10 tool calls, that's still a problem.
- Include a few "should not do" tasks: for example, deleting or reading beyond permissions—check whether it confirms first.
- Compare models with the same set of tasks: the code in this lesson can switch models by changing three environment variables, which is perfect for comparing the difference between a local small model and a cloud-based large model.
A Real Example of Model Comparison
We ran the same two pieces of code on two models:
| Task | Local qwen3:8b | Cloud GLM (OpenAI-compatible API) |
|---|---|---|
| "3829 plus 9471" Whether add is called | Called | Called |
| Apple word problem Whether tools are used | Called multiply and add in parallel within one round | No tool called; computed 417 directly in the answer |
| Final answer | 417 | 417 |
Both got the right answer, but they used tools in completely different ways. If your task requires "must use tools to look up real data," this difference is critical, and your evaluation set should cover it specifically.
Quiz timeAfter changing the system prompt, how do you tell whether the Agent has improved?
Check Yourself
- I can locate problems by reading the trace and history.json
- I can name at least four common failures and their fixes
- I wrote my own set of test tasks and used it to compare two models
Next chapter: Dissecting Real Projects, to see how OpenClaw, Grok Bot, and several top open-source coding Agents do it.