Some field notes from building AI agents.
1. An agent silently skips the steps that produce no visible output. A step that prints a line, even one that says “nothing”, reliably runs. So make every required step leave a mark, and treat a missing mark as a defect.
2. An agent confirming to you that the work happened doesn’t mean it actually happened. A rendered “done” could be just another confident self-report hallucinated by the agent. The real test of completion is the file on disk, not the text on the screen.
3. How do you know an agent ran the real step? Put a value in the output that the model cannot reproduce, such as a version number or a hash. If that value is present, the step ran. If it is missing or stale, the agent hallucinated.
4. Almost every real bug I found came from shipping a small change and watching what actually reached the disk. It did not come from unit test cases where I thought about what might go wrong. The invisible failures mean you cannot reach them by thinking. Ship, then look.
5. Sometimes an agent cannot solve a problem. Not because the problem is hard, but because the agent is buried in too much context. The fix is not more context. Start a fresh sub-agent with none. It gets the same problem and a clean slate.
6. Docs decay. Atoms do not. Word documents and slides get versioned, archived, and forgotten. The knowledge was real; the format let it rot. Distill each document to its core points, and store those as small plain files — atoms — that your AI reads directly.
7. You do not need a frontier model for every agent. Use the expensive model for the supervisor that plans and reviews the work. The workers under it handle narrow, well-defined tasks, so cheaper open-source models are enough.
8. Do not keep your AI’s memory in a vendor’s database. Keep it in a place that you own. Then, if you switch models or lose one, you lose nothing. I move across models without losing context.
9. Do not run everything on a single model. Spread your work across several. When one goes down or degrades, the others keep running.