Pattern
How to see what an AI agent did and why
You see what an AI agent did and why by recording a trace of every run, each model call, tool call and decision as a timed step, plus a separate audit log of every action that changed something and who approved it. Traces are for debugging; the audit log is the permanent record.
Explained from public sources. Mai has not built this yet.
On this page
In plain English
When a person on your team does something odd, you can ask them why. An agent cannot reliably tell you afterwards, so the system has to write it down as it goes. A trace is the step-by-step diary of one piece of work: what the agent was asked, what it looked up, what it decided and what it did. An audit log is the shorter, permanent ledger of every action that changed something, such as an email sent or a record updated, along with who approved it.
How it works
The two records serve different readers, so I keep them separate.
Traces follow a single run. The usual model, borrowed from ordinary software monitoring, is a trace made of nested spans. A span is one timed operation with a start, an end and some details, and spans sit inside each other: an agent run contains model calls, each model call may lead to tool calls, a tool call may hand off to another agent. The OpenAI Agents SDK, for example, records a trace for every run by default, with spans for model generations, tool calls, hand-offs between agents and guardrail checks.
A useful span records:
- the inputs and outputs (prompt, reply, tool arguments, tool result)
- the model and settings used, token counts and time taken
- the parent span, so the whole tree can be rebuilt
- identifiers that link it to the job, the conversation and the user
OpenTelemetry, the open standard most monitoring tools understand, now maintains semantic conventions specifically for generative AI, covering model inference, agents, tool execution and MCP. Emitting traces in that shape means they can go into whichever observability tool a business already runs.
The audit log records outcomes, not reasoning. Each entry says what changed, in which system, on whose behalf, which agent proposed it, who approved it and when, with a reference back to the trace for anyone who needs the detail. It is append-only: entries are added, never edited.
Agent run. Then Model calls, Tool calls, Hand-offs. Then Trace. Then You approve. Then Audit log.
There is a UK angle. The ICO’s guidance on individual rights in AI systems says that where an AI system makes decisions about people, those decisions should be recorded, including whether the person asked for human review or contested the outcome and whether it changed, and that the records should be monitored. The ICO notes that its AI guidance is under review following the Data (Use and Access) Act 2025, so check the current version, but a good audit log is how you would meet that expectation either way.
Where it goes wrong
- Logging everything, forever. Full traces contain whole emails, customer details and documents. Keeping them indefinitely creates a store of personal data with no purpose. Traces need a short retention period and restricted access; the audit log can be longer-lived precisely because it holds less. The OpenAI Agents SDK, for instance, has a setting to stop recording model and tool inputs and outputs.
- Sending traces somewhere unexpected. Some toolkits, the OpenAI Agents SDK among them, export traces to the vendor’s own service by default. That may be fine, or it may send client data outside the place you agreed to keep it. Check where traces go before the first real run.
- Trusting the agent’s explanation. A model’s stated reasons are generated text, not a record of how it reached the answer. The trace of inputs, tool calls and results is the evidence; the explanation is at best a summary.
- Logs no one reads. A trace store is only useful if someone looks. Failed tool calls, refusals and repeated retries should raise an alert, not wait for a complaint.
- Editable audit logs. If the agent’s own credentials can alter or delete audit entries, the log proves little. Write it to a store the agent can append to but not change.
- Missing links. A trace without the job ID, or an audit entry without the trace ID, cannot be joined up when you need to reconstruct a week-long job.
How Mai AI uses it
In a Mai AI system every agent run produces a trace, and every action that changes something outside the system writes an audit entry that names the agent, the action, the approver and the trace it came from. I design retention separately for each: traces kept short and access-controlled, the audit log kept as the business record. Traces stay inside the infrastructure the client controls unless they decide otherwise.
The AI chief of staff I built for a UK property group works this way at the level the case study describes: every action is checked against what the agent is allowed to do, anything that matters needs a signed approval token, and the whole trail is written to an audit log. Traces are also the raw material for testing agents: a run that went wrong becomes a test case so it cannot go wrong the same way twice.
If being able to answer “what did it do, and who said yes?” matters to your business, it is part of every custom AI operations system I design.
Questions and answers
What is the difference between a trace and an audit log for AI agents?
A trace records the detail of one run, every prompt, model reply and tool call with timings, and is mainly for debugging. An audit log records only the actions that changed something in the business, who or what did them and who approved them, and is kept as the permanent record.
How do you debug an AI agent?
Read the trace of the run that went wrong. It shows what the agent was given, which tools it called with what arguments, what came back and what it decided, so you can see whether the fault was the instructions, the data, a tool or the model.
Should AI agent logs include personal data?
Only what you need. Traces often capture full prompts and replies, which can include personal data, so set retention limits, restrict who can read them and redact where you can. Some agent toolkits let you switch off recording of inputs and outputs.
Is there a standard format for AI agent traces?
OpenTelemetry, the open standard for software telemetry, maintains semantic conventions for generative AI covering model calls, agents, tool execution and MCP. Using them lets traces flow into existing monitoring tools rather than a single vendor's dashboard.
Harnesses known for this
- Harness profileOpenAI Agents SDK
The OpenAI Agents SDK is an open-source library, in Python and TypeScript, for building agents that use tools, hand work to each other and pause for approval. It suits teams that want a light agent loop with tracing built in, rather than a full orchestration platform.
- Harness profileDeepSeek Harness
DeepSeek Harness (dsh) is DeepSeek's open-source agent harness, in public preview, built on the idea that every feature is a plugin. It suits developers who want a self-hosted coding and task agent they can reshape, running DeepSeek models or any provider that speaks a supported API.
- Harness profileLangGraph
LangGraph is an open-source framework from LangChain for building stateful agents as graphs of steps, with saved checkpoints so a run can pause, wait for a person and resume. It is best for long-running, multi-step work where you want explicit control over every path.
- Harness profileMicrosoft Agent Framework
Microsoft Agent Framework is Microsoft’s open-source framework for agents and multi-agent workflows in .NET and Python, and the direct successor to both AutoGen and Semantic Kernel. It is best for teams on Azure or .NET that want typed, checkpointed workflows with approvals and OpenTelemetry tracing.
Related entries
- PatternHuman approval gates
Human approval gates sort every action an AI agent can take into three levels: draft only, needs a person to approve, or allowed to run automatically. The agent prepares, a person signs off anything that matters, and the system refuses any action that does not match what was approved.
- PatternTesting agents
You test AI agents with evals, a set of realistic tasks with clear pass or fail checks that you run every time the agent, its instructions or its model changes. The best tasks come from real failures, and each one runs several times because agents do not behave the same way twice.
- PatternAgent safety
You make an AI agent safe by limiting what it can do, not by trusting what it says. Give it the narrowest tools and permissions, run risky work in a sandbox, require a person to approve consequential actions, and treat every email, document and web page it reads as possibly hostile.
- PatternLong-running jobs
An AI agent can follow a task for days or weeks only if the task lives outside the agent, as a saved record of goal, progress and next step that survives restarts. The agent wakes, reads the record, does one step, writes it back and sleeps, so nothing depends on one long conversation.