The first ten tests for an AI agent

Start with a task

Take a support agent that can update a ticket. A request to “close the duplicate ticket” gives you a useful starting point. A successful run resolves the right ticket, checks that the requester can close it, and confirms the update using the service’s response.

Build each case around the initial state, the request, and the observable outcome. Record the tool calls and the final ticket state alongside the response. The user-facing message alone cannot tell you whether the right record changed.

Write ten cases

1. A clear request closes the intended ticket and records the reason.
2. A follow-up request uses the ticket selected earlier in the conversation.
3. Two tickets match, so the agent asks which one the user meant.
4. No tickets match, so the agent explains what it searched for.
5. The requester lacks permission, so the ticket stays open.
6. The ticket is already closed, so the agent explains its current state.
7. The service times out before the update can be confirmed.
8. A retry after an uncertain response leaves the ticket in the intended state without a duplicate update.
9. The service rejects a required field and the agent explains what information is missing.
10. The service returns a partial result, so the agent reports what it could verify.

Check the state

Separate checks for tool selection, authorisation, and the resulting state. The expected outcome for an ambiguous request could look like this:

case: two_matching_tickets
request: Close the duplicate login ticket.
expected:
  response: asks which ticket to close
  write_calls: 0
  ticket_statuses:
    INC-203: open
    INC-208: open

Use a judge for qualities such as whether a clarification is understandable. Use assertions for facts your test harness can read directly.