Worknano

How to test an AI agent before you trust it

Updated 2026-11-15

To test an AI agent, give it real cases from your business where you already know the right result, and compare. An AI agent is a program that uses AI to carry out tasks for you, such as sorting enquiries or drafting replies. Because the same request can produce slightly different results each time, judge it on a whole set of cases and never on a single try.

Why a quick try is not a test

Most agents are "tested" by typing two or three requests and reading the answers. They look fine, and the agent starts work.

It is like hiring a shop assistant after watching them serve one friendly customer. The real test is the customer with a complaint, the one who wants two things at once and the one whose order cannot be found. Agents rarely fail on typical cases. They fail on the half-filled form, the email with two requests in it and the customer who is in your list twice. A real test contains the cases you would rather not think about.

A five-step test plan

  1. Say what "right" means. Write it down. For an agent that sorts enquiries: the right category and the right urgency. For a draft reply: correct facts, a friendly tone and no promise the company cannot keep.
  2. Collect real cases. Take twenty to thirty from the last few months. Mix ordinary ones with awkward ones. Add a few where the right answer is "I cannot do this, a person must look".
  3. Write down the expected result first. Before the agent runs, note for each case what a good colleague would have done. If you look at the agent's answer first, you will be tempted to accept it.
  4. Run all cases and mark each one. Right, acceptable or wrong. For every wrong result, add one line on the reason.
  5. Change one thing and run everything again. A fix for one case often breaks another. Only the full set shows that.

Cases people forget to include

What counts as a pass?

There is no single number. It depends on what happens to the agent's work. If a person reads every draft before it goes out, some "acceptable" results are fine, as long as the wrong ones are easy to spot. If the agent acts without anyone checking, it needs far more right results, and its remaining mistakes must be harmless and easy to undo.

Look at the kind of mistake as well as the count. Ten clumsy sentences matter less than one wrong amount.

Keep the test cases

The collection you built stays useful. Run it again whenever you change the agent's instructions. Each time the agent gets something wrong in daily work, add that case.

From test to trust

Passing the test does not mean full freedom. Start with a person confirming each result. Watch how often they have to correct it. Then give the agent more freedom step by step. The test tells you the agent is ready to work under supervision. The supervised weeks tell you when it needs less of it.

How Worknano handles this

In Worknano you edit an agent under Config, in Agents. There you set its instructions, its tools and its AI model. To try a variant, duplicate an agent and change its instructions. You can run an agent from a button on a record, or on a whole list of selected records at once, which suits a set of real test cases. Changes to records arrive as previews for you to confirm. The product guide gives the same advice as this article: test on a few real records, then adjust the instructions before you change the model.

Create a free workspace and run an agent on thirty of your own cases.