Recently, someone asked me a question that caught me off guard:
“What evaluation metrics are you using for your AI agents?”
I didn’t have a good answer.
Not because we weren’t testing our agents.
We were testing them constantly.
We would create different questions for the same use case, try different ways of asking the same thing, check whether the right knowledge was retrieved, see whether the correct tools were being called, investigate unexpected behavior, monitor response times, and repeat the entire process after making changes.
I just hadn’t stopped to think about all of that as agent evaluation.
A lot of this mindset was also shaped by my senior manager, who consistently pushed me to go beyond the obvious happy path when testing our agents.
Instead of stopping when an agent “worked,” I was encouraged to keep asking:
What happens if we change the question?
What happens if the agent retrieves something unexpected?
What happens if it chooses the wrong tool?
What happens if we approach the same use case from a different angle?
At the time, I simply thought of it as being rigorous with our testing.
Looking back, I realize that this was already building the foundation of a proper evaluation process.
That question made me take a step back and look at how we actually evaluate an agent.
And I realized that evaluating an agent is much more than checking whether its final answer is correct.
So, What Are We Actually Evaluating?
For me, it comes down to a few practical questions.
1. Did the agent understand the user’s intent?
A user can ask the same thing in many different ways.
If an agent only works when the question is phrased exactly the way we expected, it isn’t really robust.
So instead of testing one question, we test multiple variations of the same intent.
Different wording. Different levels of detail. Short questions. Longer questions.
The goal is to see whether the agent understands the intent, not just a particular sentence.
2. Did it retrieve the right information?
For agents using knowledge sources, getting the final answer right isn’t enough.
We also need to know whether the agent retrieved the right information to produce that answer.
If the user asks about a specific policy, we want the relevant document or content to be retrieved—not something that merely looks related.
This is where retrieval quality and groundedness become important.
A good LLM can’t magically fix consistently bad retrieval.
3. Did it use the right tools?
This is one of the areas that makes agent evaluation different from simple chatbot testing.
If an agent has multiple tools available, we need to check:
Did it choose the correct one?
And even if it did:
Did it pass the correct parameters?
An agent might correctly identify that it needs to call an API but still provide the wrong argument.
So we evaluate both tool selection and tool execution.
We also look for unnecessary tool calls.
If a simple question requires four tool calls when one should have been enough, that’s something worth investigating.
4. Was the response correct, grounded, and fast?
Eventually, we still have to evaluate the final response.
Was it correct?
Was it relevant?
Was it supported by the information retrieved?
Did it hallucinate anything?
And how long did it take?
Because a technically correct answer that takes 30 seconds isn’t necessarily a great user experience.
Latency can also reveal deeper problems—unnecessary retrieval, repeated tool calls, inefficient orchestration, or other bottlenecks.
Then We Try to Break It
One of the most important parts of our testing is not just checking the happy path.
We try different things:
- Ambiguous questions
- Unexpected wording
- Similar but different intents
- Missing information
- Multiple questions in one message
- Questions outside the knowledge base
- Scenarios requiring different tools
We’re not just trying to prove that the agent works.
We’re trying to find how it fails.
Because those failures tell us what needs to change.
Evaluation Is a Continuous Loop
For me, the process now looks something like this:
Test → Observe → Diagnose → Improve → Test Again
Maybe the agent retrieved the wrong document.
Maybe it selected the wrong tool.
Maybe it called the right tool with incorrect parameters.
Maybe the answer was correct but unnecessarily slow.
We identify the failure, make a change, and run the tests again.
And importantly, we don’t just rerun the exact same question.
We introduce new variations to make sure the improvement wasn’t specific to one test case.
The Realization
Looking back, the interesting part of that question wasn’t that I didn’t know the formal terminology.
It was realizing that we were already doing many of the things that fall under agent evaluation.
We were asking:
Did it understand the intent?
Did it retrieve the right information?
Did it use the right tool?
Was the response grounded?
How long did it take?
What happens if I change the way I ask the question?
What happens when I give it something unexpected?
I just hadn’t connected all of those questions to a formal evaluation mindset.
And that changed the way I think about testing agents.
Testing asks: “Does this scenario work?”
Evaluation asks: “How well does the agent work, where does it fail, and is it getting better?”
That’s an important distinction.
Because when we’re building AI agents, we shouldn’t only ask:
“Does it work?”
We should also ask:
“How do we know?”