May 12, 2025

Agents are multiple LLMs that talk to each other and sometimes make minor decisions. Each agent is allowed to either use a tool (e.g., search the web, read a file, make an API call to get the weather) or to choose from a menu of options based on the information it is given.

Chat assistants can only go so far, and many repetitive business tasks can be automated by giving LLMs some tools. Agents are here to fill that gap.

But it is much harder to get predictable and accurate performance out of complex LLM systems. When agents make decisions based on outcomes from each other, a single mistake cascades through, resulting in completely wrong outcomes. And every change you make introduces another chance at making the problem worse.

So with all this complexity, how do you actually know that your agents are doing their job? And how do you find out without spending months on debugging?

First, let’s talk about what LLMs actually are. They convert input text into output text. Sometimes the output text is an API call, sure, but fundamentally, there’s stochasticity involved. Or less technically speaking, randomness.

Example: I ask an LLM what coffee shop I should go to based on the given weather conditions. Most of the time, it will pick the closer one when there’s a thunderstorm, but once in a while it will randomly pick the one further away. Some bit of randomness is a fundamental aspect of LLMs. The creativity and the stochastic process are two sides of the same coin.

When evaluating the correctness of an LLM, you have to look at its behavior in the wild and analyze its outputs statistically. First, you need  to capture the inputs and outputs of your LLM and store them in a standardized way.

You can then take one of three paths:

  1. Manual evaluation: a human looks at a random sample of your LLM application’s behavior and labels each one as either “right” or “wrong.” It can take hours, weeks, or sometimes months to start seeing results.
  2. Code evaluation: write code, for example as Python scripts, that essentially act as unit tests. This is useful for checking if the outputs conform to a certain format, for example.
  3. LLM-as-a-judge: use a different larger and slower LLM, preferably from another provider (OpenAI vs Anthropic vs Google), to judge the correctness of your LLM’s outputs.