What I learned from evaluating multi-agent LLM systems.

Field note / 2026

A comparison needs a shared task

Multi-agent systems can look impressive in a diagram. But more agents do not automatically mean better answers. To compare systems fairly, I had to give them the same questions and judge their answers by the same rules. If the setup changes too, it becomes hard to tell what caused the difference.

Look beyond a single score

One overall accuracy score only tells part of the story. With financial questions, an answer can go wrong in several places: understanding the question, finding the right figures, doing the calculation, or formatting the answer. Looking at those mistakes gives me a clearer idea of what to improve than a score on its own.

Complexity has to earn its place

Every extra agent or tool call adds another moving part. If I were choosing a system for a product, I would also want to understand its speed, cost, consistency, and how easy it is to debug. Those questions matter alongside answer quality, especially when the more complex option does not clearly help.

My takeaway

My main takeaway is that evaluation is part of building the product. Be clear about the task, compare against a baseline, look closely at the errors, and explain what the data can and cannot tell you. That makes the result easier to trust and gives you a practical next step.

This note draws on my completed MSc thesis project. Results and details should be read in the context of its dataset and experimental setup.