Evaluating the Datadog MCP server is hard.
Because the Datadog MCP server has to handle ANYTHING a user might want to do: search logs, query metrics, filter by status codes, IP addresses, time ranges, boolean operations… on and on.
And they’re evaluating those BAJILLION scenarios against live data. Data that changes.
In this episode of Datadog Illuminated, Scott Yak explains how he and Datadog’s Applied AI team evaluate the Datadog MCP server, which has to handle a limitless amount of possible human requests, against live data that can go stale before they finish testing it.
They found a way to generate that many evals without drowning in toil, and it’s a spicy one. ♫
Full episode here: youtu.be/HZ4n1h8j8…