Evaluating the Datadog MCP server is hard.

Because the Datadog MCP server has to handle ANYTHING a user might want to do: search logs, query metrics, filter by status codes, IP addresses, time ranges, boolean operations… on and on.

And they’re evaluating those BAJILLION scenarios against live data. Data that changes.

In this episode of Datadog Illuminated, Scott Yak explains how he and Datadog’s Applied AI team evaluate the Datadog MCP server, which has to handle a limitless amount of possible human requests, against live data that can go stale before they finish testing it.

They found a way to generate that many evals without drowning in toil, and it’s a spicy one. ♫

Full episode here: youtu.be/HZ4n1h8j8…

Split-screen Datadog Illuminated thumbnail. On the left, Scott Yak, an Asian man in a gray blazer, smiles at the camera, with "with SCOTT YAK" in yellow pixel-style text below him. Center background shows a lightboard covered in green handwritten notes about MCP server evals. Large yellow and white pixel-style text reads "EVALUATING THE DATADOG MCP SERVER" over the board. On the right, Whitney Lee laughs with her hand raised, wearing a black floral top, against a purple brick background with neon zigzag shapes. The Datadog Illuminated logo appears in the top left corner.