In this Datadog Illuminated episode, Scott Yak & I discuss how evals for the Datadog MCP server run against live data that goes stale FAST

How did Scott & team solve this?

See the lightboard below & read the board notes here: gist.github.com/wiggitywh…

Handwritten green, yellow, and white notes on a black lightboard titled "Evaluating the Datadog MCP Server." Left column shows "Why eval MCP server?" with a diagram of a query flowing from a person through an Agent (LLM), Datadog MCP, and Datadog Backend and back. Below it, "How eval MCP server?" defines eval = Agent harness + MCP server + Q&A pair, with Actual ≈ expected response (if passing). Middle column shows "Automate eval generation," a flow starting with a seed query fanning out into "FUZZING" to make many question variants, with a note that one seed produces many Q/A pairs and evals get re-run to update answers. Right side lists "Benefits of Great Evals" in four numbered sections: Speed (fast to generate, fast to run, see impact quickly), Visibility Into Progress (high quantity of evals, tagging, traces), Mutual Benefit (Datadog AI agents like Bits AI SRE and Bits AI Assistant use the Datadog MCP, so a better MCP server means better agents), and Devs Like Writing Evals (fast feedback, showing impact, preventing regression).