ServiceNow AI Research has released EVA-Bench Data 2.0, a significant new benchmark designed to evaluate the sophisticated tool-using capabilities of large language models. The open-source dataset, announced on the Hugging Face blog, introduces 121 unique tools and 213 multi-step scenarios, aiming to simulate the complexity of real-world digital tasks. This initiative addresses a critical need for more realistic testing environments for AI agents.
A More Realistic Test for AI Agents
Existing benchmarks often test an AI's ability to use a single tool in isolation. However, real-world problems frequently require an agent to plan and execute a sequence of actions using multiple tools—a process known as tool orchestration. EVA-Bench 2.0 is specifically designed to measure this advanced capability, pushing models beyond simple command execution to demonstrate genuine problem-solving skills.
According to the release, this new dataset provides a much-needed standardized method for developers and researchers to gauge how well their AI agents can handle complex workflows. The goal is to move beyond theoretical capabilities and measure practical performance in scenarios that mirror human interaction with digital applications.
What's Inside EVA-Bench 2.0?
EVA-Bench 2.0 significantly expands on previous evaluation datasets by focusing on diversity and complexity. The framework is built to challenge LLMs in ways that single-function tests cannot. For AI developers building the next wave of autonomous agents, understanding these metrics is crucial. To stay ahead of the curve, consider joining thousands of AI professionals who get weekly insights from the AI Breaking Wire newsletter.
Key components of the new dataset include:
- 121 Unique Tools: A diverse collection of digital tools an AI can learn to operate.
- 213 Complex Scenarios: Multi-step problem-solving tasks that require planning and sequencing tool usage.
- 3 Distinct Domains: Scenarios are spread across different domains to ensure the AI's skills are generalizable and not overfitted to one type of task.
At its core, the benchmark tests an agent's ability to select the right tool, provide the correct parameters, and chain multiple tool calls together to reach a final goal. The scenarios are designed to be challenging, requiring an average of 4.3 tool calls per solution, a significant step up in complexity from previous benchmarks.
Why It Matters
As AI agents become more integrated into software and business workflows, their ability to reliably use digital tools is paramount. EVA-Bench 2.0 provides the community with a robust and transparent yardstick to measure real progress. By creating a challenging environment that mimics real-world complexity, ServiceNow is helping push the entire field toward creating more capable, reliable, and genuinely useful AI agents that can automate tasks and augment human productivity.