ServiceNow CoreAI has introduced AutoSynthData, a system designed to transform the capability gaps of AI agents into high-quality synthetic training data. The pipeline addresses the difficulty of creating training tasks that are feasible, realistic, and sufficiently difficult for a specific enterprise environment.
ServiceNow CoreAI Unveils AutoSynthData to Generate Synthetic Training Data for Enterprise Agents
AutoSynthData operates by identifying patterns in tasks where a target model struggles. Using a stronger "teacher" model to demonstrate successful behaviors, the system generates new, executable tasks that include a system specification, a user prompt, and a verifier. These tasks are then validated through a rigorous quality-control loop involving execution in the target environment and automated verification to ensure they are both solvable and sound.
The dataset is built in two phases: a "target" phase that generates core samples and a "multiply" phase that creates novel variants of those samples. This dual approach provides a path toward training-scale datasets while preventing data drift.
In experiments conducted using the EnterpriseOps Gym benchmark, the pipeline demonstrated significant performance improvements. When using Gemma-4-26B-A4B-it as the target model and Qwen3.8-27B as the teacher in a Hybrid domain, fine-tuning on 2,000 synthetic samples generated over 18 hours resulted in a 35% relative improvement in mean Pass@1. The approach also showed effectiveness in the ITSM domain, where synthetic supervised fine-tuning (SFT) raised the mean Pass@1 from 18.77% to 27.18%.
Sources
- AutoSynthData: Generating Training Data for Enterprise Agents （Hugging Face Blog, 2026-10-02）