OpenAI's Regression Workflow tracked more than 1,700 investigations and root caused over 600 cases in just over three months. The operation relies on the Performance Factory, an autonomous system that uses Codex and internal models to find and fix regressions at scale, running in the cloud and automating investigating and resolving performance problems.

Traditional engineering approaches to performance management have collapsed under the operational overhead of software delivery. As OpenAI documented, the pace of agentic development has made traditional approaches to performance engineering unsustainable, forcing the company to replace human log reviews with automated cycles that resolve latency spikes before human operators intervene.

The operational issue facing large-scale inference providers centers on labor costs: engineering teams cannot scale headcount linearly with the alerts generated across complex microservices. At OpenAI, maintaining latency baselines previously required engineering teams to accumulate more than 30 performance tools, from CPU monitors to allocation profilers, to diagnose recurring degradations such as ChatGPT first-token latency.

The Shift to Autonomous Forensics

Before deploying Codex into their operational loops, OpenAI engineers manually examined metrics and logs, compared traces, debugged live machines, and read code to understand why services decelerated. This manual debugging required deep, service-specific expertise that created severe operational bottlenecks. Signals were easily missed, and each alert forced senior staff to navigate fragmented instrumentation.

To standardize this workload, OpenAI packaged its performance team's investigation process into a shared skill called perf-forensics. This skill tells Codex which tools are available, when to use them, and how to work through a slowdown by establishing when the regression began, identifying affected requests, and checking recent deployments or configuration updates.

"We built the Performance Factory, an autonomous system that uses the full capabilities of Codex and our models to find and fix regressions at an unprecedented speed."

Using this shared skill alongside evaluations based on past regressions, Codex evaluates an incoming system alert and produces a concrete diagnosis in a single automated session, removing the requirement for an engineer to steer every analytical step.

How do operators prevent these agentic sessions from piling up in unstructured communication channels?

They convert the entire diagnostic pathway into an automated pipeline that directly authors and validates code. OpenAI initially saw engineers act as copy-paste proxies between monitoring alerts and Codex, eventually running cloud sessions that merely dumped findings into Slack threads where regressions still had to be manually sorted and addressed. The Performance Factory consolidates these disjointed steps into an end-to-end operational loop.

Architecture and Workload Throughput

The infrastructure powering the Performance Factory relies on agent workflows running in the cloud that automate investigating and resolving performance problems. Engineers configure these operational pipelines by submitting configuration files and instructions written in Markdown directly to the task service.

Can autonomous agents truly replace deep engineering expertise when diagnosing complex failures in distributed systems without human oversight? The system design indicates that human validation remains intact at the final approval stage. In the Regression Workflow, an autonomous agent investigates a slowdown, records its findings in the task, and proposes a fix for an engineer to review. After deployment, another agent picks up the same task to check whether performance has returned to baseline.

Rather than eliminating human authority, the infrastructure automates telemetry gathering, initial code authoring, and verification. Through this pipeline, OpenAI sustains an average of 17 automated investigations every single day.

Against the field

Moving traces, logs, and diagnostic scripts into an automated pipeline around Codex replaced constant engineer handoffs with a unified self-running loop. The shift ultimately allows OpenAI to clear an average of 17 automated investigations daily.

Artificial IntelligenceAutomationOpenAIWorkflow Automation