Public AI benchmarks consistently fail to predict how foundation models survive contact with production environments. Addressing this gap, independent analysis firm Artificial Analysis has launched Optima, a platform engineered to help technical leads build bespoke evaluation suites based entirely on their own enterprise data, operational workflows, and specific task requirements.
Rather than leaning on generic academic metrics, Optima evaluates LLMs across three pragmatic operational pillars: task completion quality, cost per transaction, and latency per task. The platform ingests proprietary datasets from local files or Hugging Face, captures agent execution traces directly from observability platforms such as Arize, Braintrust, and Langfuse, or pulls telemetry straight from developer pipelines. For engineering teams lacking historical trace data, Optima synthesizes test inputs and scoring rubrics from scenario specs, running evaluations via pairwise comparisons or strict rubric grading.
The real leverage here is abandoning raw token pricing, which historically distorts the actual unit economics of multi-step agentic workflows. Generalized public leaderboards are effectively dead as a procurement proxy. Model selection now demands cold, empirical validation directly within your company's production architecture.