AWS UAEF Applies 36 Metrics Across Four Agent Frameworks

AWS's Universal Agent Evaluation Framework (UAEF) uses 36 built-in metrics across seven dimensions, according to the technical introduction published on November 11, 2026. AWS's introduction describes the system as open source under an Apache-2.0 license.

AWS designed UAEF to keep evaluations comparable across LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore. AWS says those frameworks record messages, tool calls and agent-to-agent handoffs differently, which can force teams to rewrite metrics, scoring logic and result storage after a framework switch. The introduction says this churn makes it harder to determine whether a prompt, model or tool change improved an agent.

AWS reports that customers across 12 organizations in telecom, healthcare and financial services used UAEF to apply consistent metrics and compare agent versions before release. The awslabs/uaef repository includes the Python library, framework adapters, metrics, AWS deployment infrastructure, a web interface and runnable notebooks.

What does UAEF normalize?

AWS's framework adapters convert each agent's messages, tool calls and handoffs into a canonical AgentTrace. The introduction says the scoring engine accepts only that representation, allowing different frameworks to retain their native execution models while producing comparable results.

AWS says UAEF can adapt an existing trace from a past run, log or observability store. It can also invoke an agent through a runtime Amazon Resource Name or HTTP endpoint, then convert the resulting execution. A generic JSON adapter extends the same process to custom frameworks.

How does UAEF score an agent?

AWS groups the 36 built-in metrics into seven dimensions:

• tool calling
• response quality
• responsible AI
• performance
• multi-turn conversation
• multi-agent coordination
• reasoning

AWS normalizes scores from 0 to 1, with 1 representing the best result. Deterministic metrics calculate directly from traces and ground truth. AWS says LLM-as-a-judge metrics use a model on Amazon Bedrock to assess qualities such as relevance and safety. Custom metrics, RAGAS integrations and DeepEval integrations are also supported.

Under AWS's design, ground-truth answers are optional across UAEF, although reference-comparison metrics require them. Tool-calling evaluations can define expected tools, their order and their arguments. When a metric lacks necessary input, it returns no score with a warning. AWS says that empty result stays out of dimension averages rather than counting as zero, allowing partially labeled datasets and unlabeled production traffic to be evaluated with metrics whose inputs are available.

How do experiments expose regressions?

AWS uses Amazon DynamoDB to store experiment metadata for listing and comparison, including evaluation counts and average scores. It uses Amazon S3 for full JSON results containing per-metric scores and judge reasoning.

Each run can record the framework, model, tools, system prompt, parameters, a configuration snapshot and Git commit, which ties the score to the design under test. AWS says UAEF screens traces for personal information before storage, and users can redact stored results.

AWS's comparison engine calculates absolute and percentage changes against a baseline for metrics shared by two runs. A configurable threshold defaults to 5%, separating improvements, regressions and unchanged results. AWS grades regressions as minor, moderate or severe and groups them by dimension. The repository says this can reveal a prompt revision affecting tool calling without changing response quality.

What does the reporting layer show?

AWS's web interface presents per-test-case and per-dimension results. An optional statistical report covers score distributions, confidence intervals, outliers, metric correlations and failed test cases without using an LLM. AWS says a separate Bedrock-generated analysis can propose root causes and prioritized next steps for either one run or a comparison.

AWS says the Python library and low-code web service use the same evaluation engine and stores. Teams can run evaluations in notebooks, scripts and continuous integration and continuous delivery pipelines, or through a browser. Those routes retain the same scoring and comparison behavior.

What dependencies and security limits apply?

AWS's installation guide requires Python 3.10 or later and the uv package manager. AWS credentials are required for Bedrock-based judging and for DynamoDB and S3 persistence. AWS says teams using those features remain connected to AWS services even when their agents run on other frameworks.

AWS says UAEF treats evaluated-agent output as untrusted input to judge prompts. It sends the rubric separately, escapes the content, strictly parses judge responses and requires a verbatim evidence span before accepting a score. AWS says, “judge scores should not gate security controls; runtime enforcement belongs in a separate Amazon Bedrock Guardrails invocation path.”

The repository describes UAEF as a reference architecture designed to run inside a platform that supplies user and tenant isolation. It states: “Direct multi-tenant deployment without that host enforcement is unsupported.” Each tenant needs a separate deployment with its own AWS account or, at minimum, its own stack, DynamoDB tables, S3 buckets and Cognito user pool.

For enterprises running several agent frameworks, AWS positions UAEF as a stable scoring baseline that can survive prompt, model, tool and implementation changes. AWS says the remaining work sits outside the library: teams must supply suitable metrics and datasets, while multi-tenant use depends on isolation supplied outside UAEF.