Innovation & Tech

AWS Ranks 70-Plus Models on Quality, Speed and Cost in SageMaker

AWS launched the SageMaker JumpStart Model Leaderboard on November 18, comparing 70-plus models on three benchmarks, latency, throughput and hourly instance cost in a single view.

By Amara Osei

3 min read

AWS Adds Quality, Speed and Cost Rankings to SageMaker JumpStart
AWS Adds Quality, Speed and Cost Rankings to SageMaker JumpStartTumisu / pixabay

What's News

  • AWS launched the SageMaker JumpStart Model Leaderboard on November 18, 2026, covering more than 70 models.
  • Launch coverage includes models from Meta, Google, Qwen, IBM and Arcee.
  • The board reports three benchmarks: BIG-Bench Hard, GPQA Diamond and MMLU-Pro, with scores kept separate.
  • The hourly cost estimate excludes data transfer, storage, inference-component charges and discounts.
  • GPQA Diamond results for reasoning models are averaged over 10 passes.

Amazon Web Services launched the SageMaker JumpStart Model Leaderboard on November 18, 2026, giving SageMaker Studio users a single view of benchmark results, response performance and estimated instance pricing across more than 70 models. The board covers models from Meta, Google, Qwen, IBM and Arcee at launch and carries a shareable URL. It sits beside Categories and All models on the JumpStart base-models page and opens directly to large language model rankings.

The launch targets a selection problem created by JumpStart's catalog of hundreds of open-weight foundation models. Published results often come from different prompts, settings and hardware, and deployment behavior and cost vary by configuration. AWS says it runs each benchmark with a common evaluation harness and settings across the models it tests.

What does the leaderboard measure?

The board reports three quality benchmarks:

  • BIG-Bench Hard, covering multistep reasoning
  • GPQA Diamond, using graduate-level science questions that are difficult to answer through web searches
  • MMLU-Pro, testing broader knowledge and reasoning with harder multiple-choice questions

AWS keeps the scores separate rather than merging them into a composite. Teams can prioritize the test that matters most for their application. Higher scores are better on all three benchmarks.

On the operations side, AWS sends a standardized workload to endpoints using the same JumpStart deployment configurations available to customers. The board reports time to first token, throughput per request in tokens per second, and estimated instance cost per hour.

Two controls shape those figures. A use-case profile defaults to Max Context, and an optimization goal defaults to Balanced. Changing the profile alters performance and cost results while leaving quality scores unchanged. Balanced picks the configuration with the lowest estimated cost per token, accounting for both hourly price and generation speed. The other goals select the lowest hourly instance price, lowest time to first token or highest throughput; some throughput configurations use speculative decoding.

That makes the ranking specific to a deployment configuration rather than a universal ordering of model quality.

How does selection carry into deployment?

Teams can first exclude models that fail requirements such as license, provider or input and output modality, then sort the remaining entries by a chosen benchmark. Blue heatmaps highlight stronger results within columns. Adjustable columns and frozen model names support wider comparisons.

Selecting a model opens deployment options and, where serverless customization is supported, customization controls. The SageMaker JumpStart documentation notes that customization, fine-tuning and evaluation are not available for every model.

AWS advises deploying with the same use-case profile and optimization goal used during comparison. A different configuration can change quality, latency, throughput and cost.

The hourly estimate also has limits. It covers only the SageMaker instance at the on-demand rate for the configuration's default instance type. Data transfer, storage, inference-component charges, Reserved Instance discounts and AWS Savings Plans sit outside the figure, so the leaderboard's price column does not represent the full bill.

What are the benchmark's limits?

AWS recommends using the board to build a shortlist, then testing the top two or three candidates on an organization's own data and prompts before production. Scores should be compared within each benchmark, and close results treated as ties, because small differences can reflect run-to-run variation.

Reasoning and non-reasoning models use different generation parameters, and GPQA Diamond results for reasoning models are averaged over 10 passes. Comparisons between those model types are approximate.

A missing score does not indicate poor quality. Some models simply have not been benchmarked, and the launch benchmarks focus on LLMs.

AWS says it is exploring leaderboards for curated private hubs, additional task boards for object detection, text embeddings and image understanding, and more models and benchmarks.

For JumpStart users, model selection can now begin with deployment-specific evidence inside the service where the model will run. The remaining decision is narrower but still consequential: whether a model's benchmark advantage holds on an organization's own workload, and whether any gain justifies its measured response behavior and instance cost.

Source: unite.ai

Share this article:

More from Amara Osei

Amara Osei

Show full bio

Senior reporter covering companies, deals and strategy in entrepreneurship, startups and venture funding at Business Bearings.

2,336 articles

Related articles

« Previous articleNext article »