The First AMD Instinct™ Submission to MLPerf® Inference with Prefill/Decode-Disaggregated Deployment, and the Largest Multi-Region Cluster Ever Submitted

KEY TAKEAWAYS FOR MLPERF® INFERENCE v6.1

➔ First prefill/decode disaggregated submission on AMD Instinct GPUs, in two configurations.

• Single node: Four prefill GPUs and four decode GPUs within one node containing 8 x AMD Instinct MI355X GPUs, submitted in both air-cooled and liquid-cooled configurations. On GPT-OSS 120B Interactive, our liquid-cooled node delivers 14% more throughput than NVIDIA's B300, and 2.25x the NVIDIA B200 result. Those comparisons are all between single 8 GPU nodes, against the best published B300 and B200 results in MLPerf® Inference [1].
• Multi-node: One prefill node + one decode node, totaling 16 x AMD Instinct MI355X GPUs, with the KV cache crossing the inter-node fabric. Normalized per node, the two-node deployment matches our single-node PD result: crossing the node boundary costs nothing. That unlocks independent scaling of the two phases. Add prefill nodes for long prompts, add decode nodes for long outputs, and tune the ratio to the workload instead of to what fits in one 8 GPU node.

➔ Largest multi-region cluster ever submitted to MLPerf® Inference. We combine 16 x AMD Instinct MI300X GPUs and 16 x AMD Instinct MI355X GPUs, 32 GPUs in total from four providers (MangoBoost, Dell, TensorWave, and Microsoft Azure). They run as four nodes at four sites on two continents, one node under each provider. All 32 GPUs serve as one endpoint, at up to 97% scaling efficiency.

➔ The only GPT-OSS 120B results on AMD Instinct MI300X GPUs in MLPerf® Inference. Against the best published result on 8 x NVIDIA H200 GPUs [1], our single node with 8 x AMD Instinct MI300X GPUs delivers 5% more throughput at 36% lower cost per token.

Pushing the Boundaries of LLM Inference

MangoBoost has submitted five GPT-OSS 120B entries to MLPerf® Inference v6.1, all served by LLMBoost on AMD Instinct GPUs, all in the Closed division and the Available category. This round includes three results with no precedent in the benchmark: the first submission of prefill and decode (PD) disaggregated results on AMD Instinct GPUs, the largest multi-region cluster ever submitted to MLPerf® Inference, and the only GPT-OSS 120B results on AMD Instinct MI300X GPUs.

Real inference fleets are not shaped the way a benchmark assumes. Capacity arrives in pieces and rarely in one generation of hardware, and the workloads that matter commercially are interactive, so the binding constraint is time per output token rather than aggregate throughput. We built this round around both conditions: PD disaggregation to hold latency under load, and a four-region pool to serve fragmented capacity as one endpoint.

1. Breaking the Prefill and Decode Compromise: PD Disaggregation on AMD Instinct

Every request runs in two phases, prefill then decode. Prefill processes the whole prompt in one pass, so it saturates the GPU's compute units. Decode emits one token at a time, so it is limited by how fast weights and KV cache can be read from memory. Both phases compete for the same GPUs, and that forces a compromise: long prefills stall the decode queue, and time per output token suffers.

PD disaggregation separates the two phases onto different GPUs and transfers the KV cache between them. The benefit shows up in time per output token rather than in raw throughput, so we applied it only in the Interactive scenario, where latency is the binding constraint. LLMBoost handles the routing, the KV cache transfer, and the ratio of prefill workers to decode workers.

Interactive results are reported in tokens per second, but they are not plain throughput numbers. The Interactive scenario fixes a time to first token (TTFT) bound and a time per output token (TPOT) bound, and a run is valid only if it meets both. Interactive throughput is therefore the load a system can carry while holding those latency limits.

Two Topologies: Across Nodes, and Inside One Node

We deployed PD disaggregation in two configurations to measure what crossing a node boundary costs. Neither configuration has been submitted on AMD Instinct GPU before.

• Multi-node PD. Two nodes and 16 x AMD Instinct MI355X GPUs. One node serves prefill, one node serves decode, and the KV cache crosses the inter-node fabric
• Single-node PD. One node and 8 x AMD Instinct MI355X GPUs. Four GPUs serve prefill, four serve decode, and the KV cache stays inside the node.

The two topologies perform comparably per node. In Interactive, multi-node PD reached 59,297 tok/s across two nodes, which is 29,648 tok/s per node. Single-node PD reached 29,586 tok/s on a single liquid-cooled node. The two node entry is 0.2% ahead per node, so under the Interactive latency bound, crossing the node boundary costs nothing. That makes PD disaggregation flexible to deploy: run it inside a single node that already exists, or spread it across nodes, where the prefill-to-decode ratio is no longer limited by the eight GPUs allow.

Interactive Throughput per Node Against Current NVIDIA Silicon

Normalized per node, LLMBoost on MI355X reaches 29,586 tok/s in Interactive on a single liquid-cooled node, and 29,648 tok/s per node across the two node entry. The best NVIDIA results published in MLPerf® Inference v6.0 Closed are 26,006 tok/s per node on B300, submitted by ASUSTeK, and 13,155 tok/s on B200, submitted by Nebius [1]. Node against node, our single node AMD MI355X result is 14% ahead of NVIDIA B300 and 2.25x the B200 result.

The comparison is like for like on configuration, with one difference to state plainly: the round. Our rule is to compare against the best published result for each part, whichever round it comes from. For B300 and B200 in Interactive, that is v6.0: no B300 or B200 Interactive result was published in v6.1, so the v6.0 entries remain the best on record and are the ones quoted here. Both sides are 8 GPU nodes running GPT-OSS 120B in the Interactive scenario, in the Closed division and the Available category, under the same MLCommons latency bounds and the same accuracy target. Closed division rules fix the model and constrain the quantization, so the difference is in the serving stack and the hardware rather than in the workload.

In Offline and Server, where our runs use no PD disaggregation and prefill/decode share the same GPUs, MI355X and B300 land close together: per node in Offline, our entries reach 116,953 tok/s against 114,569 for the best B300 result (submitted by HPE in v6.1), and 93,071 for the best B200 (submitted by Red Hat in v6.0). In Server we reach 110,153 tok/s against 113,685 for B300 (HPE, v6.1) and 87,444 for B200 (Nebius, v6.0) [1]. That is 2% ahead of B300 in Offline and 3% behind it in Server.

2. One Endpoint Across Four Regions: The Largest Multi-Region Cluster Ever Submitted

Most MLPerf® submissions run inside one datacenter. Real inference fleets do not. Capacity arrives in pieces, from a colocation site, from a cloud region, and from a specialist GPU provider, and it rarely arrives in one place. The practical question is whether that fragmented capacity can serve as a single endpoint without losing throughput.

We integrated four clusters, one node each, into a single LLMBoost serving pool. The pool spans four providers, two GPU generations and two continents, 32 GPUs in total.

• MangoBoost AI DC, South Korea. 8 x MI300X.
• Microsoft Azure, United States. 8 x MI300X.
• Tensorwave, United States. 8 x MI355X.
• Dell, United States. 8 x MI355X.

These entries are not PD-disaggregated. Prefill and decode run colocated on every node in the pool, which is why the cluster was submitted in Offline and Server rather than in Interactive.

LLMBoost’s scheduler places work in proportion to what each node can absorb, so mixing fast and slow nodes costs almost nothing. Offline held 97% of the standalone sum, and the shortfall traces to the trans-Pacific hop rather than to any imbalance in how work was placed.

Scaling Efficiency Against the Sum of the Parts

The right reference point is the sum of the four nodes measured standalone. Anything close to that sum means the cluster layer is not taking a cut.

Offline reached 285,454 tok/s at 97% scaling efficiency. The remaining 3% comes from the round trip between South Korea and the United States, which is a physical cost rather than a scheduling cost. Server reached 253,501 tok/s at 94% scaling efficiency. Server scales slightly lower than Offline because the latency constraint is stricter, so the scheduler has less freedom to hide the same network latency behind queued work.

Operators who hold capacity in a particular country for data residency or sovereignty reasons have had to treat each region as its own deployment, with its own utilization and its own headroom. This result says the split itself is close to free: four sites on two continents behaved as one pool at 97% of what the same nodes deliver standalone. For a GPU operator or a neocloud, that is the difference between selling capacity site by site and selling a single endpoint.

3. The Only GPT-OSS 120B Results on AMD Instinct MI300X GPUs

GPT-OSS 120B was added to MLPerf® Inference in v6.0 and had no published results on AMD Instinct MI300X GPUs before this round. We built the quantized model and the optimized serving recipe for it, and submitted it on a single node of 8 x AMD Instinct MI300X GPUs.

Our single node result is 30,198 tok/s in Offline and 24,863 tok/s in Server. NVIDIA’s H200 result reaches 28,680 tok/s in Offline and 24,103 in Server on 8 GPUs [1][3], so our result lands 5% ahead in Offline and 3% ahead in Server.

For operators who already run MI300X, that is the practical result. GPT-OSS 120B had no published results on this part before this round, and these numbers put an installed Instinct fleet at H200-class throughput on a current open model, with no incremental acquisition cost. Set against a refresh cycle to current-generation accelerators, the comparison is not a lower cost per token but zero capex against a purchase order.

4. And All of It Costs Less

Throughput is half of an infrastructure decision. The other half is what the hardware costs to buy or rent, and on that axis the AMD parts in this round rent for less than the NVIDIA parts they are compared against, with one exception at near parity.

Reserved cloud rates are the most directly observable prices available, so we use those. For every part, ours and NVIDIA’s alike, we take the midpoint of a provider’s published reserved range [5][6]. MI355X rents for about $3.64 per GPU-hour, 32% less than the $5.38 for B300. MI300X rents for about $1.89 against $2.79 for H200. B200 is the one near-parity case, at $3.57 against MI355X's $3.64 [5]. A second provider publishes floors rather than ranges, H200 from $4.38 and B300 from $7.38 [4], and both floors sit above the midpoints used here, so those rates would widen every margin in the table.

In the bar chart above, B300 delivers 3% more throughput than MI355X, yet MI355X still returns 1.43x the throughput per dollar, because the price gap is far larger than the performance gap. Where we also lead on throughput, in Interactive, the two effects compound to 1.68x. The same arithmetic in cost per token: MI300X against H200 in Offline works out to $0.139 per million tokens against $0.216 at these rates, which is 36% lower cost per token for 1.55x the throughput per dollar.

Purchase Prices Point the Same Way

For customers who buy rather than rent, the ratio holds. Published public prices put MI300X at an estimated $24,000 to $30,000 per GPU against $38,000 to $42,000 for H200 [5], which works out to 1.56x the throughput per dollar of GPU spend, almost exactly what the rental rates give. A second source prices MI300X lower still and would put the figure at 2.5x [2]. We quote the conservative number.

A Record-Setting Collaboration with AMD

Every entry in this round ran on AMD Instinct GPUs. The PD-disaggregated results ran on MI355X, the multi-region pool combined MI355X and MI300X across four operators, and the MI300X entries extended GPT-OSS 120B to a generation of AMD Instinct hardware that had no published results for this model. Close collaboration with AMD on the software stack is what made all three possible.

“Across our recent MLPerf® Inference collaborations with MangoBoost, each round has delivered a new milestone for AMD Instinct™ GPUs. In v5.1, we introduced the first heterogeneous multi-node submission on AMD Instinct GPUs. In v6.0, that expanded to the first multi-region, three-GPU submission. And now, in MLPerf® Inference v6.1, we are introducing the first submissions on AMD Instinct GPUs to demonstrate both intra-node and inter-node prefill–decode disaggregation. Applying prefill–decode disaggregation to the Interactive scenario is especially meaningful for enterprise deployments, where latency is a critical constraint alongside throughput. Our multi-region work has also expanded from systems spanning two continents in v6.0 to four independent clusters operated by four providers, all orchestrated as a single serving endpoint and achieving 97% scale-out efficiency. MangoBoost also used LLMBoost™ to submit the first MLPerf® Inference results for gpt-oss-120b on AMD Instinct™ MI300X GPUs. MangoBoost continues to be a consistent and technically ambitious collaborator. Built on AMD ROCm™ software, LLMBoost helps make these capabilities available to enterprises through a streamlined, enterprise-ready inference platform. ”

“Meena Arunachalam Fellow and Director of AI Workloads Performance Engineering, AMD ”

Working with Dell Technologies

Three of the five entries were submitted jointly with Dell Technologies: the multi-region cluster and both single node MI355X results. Dell supplied PowerEdge XE9785 air-cooled and XE9785L liquid-cooled node configurations, which is why an otherwise identical node appears twice in the submission table, once in each cooling configuration

Liquid Cooling Pays Most Under a Latency Bound

The two MI355X entries are a controlled comparison. Same node, same GPUs, same LLMBoost configuration, same model. The only variable is how the heat leaves the box. Results like this are rare in MLPerf®, because submitters do not usually run the same system twice.

Liquid wins in all three scenarios, but not by the same margin. Offline improves by 1.4% and Server by 1.0%. Interactive improves by 4.7%, more than three times as much.

The asymmetry is the interesting part. Offline and Server are throughput bound, and a node that thermally throttles simply finishes the queue a little later. Interactive is latency bound: every request has to meet a time per output token ceiling, and a clock that sags mid-generation pushes requests past that ceiling rather than just slowing them down. Thermal headroom converts into sustained clocks, and sustained clocks are what hold a latency bound at load. That is why the same cooling change is worth three times more in Interactive, the scenario that matches how customers serve live traffic.

Worth noting for anyone running air-cooled: the air-cooled node still reaches 28,266 tok/s in Interactive, which is 8.7% ahead of the best published v6.0 B300 Interactive result. The lead over current NVIDIA silicon does not depend on liquid cooling. Liquid extends it.

Full Submission Breakdown

All five entries ran GPT-OSS 120B in the Closed division and the Available category. Throughput is reported in tokens per second. PD disaggregation applies to the Interactive scenario only: the two node entry uses inter-node PD and the single node MI355X entries use intra-node PD. Every Offline and Server entry ran prefill and decode colocated. A dash means we did not submit that entry in that scenario. The full configuration for every entry, including quantization, parallelism layout and the prefill-to-decode worker ratios, is published with our submission code alongside the round.

MangoBoost AI Infrastructure

LLMBoost is the inference serving platform behind every result in this post: model deployment, scheduling, PD disaggregation and multi-region pooling in one package

Alongside LLMBoost™, MangoBoost accelerates infrastructure from the ground up with advanced DPU-based hardware acceleration:

• Mango BoostX™: A powerful FPGA-based DPU designed for offloading networking, storage, and security workloads.
• Mango BoostX™ RNIC: A RoCEv2 NIC for ultra-fast, intra-rack GPU-to-GPU connectivity.
• Mango Kesar™ Storage Server: A high-density, disaggregated storage platform built for high-bandwidth, low-latency NVMe access.
• Mango Alphonso™ GPU Server: A full-stack system purpose-built to maximize AI performance and optimize tokens/$.

Try LLMBoost™ Today

To see our record-setting MLPerf® v6.1 performance, register on our virtual demo page.

References

• [1] MLCommons. "Benchmark MLPerf Inference: Datacenter." https://mlcommons.org/benchmarks/inference-datacenter/ Round v6.0, Closed division, available systems, gpt-oss-120b; data as of 3 April 2026. B200-SXM-180GB, 8 GPUs: 85,921 tok/s Offline, 87,444 Server, 13,155 Interactive (Nebius, 6.0-0079); 93,071 Offline, 71,588 Server (Red Hat, 6.0-0095). B300-SXM-270GB, 8 GPUs: 111,496 Offline, 110,655 Server (Cisco, 6.0-0012); 26,006 Interactive (ASUSTeK, 6.0-0006, the only B300 entry with an Interactive result). H200-SXM-141GB, 8 GPUs: 28,680 Offline, 24,103 Server (Red Hat, 6.0-0094), no Interactive entry.
• [2] Spheron. "AMD MI300X vs NVIDIA H200: 192GB vs 141GB, Which Wins?." https://www.spheron.network/blog/amd-mi300x-vs-nvidia-h200/ 6 February 2026. MI300X at approximately $10,000 to $15,000 per GPU, H200 SXM at roughly $25,000 to $35,000 per GPU. Market estimates, not vendor list prices.
• [3] Red Hat. "Red Hat AI tops MLPerf Inference v6.0 with vLLM on Qwen3-VL, Whisper, and GPT-OSS-120B." https://www.redhat.com/en/blog/red-hat-ai-tops-mlperf-inference-v60-vllm-qwen3-vl-whisper-and-gpt-oss-120b 1 April 2026. Reports 8 x H200 on GPT-OSS 120B at 28,680 tok/s Offline and 24,103 Server, and states it was the only H200 submission for this model in that round.
• [4] VESSL AI. "NVIDIA B300 GPU Cloud." https://vessl.ai/en/gpu/b300 2026; compiled from VESSL Cloud published rates. Cloud rates per GPU-hour: A100 from $1.48, H100 from $2.98, H200 from $4.38, B200 from $5.88, B300 from $7.38 (Reserved only). Cited as the higher-priced provider: its H200 and B300 floors both sit above the midpoints used in Table 2.
• [5] gpu.fm. "GPU Buying Guide 2026: H100, H200, B200, GB300, MI355X." https://www.gpu.fm/buying-guide 1 April 2026, updated 25 April 2026; compiled from vendor datasheets, hyperscaler list pricing and published cloud rate cards. Street: MI300X $24,000 to $30,000, H200 SXM $38,000 to $42,000. Reserved cloud rates per GPU-hour: MI300X $1.49 to $2.29, H200 $2.29 to $3.29, MI355X $2.99 to $4.29, B200 $2.65 to $4.49.
• [6] Bentaus. "NVIDIA B300 GPU Cloud." https://www.bentaus.com/nvidia-b300 Accessed 22 August 2026. Reserved: $4.25 to $6.50 per GPU-hour. On demand: $6.50 to $8.50 per GPU-hour. Source of the $5.38 B300 reserved midpoint used in Table 2 and Figure 6.

The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information. MangoBoost, LLMBoost, Mango BoostX, Mango Kesar, and Mango Alphonso are trademarks of MangoBoost Inc. AMD, the AMD Arrow logo, Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. NVIDIA and the NVIDIA logo are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Dell Technologies, Dell, and PowerEdge are trademarks of Dell Inc. or its subsidiaries. Microsoft and Azure are trademarks of the Microsoft group of companies. Other product and company names mentioned are trademarks of their respective owners and are used for identification purposes only.