Our team is at the ready to create a customized plan for you to optimize and scale your business. Contact us today to set up a call.

Curated content and latest news for MangoBoost solutions

MangoBoost, a provider of cutting-edge system solutions designed to maximize AI data center efficiency, has set a new industry benchmark with its latest MLPerf Training v5.0 submission. The company’s Mango LLMBoost™ AI Enterprise MLOps Software together with Mango GPUBoost™ RoCEv2 RDMA solution has demonstrated unparalleled performance on AMD Instinct™ MI300X GPUs, delivering the first-ever MLPerf multi-node results on AMD GPUs.

RoCEv2 RDMA solution on AMD Instinct™ MI300X servers.

First-ever multi-node MLPerf submission using AMD Instinct™ GPUs

, achieving the fastest Llama2-70B-LoRA fine-tuning on AMD hardware at just 

Customizable RoCEv2-based DPU-powered communication stack

 ensures seamless multi-node scalability—purpose-built for high-throughput AI and cloud infrastructure.

 (95–100% efficiency) on additional models like Llama2-7B and Llama3.1-8B, validating real-world scalability beyond benchmark tuning.

Model-agnostic, hardware-flexible, and deployment-ready

 across public cloud or on-premise—MangoBoost enables enterprise-grade training anywhere.

End-to-End Software-Hardware Stack for Performance at Scale

At the heart of this achievement is MangoBoost’s integrated platform, illustrated in Figure 1:

, featuring patent-pending technology for auto-tuning, model parallelism, batch scheduling, and memory management.

, enabling customizable, low-latency, high-throughput communication across nodes using RDMA over converged Ethernet.

First-Ever Multi-Node MLPerf Submission on AMD GPUs

Llama2-70B-LoRA fine-tuning across four AMD Instinct™ MI300X nodes (32 GPUs)

. This breakthrough enabled us to achieve:

The fastest Llama2-70B-LoRA fine-tuning time on AMD GPUs

 using MangoBoost’s end-to-end AI stack

 that demonstrate the maturity of AMD MI300X for real-world AI workloads

first-ever multi-node MLPerf Training result on AMD Instinct™ GPUs

high-performance large-scale AI training is now possible beyond vendor-locked platforms

, our AMD MI300X runs—spanning 8, 16, and 32 GPUs—exhibit 

scaling behavior comparable to prior NVIDIA H100 submissions in MLPerf Training v4.1

MI300X can deliver competitive multi-node performance

 backed by our software-hardware co-design.

This milestone was also recognized by industry leaders:

"

I'm excited to see MangoBoost's first MLPerf Training results, pairing their LLMBoost AI Enterprise MLOps software with their RoCEv2-based GPUBoost DPU hardware to unlock the full power of AMD GPUs, demonstrated by their scalable performance from a single-node MI300X to 2- and 4-node MI300X results on Llama2-70B LoRA. Their results underscore that a well-optimized software stack is critical to fully harness the capabilities of modern AI accelerators.

We congratulate Mangoboost on their MLPerf 5.0 training results on AMD GPUs and are excited to continue our collaboration with them to unleash the full power of AMD GPUs. In this MLperf Training submission, MangoBoost has achieved a key milestone in demonstrating training results on AMD GPUs across 4 nodes (32 GPUs). This showcases how the AMD Instinct™ MI300X GPUs and ROCm™ software stack synergize with MangoBoost's LLMBoost™ AI Enterprise software and GPUBoost™ RoCEv2 NIC.

, Fellow, AI Performance Design Engineering, AMD

As our first multi-node MLPerf Training submission on AMD Instinct™ MI300X, this result already demonstrates strong performance and scalability. With continued tuning and deeper hardware-software co-optimization, we expect even stronger results in future rounds.

Demonstrating Near-Linear Scaling on Llama2-7B and Llama3.1-8B

MangoBoost has benchmarked additional large language models—including 

 of its LLMBoost™ platform. Unlike our MLPerf Training v5.0 submission, these results are from 

, as MLPerf currently does not support these two models.

As shown in Figure 3, both models exhibit 

 as GPU count increases, confirming that our 

MLOps software and RDMA communication stack

 can efficiently orchestrate training across a variety of cluster sizes and workloads.

architecture-aware scheduling and optimized RDMA communication

, adapting well to diverse real-world training scenarios.

MangoBoost’s LLMBoost™ platform is production-ready, scalable, and optimized for modern AI training workloads—regardless of model size or structure.

Mango GPUBoost™ RDMA: Unlocking Multi-Node Training at Scale

Multi-node training isn’t just about adding more GPUs—it's about ensuring those GPUs communicate 

 that outpace standard RNICs in the market.

Up to 3.2Tbps RDMA (RoCEv2) throughput in total (up to 400Gbps per NIC)

Peer-to-peer communication between RNIC and GPU (i.e., ROCmRDMA support)

Customized congestion control and load balancing

Workload-specific optimization hooks using RoCEv2 extended headers

High connection scalability: Maintains line-rate throughput at thousands of QPs—3.4× higher than standard RNICs under heavy load

MangoBoost’s RDMA stack ensures that your training isn’t bottlenecked by the network—one of the biggest pain points in distributed deep learning today.

Collaboration with AMD: Unlocking the Full Potential of MI300X GPUs

MangoBoost’s ground-breaking MLPerf performance was made possible through deep collaboration with our partner 

 with industry-leading compute density and massive memory bandwidth.

Together, AMD’s ROCm platform and MangoBoost’s LLMBoost stack deliver an 

AI training and finetuning solution that is fast, scalable, and easy to deploy

—whether on a single node or across a multi-node cluster. MangoBoost’s GPUBoost RoCEv2 RDMA solutions further pushes the performance and scalability of the GPUs, and allows training across multiple MI300X nodes.

Train Anywhere with Mango LLMBoost™: Model-Agnostic, Hardware-Flexible, and Deployment-Ready

Modern AI infrastructure demands flexibility—not lock-in. MangoBoost’s LLMBoost AI Enterprise MLOps Software is built to support 

enterprise-grade LLM training wherever you need it

, whether that’s on public cloud, private datacenters, or hybrid environments.

MangoBoost supports a broad range of open and custom LLM architectures out of the box:

Whether you're training from scratch or fine-tuning with adapters, MangoBoost’s stack adapts to your model with minimal configuration.

Our full-stack platform is optimized for multiple GPU backends, including:

 — deeply optimized with ROCm and RDMA acceleration

 — compatible via standard CUDA/NCCL support

Future support for other AI accelerators via our abstraction layer

 and can make hardware choices based on price, performance, or availability.

☁️ Cloud & On-Premise Deployment Ready

MangoBoost offers seamless deployment modes:

: Preconfigured Docker images for AWS, Azure, and GCP

: Hardware-aware deployment scripts and RDMA stack support

: Easily integrated into your existing scheduling and orchestration frameworks

—any model, any hardware, any environment.

MangoBoost AI Infrastructure Hardware Solutions based on DPUs

Beyond LLMBoost software and GPUBoost RDMA solutions, MangoBoost offers hardware acceleration solutions based on Data Processing Unit (DPUs) for AI and cloud infrastructure, such as:

 – Offloads TCP/IP stack to free up CPU resources

 – High-performance NVMe initiator/target stack for scalable AI storage and JBoF solutions

This is just the beginning. Our R&D team is already implementing next-generation communication optimizations, hybrid parallelism strategies, topology-aware scheduling, and application-specific hardware acceleration to push multi-node performance even further.

If your organization is building large-scale AI models or exploring alternative GPU architectures, MangoBoost is your partner in unlocking full-stack efficiency and scalability.

To learn more or request a demo, contact us at 

Let us help you deploy high-performance, cost-efficient, and vendor-flexible AI infrastructure—powered by MangoBoost’s advanced MLOps software and communication stack.




The performance claims in this document are based on the internal cluster environment. Actual performance may vary depending on the server configuration. Software and workloads used in performance tests may have been optimized for performance only on MangoBoost products. Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. Results that are based on pre-production systems and components as well as results that have been estimated or simulated using MangoBoost reference platform for informational purposes only. Results may vary based on future changes to any systems, components, specifications, or configurations. Statements in this document that refer to future plans or expectations are forward-looking statements. These statements are based on current expectations and involve many risks and uncertainties that could cause actual results to differ materially from those expressed or implied in such statements. MangoBoost does not guarantee any specific outcome. Nothing contained herein is, or shall be relied upon as, a promise or representation or warranty as to future performance of MangoBoost or any MangoBoost product. The information contained herein shall not be deemed to expand in any way the scope or effect of any representations or warranties contained in the definitive agreement for MangoBoost products.

The information contained herein may not be reproduced in whole or in part without prior written consent of MangoBoost. The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. MangoBoost assumes no obligation to update or otherwise correct or revise this information and MangoBoost reserves the right to make changes to the content hereof from time to time without any notice. Nothing contained herein is intended by MangoBoost, nor should it be relied upon, as a promise or a representation as to the future.

MANGOBOOST MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION.

MangoBoost Sets a New MLPerf Training v5.0 Standard for Multi-nodes Llama2-70B-LoRA on AMD Instinct™ MI300X GPU!

 in its ground-breaking MLPerf Inference v5.0 submission on 

, beating all other MLPerf inference results to date

 vs. 32x H100 GPUs published MLPerf result

, hardware-flexible, deployable anywhere (cloud/on-prem)

🛠️Beyond LLMBoost software, MangoBoost also offers 

 is proud to announce a ground-breaking MLPerf inference V5.0 submission: we are the 

first-ever multi-node MLPerf submission using AMD Instinct™ GPUs, 

the highest result in Llama2-70B offline inference category in history, beating all other submitted results

, MangoBoost has not only showcased the scalability of 

AMD Instinct for large-scale AI workloads

robustness and high performance of our MLOps enterprise software - LLMBoost

🔧 LLMBoost: Easy to Use, High Performance, Scalable, and GPU-flexible

 is an easy-to-use AI Enterprise MLOps software that offers 

open 50+ models like LLaMA, Qwen, DeepSeek

, with no custom orchestration required. Summary of LLMBoost highlights are as follows:

or full control, security, and performance

 – Up to 138× faster than Ollama and 74× faster than HuggingFace

significantly more efficient than vLLM and HuggingFace TGI

LLMBoost integrates several patent-pending technologies that further enhance the performance and scalability, including:

 – Efficiently distributes large models across GPUs and nodes

– Optimizes runtime parameters based on workload characteristics

– Dynamically adapts LLM’s memory usage to maximize GPU utilization

– Flexible deployment across multiple inference stages

No vendor lock-in. No complex setups. Just performance, efficiency, and simplicity

🤝 MangoBoost MLPerf Inference Result: Powered by AMD MI300X, Supercharged by LLMBoost Software

MangoBoost record-breaking MLPerf performance was made possible through deep collaboration with 

AI inference solution that is fast, scalable, and easy to deploy

—whether on a single node or across a multi-node cluster.

🚀 LLMBoost on 32x MI300s Outperforms 32x H100s — At Lower Cost

LLMBoost delivers a 24% higher throughput

 than the previous best multi-node MLPerf result from Juniper Networks using 

 which reported 82,749 tokens per second (TPS) on Llama2-70B offline inference.

LLMBoost with AMD Instinct MI300X achieved a leap to 

surpassing the performance of a 32x H100 setup. 

With each MI300X GPU estimated price of around $15,000–$17,000, compared to $32,000–$40,000 per H100— LLMBoost on AMD Instinct MI300X 

offers up to 62% savings in GPU cost while delivering superior inference throughput.

the LLMBoost with MI300X system delivers approximately 2.8× more inference throughput per $1,000 spent than the H100-based system

, making it the clear choice for high-performance, budget-conscious deployments.

📈 Highest Llama2-70B Offline Inference Results in MLPerf v5.0

MangoBoost achieved the highest LLaMA2-70B offline inference performance

 in MLPerf v5.0 compared with all other submitters (e.g., Nvidia, Intel, Google). Compared to the previous 8-GPU MI300X submission,

 MangoBoost’s latest MLPerf results showcase 

from a single-GPU to a distributed multi-node, multi-GPU production cluster.

Whether you're running a handful of inference threads or deploying a high-throughput LLM service, 

LLMBoost ensures you get the maximum efficiency at any scale

Want to replicate our MLPerf v5.0 results? Try it yourself on 

] or follow the step-by-step instructions in our 

🔧 Additional Result Highlights on 8x Nvidia A100 GPUs on AWS

Beyond the specific MLPerf results mentioned above, LLMBoost has been extensively evaluated with many platforms and GPUs. As an example, in our evaluation using an 

8×NVIDIA A100 GPU server from AWS, LLMBoost delivers up to 138× faster inference compared to Ollama

outperforms both HuggingFace TGI and vLLM across all model sizes tested

-R1-Distill-Qwen-32B, and LLaMA3.1-8B. In terms of cost-efficiency, LLMBoost also leads the pack with 

Whether you're optimizing for performance, cost, or deployment simplicity, 

LLMBoost offers unmatched efficiency and scalability across a wide range of LLMs.

🧪 From Research to Production in Minutes

 bridges the gap between experimentation and deployment, allowing developers and data teams to serve powerful open-source models 

, not days. Whether you're testing a quantized 

a consistent, streamlined experience at every scale

Getting started is as simple as a running single command:

⚡MangoBoost AI Infrastructure Hardware Solutions based on DPUs

Beyond LLMBoost software, MangoBoost offers hardware acceleration solutions based on Data Processing Unit (DPUs) for AI and cloud infrastructure, such as:

 – RDMA acceleration for multi-node inference/training via RoCEv2

 – High-performance NVMe initiator/target stack for scalable AI storage

References
[1] Benchmark Suite Results for MLPerf Inference: Datacenter
•https://mlcommons.org/benchmarks/inference-datacenter/
[2] Tom's Hardware – H100 vs. MI300X Pricing
•https://www.tomshardware.com/tech-industry/artificial-intelligence/nvidias-h100-ai-gpus-cost-up-to-four-times-more-than-amds-competing-mi300x-amds-chips-cost-dollar10-to-dollar15k-apiece-nvidias-h100-has-peaked-beyond-dollar40000
[3] Mango LLMBoost for AMD MI300x on Azure Marketplace
•https://azuremarketplace.microsoft.com/en-us/marketplace/apps/mango_solution.mango_llmboost_amd?tab=Overview
[4] MangoBoost’sMLPerfGitHub README
•https://github.com/mlcommons/inference_results_v5.0/blob/main/closed/MangoBoost/README.md



MangoBoost Sets The Highest MLPerf Inference v5.0 Result in History for Llama2-70B Offline!

 unlocks the power of state-of-the-art GPUs for AI training systems with multi-GPUs. As an example, MangoBoost demonstrates linear scaling using our RDMA solution, for MLperf Training v4.1 Llama2-70B-LoRA workload (BF16), using 24x AMD MI300X GPUs across multiple nodes.

 to allow direct communication across GPUs to scale AI systems. Furthermore, MangoBoost HW-based 

in-line customizable congestion control and custom headers

 enable even more optimal solutions at scale.

Mango GPUBoost RDMA™ is ready to ship. Pre-orders are open for customers who would like to accelerate their AI infrastructure.

 and plans to offer an Ultra Ethernet solution in the near future.

Efficient GPU-to-GPU communication across multiple nodes is essential for large-scale AI training workloads, where dozens to thousands of GPUs work together to train complex models. 

 has emerged as a key technology for enabling this scalability, as it allows a direct data transfer between GPU memory on different nodes without the help of the CPU host significantly reducing the latency of GPU-to-GPU data transfers. Among various RDMA solutions,

has been shown to be a promising solution that offers unique advantages over traditional Infiniband for large-scale AI training environments[1].

One of RoCEv2’s primary advantages is its 

. Because it uses an IP header for routing, RoCEv2 can scale over larger, more complex network topologies than Infiniband. This IP-based routing capability makes RoCEv2 especially appealing in hyperscale AI infrastructures, where scalability is crucial for managing massive AI models and multi-node GPU configurations.

, which directly impacts the physical layout of the AI networking infrastructure. Ethernet switches typically support more ports than Infiniband switches of the same generation, reducing the level of switches that introduces latency and potential oversubscription. With RoCEv2’s use of high-port-count Ethernet switches, system architects can create

 streamlined, low-latency network designs

 that improve data flow across the entire GPU cluster.

a broad ecosystem, with multiple vendors supporting Ethernet-based solutions

. Unlike Infiniband, which is primarily produced by NVIDIA, RoCEv2 gives hyperscalers and enterprises the flexibility to choose from a variety of vendors, helping them to avoid vendor lock-in and maintain more control over their supply chain.

Figure 1 shows how MangoBoost set up a large-scale AI Training system using MangoBoost DPUs. Each server is equipped with eight MangoBoost DPUs and eight GPUs, linked via PCIe for GPU-NIC communication. In this system, high-bandwidth interconnects like AMD's Infinity Fabric or NVIDIA's NVLink manage intra-node communication. For inter-node communication, MangoBoost DPUs enable RoCEv2, connecting multiple nodes over Ethernet switches and creating a high-performance and scalable setup optimized for large-scale AI workloads.

Mango GPUBoost RDMA™ enables linear performance scale-up of MLPerf Training v4.1 Llama-70B-LoRA with AMD MI300X GPUs

Our Llama-70B LoRA training leveraged from 8 to 24 MI300X GPUs with MLPerf datasets and was conducted in full compliance with the MLPerf Training v4.1 standards using BF16 precision (code provided by MLPerf). The setup focused on 

maximizing GPU-to-GPU communication efficiency to boost throughput

. We used AMD’s Infinity Fabric for intra-node communication, while inter-node communication was powered by RoCEv2 RDMA, which is crucial for seamless scaling across multiple servers.

To ensure high-efficiency RoCEv2 performance, we installed MangoBoost DPUs, assigning one DPU per GPU in each server. This 

1:1 mapping enabled optimized data transfer across nodes

, taking advantage of RDMA’s capabilities to support 

. Each server used an identical configuration, providing a robust and scalable infrastructure capable of handling the intense demands of large language model (LLM) training.

 as we increased the number of servers, which is paramount in large-model training where hardware limitations and network overhead often constrain scalability. With our setup, we maintained nearly optimal performance even as additional resources were added.

we ensured strict adherence to the MLPerf v4.1 validation requirements

. This included following guidelines for hyperparameters, evaluation methodology, and quality targets, such as achieving a 0.925 evaluation loss threshold. We validated our setup against the Reference Convergence Point (RCP) rules, ensuring accuracy and reliability in producing high-quality training results for large language models.

Mango GPUBoost RDMA™ powers accelerated AI with industry-standard and customizable features

high-performance RoCEv2-based RNIC solution 

that unleashes the full potential of ethernet-based large-scale AI workloads. It provides greater network scalability alongside peer-to-peer communication between GPU and RNIC (e.g., GPUDirect, ROCmRDMA) to enable optimized connectivity among GPUs at scale.

 achieves impressive performance, sustaining 90% of line-rate throughput

 even as the number of QP (Queue Pair) connections increases. It also delivers 

compared to other standard RDMA network interface cards (RNICs) available on the market. Figure 3 demonstrates these advantages using results from Perftest, a widely-used RDMA benchmarking tool, where Mango GPUBoost RDMA™ achieves 3.4 times higher throughput and reduces 99.9% latency by 52%.

two important features (multipath and custom congestion control) 

to resolve well-known problems within RoCEv2.

First, due to the nature of the RoCEv2 protocol — which enforces packets to be delivered in the order they were sent — network hotspots often occur because of path collision [2]. To alleviate these hotspots and take advantage of multiple rich paths, Mango GPUBoost RDMA™ supports a 

unique out-of-order packet handling mechanism and provides multipath extension

Moreover, no single congestion control solution fits all use cases for RoCEv2. Although DCQCN has become the de-facto standard for RoCEv2, it is hard to configure in practice [3] and has been shown to be less effective for AI workloads [4]. Instead, each workload demands a tailored congestion control algorithm depending on its unique characteristics [5] (e.g., traffic pattern, incast, etc). To maximize workload performance, Mango GPUBoost RDMA™ offers a 

placeholder for custom congestion control

, allowing users to deploy their own algorithms directly on our RNIC.

Please check the product specification and more details 

Try our Mango GPUBoost RDMA™ - Pre-Order is now open!

We are ready for customers who want to innovate their AI infrastructure by accelerating network communication with Mango GPUBoost RDMA™. Our DPUs have been tested on various AI training and inference workloads with top-of-the-line GPUs, demonstrating its flexibility along with industry-leading performance. We support standard RoCEv2 with customizable features and compatibility with common server products.

We continue to innovate performance and scalability in AI infrastructure. As a member of the UltraEthernet Consortium, we also plan to showcase our UltraEthernet DPU in the near future.

References
[1] RDMA over Ethernet for Distributed Training at Meta Scale (Meta, SIGCOMM’24)
[2] Multi-Path Transport for RDMA in Datacenters (Microsoft, NSDI’18)
[3] HPCC: High Precision Congestion Control (Alibaba, SIGCOMM’19)
[4] Impact of RoCE Congestion Control Policies on Distributed Training of DNNs (Meta, HOTI’22)
[5] Datacenter Ethernet and RDMA: Issues at Hyperscale (Microsoft/Broadcom/HPE/Google, IEEE Computer’23)

Blog

Ready for
a boost?

Blog

MangoBoost Sets a New MLPerf Training v5.0 Standard for Multi-nodes Llama2-70B-LoRA on AMD Instinct™ MI300X GPU!

MangoBoost Sets The Highest MLPerf Inference v5.0 Result in History for Llama2-70B Offline!

MangoBoost Demonstrates Mango GPUBoost RDMA™ with MLPerf Training v4.1 on a Multi-Node System Powered by AMD MI300X GPUs

MangoBoost Demonstrates Turn-Key Storage Solution for AI Training with MLPerf® Storage v1.0

MangoBoost Demonstrates a Turn-Key LLM Inference Serving Solution on AMD Instinct™ MI300X GPUs with the MLPerf® Inference v4.1 Llama Benchmark

Ready for a boost?

Ready for
a boost?