Our team is at the ready to create a customized plan for you to optimize and scale your business. Contact us today to set up a call.

Curated content and latest news for MangoBoost solutions

 Highest MLPerf inference performance for 

 Demonstrated near linear performance scaling across multi-architecture clusters combining AMD 

 First and only third-party submission showcasing AMD Instinct™ 

 across diverse hardware platforms from various server vendors.

breakthrough results across diverse workloads

Interested in trying our LLMBoost software

unlocks the full potential of heterogeneous and multi-node GPU clusters

performance, scalability, and cost efficiency

The scale and innovation of MangoBoost’s submission have been recognized across the industry:

David Kanter – Founder & Head of MLPerf, MLCommons: 

“I am thrilled to see MangoBoost’s creativity in pushing the limits of LLM serving with the first-ever heterogeneous results for MLPerf Inference. This submission integrates multiple generations of AMD GPUs with MangoBoost's LLMBoost platform to deliver impressive performance for Llama2-70B at 160K tokens/s, underscoring the power of software and system architecture in inference serving.”

“We are thrilled with our MLPerf Inference v5.1 co-submission with MangoBoost. Together, we set a new performance record for Llama2-70B using AMD Instinct™ MI355X GPUs, scaling to 4-node and 8-node configurations. MangoBoost also delivered the first heterogeneous multi-node submission showcasing the combined power of AMD Instinct MI300X and MI325X GPUs, along with strong multi-node results on Dell and Supermicro servers. With nine high-performance MLPerf inference submissions, MangoBoost’s LLMBoost GenAI, powered by AMD ROCm™, provides seamless scalability and easy deployment for enterprise AI workloads.”

Frank Han – Distinguished Member of Technical Staff, Dell: 

“Dell's collaboration with MangoBoost is built on a foundation that began with MLPerf Training v5.0. In our official MLPerf Inference v5.1 co-submission (Dell_MangoBoost), MangoBoost’s LLMBoost software demonstrated impressive linear scaling across multi-node Dell PowerEdge server configurations with AMD accelerators.”


LLMBoost™: GenAI Software for Any GPU, Any Scale

turn-key, high-performance inference stack

 with broad compatibility and advanced optimizations:

 Launch models instantly using Docker with 

 for maximum control, security, and performance.

99.5% lower cost per million tokens vs. Ollama

 for efficient distributed inference on homogeneous or heterogeneous GPU clusters.

 across the AI stack for real-time visibility into GPU utilization, memory usage, latency, throughput, and queue states.Exports metrics in standard formats for 

Full-Stack Auto-Tuning (Patent Pending) : 

Dynamically adjusts runtime configurations and model deployment strategies based on 

model architecture, hardware topology, and live traffic patterns

dynamic scaling, load balancing, straggler mitigation, and automatic failover

 to ensure high availability and consistent latency during heavy workloads.

Deep Collaborations with Industry Leaders

 is built on deep, strategic collaborations with industry leaders such as 

(listed alphabetically), alongside validation on Gigabyte platforms. These partnerships ensure 

 across a wide range of hardware ecosystems to 

 requires more than just software innovation — it depends on 

 and validation across diverse platforms. By working directly with leading technology partners, MangoBoost ensures that 

 is not only optimized for the latest AMD Instinct™ GPUs, but also proven on a variety of server architectures from major vendors. These collaborations demonstrate that enterprises can 

trust LLMBoost™ to deliver consistent, high-performance results

 on their infrastructure of choice, whether in 

single-node systems, multi-node clusters, or heterogeneous GPU deployments

Through co-engineering efforts with AMD, MangoBoost gained early access to next-generation AMD Instinct™ GPUs such as the MI355X, enabling us to 

tightly integrate LLMBoost™ with the ROCm™ software stack

, achieving 648K tok/s on 64 GPUs and delivering 

record-setting performance in the open division with 8-node MI355X clusters

Read more on AMD’s perspective in their 

In collaboration with Dell Technologies, MangoBoost validated 

, demonstrating reliable performance across enterprise-scale deployments. This collaboration included validation on 

multi-node MI300X clusters with near-linear scalability

LLMBoost™’s enterprise-ready performance

MangoBoost Collaboration with Supermicro:


Working with Supermicro, MangoBoost validated 

 across a wide range of single-node, multi-node, and heterogeneous GPU deployments. This included the 

MI300X nodes (Supermicro AS-8125GS-TNMR2)

MI325X nodes (Supermicro AS-8126GS-TNMR) 

LLMBoost™ can seamlessly orchestrate workloads across different GPU generations

 while maintaining consistent performance and efficiency. 

 achieved by MangoBoost and our partners across 

, demonstrating LLMBoost™’s scalability across diverse hardware platforms.

 Consistent near-linear performance gains were observed as the number of nodes increased, even in heterogeneous setups combining 

, validating LLMBoost™’s optimizations with AMD’s latest GPUs.

servers (listed alphabetically), ensuring consistent performance across hardware ecosystems.

Below are some of the key highlights from this round:

As a result of deep collaboration with AMD and full integration of the optimized ROCm software stack, our submission delivered 

, outperforming the next-best NVIDIA result by 

 higher performance on average across all submissions by primary metric.

In the open division, our joint submission with AMD scaled an 

 of LLMBoost™ on the latest GPU architectures.

2. First-Ever Heterogeneous GPU Deployment

For the first time in MLPerf history, MangoBoost submitted results using 

, proving LLMBoost™’s ability to efficiently orchestrate workloads across multiple GPU generations.

, allowing them to integrate newer GPUs into their infrastructure 

intelligent load balancing, optimized communication libraries, and full-stack auto-config tuning

 across a wide range of workloads — from 

highly complex vision-text (multi-modal) applications

. Its ability to scale across different models and workloads ensures 

enterprise-grade reliability and efficiency

1. Text-Only Models: Superior Throughput and Cost Efficiency

, LLMBoost™ consistently outperforms competing solutions like 

LLMBoost™ achieves up to 186× faster performance than Ollama and 4× faster than vLLM

, setting a new benchmark for throughput and cost efficiency.

This unmatched performance translates into 

scale inference workloads at a fraction of the cost

2. High-Performance Vision-Text Multi-Modal Deployments

 remain minimal, even as the number of images per prompt increases from 

LLMBoost™ maintains stable throughput and low latency

LLMBoost™ as the preferred choice for vision-text workloads

high-performance, scalable, and cost-efficient solution

Getting started with LLMBoost™ is as simple as selecting the model you want on HuggingFace and then running one command.

Beyond Software: MangoBoost DPU Solutions

Alongside LLMBoost™, MangoBoost accelerates infrastructure with 

 RDMA acceleration for multi-node training and inference.

 TCP/IP stack offloading for optimized CPU utilization.

 High-performance NVMe stack for AI workloads.

To experience the record-setting performance of 

Pioneering Multinode Heterogeneous Inference: MangoBoost and AMD, in Collaboration with OEM Partners, Sets Record Llama2-70B Performance in MLPerf Inference v5.1

Record Breaking Performance for DPU-accelerated NVMe/TCP system submission for MLPerf™ Storage.

Best-performing ethernet-based systems result in MLPerf Storage v2.0.

Near-local SSD performance by accelerating the entire NVMe/TCP stack fully on hardware on our DPU solutions and achieving line-rate performance on 400G network.

Lower TCO and hardware cost for high-performance storage.

Easy to deploy on standard systems. Our DPU solutions provide the standard interface in both hardware and software without modification to the existing infrastructure.

Try it now. If interested in the MangoBoost AI solutions, please contact us at contact@mangoboost.io

In this MLPerf Storage 2.0 round, MangoBoost is proud to announce how our Mango StorageBoostTM addresses the scalability and performance challenges with its patent-pending DPU technology. StorageBoostTM technology offloads infrastructure and data-processing tasks from the host to datacenter-deployable FPGA PCIe cards. By deploying our best-in-class NVMe/TCP Initiator on the host and NVMe/TCP Target on the storage server, we further push the performance of 3D-UNet training on both A100 and H100 configuration making it the leading Ethernet-based storage system for AI training.

DPU-accelerated NVMe/TCP submission for MLPerf™ Storage Solutions

Modern high-performance computing applications, including large-language models, are growing in complexity and size. This trend places significant demands not only on compute and memory, but also on storage systems. Storage I/O bottlenecks can lead to underutilized GPUs. As models increase in size, the need for high-performance, high-capacity storage becomes even more critical. While technologies like NVMe/TCP enable remote block devices for ML systems and help meet capacity demands, these remote storage solutions often experience performance degradation due to the CPU overhead associated with NVMe/TCP software stack.

To address this performance bottleneck, DPU-accelerated Mango StorageBoostTM Initiator and Mango StorageBoostTM Target fully offload NVMe/TCP to the hardware, delivering line-rate performance and allows remote storage to operate at the speed similar to a local high-performance NVMe.

In this round, we have set up our evaluation for MLPerf Storage 2.0 Training workload by connecting the host with Mango StorageBoostTM Initiator DPU with the storage server with Mango StorageBoostTM Target through ethernet switch with a 400G network connection between the host and the storage server as shown in Figure 1. The host then emulates the demand of ML workloads by running multiple copies of emulated A100 and H100 3D-UNet traces on the storage system with the goal of testing the maximum number of GPUs that can be connected to the storage system.

MangoBoost shows the best-performing Ethernet-based system result in MLPerf Storage v2.0 and provides near-local SSD performance.

Measuring the performance efficiency of a remote storage system targeting ML workloads can be done through two key metrics: number of GPUs that the storage system can handle normalized by the number of clients, and the number of GPUs normalized by the network bandwidth. Figure 2 shows the overall GPU scalability per unit relative to other submitters in the Fabric-attached Block Storage category (both NVMe/TCP and NVMe/RDMA).These results show how MangoBoost’s DPU solution delivers the best performance: 6.2x for 3D-UNET A100 and 1.25x-7.5x for 3D-UNET H100, and how offloading NVMe/TCP to hardware allows more GPUs to be connected to the storage system.

Figure 3 shows the performance measured as the number of GPUs that can be connected normalized by the network bandwidth for both NVMe/RDMA and NVMe/TCP submitters. This result highlights the efficiency of our solution in utilizing the available network bandwidth as MangoBoost solutions provide the best performance: 1.57x for 3D-UNET A100 and 1.27x-2.05x for 3D-UNET H100.

Figure 4 also provides further performance breakdown across two of our MLPerf Storage submissions (MLPerf Storage 1.0 and MLPerf Storage 2.0), showing that the DPUs can achieve near line-rate performance regardless of GPU architectures (H100 and A100 architectures).

The combination of the Mango StorageBoostTM Initiator DPU on the host and the Mango StorageBoostTM Target DPU on the target also allows ML workloads to operate remote block storage at performance near that of local storage. This is highlighted in Figure 5 through the performance comparison between MangoBoost solutions against local SSD baselines. Our system deployed four Solidigm D7-PS1030s on the target. We normalized the score by the number of SSDs to compare with other non-Fabric attached submissions using a single SSD.

Industry-leading DPU storage solutions, delivering higher performance and lower TCO

While MangoBoost is the only submitter focusing on DPU solutions, this section now focuses on the performance comparison of Mango StorageBoost DPUs against the most apple-to-apple DPU solution available in the market – the NVIDIA BlueField-3 400G DPU, using the same MLPerf Storage 2.0 methodology but replacing MangoBoost DPU with NVIDIA BlueField-3 as shown in Figure 6.

To highlight our performance and cost-efficiency, we rerun MLPerf Storage 2.0 on 3D-UNET benchmarks using the same methodology as the official MLPerf Storage 2.0. Figure 7 compares the scores of BlueField-3 running NVMe/TCP and NVMe/RDMA with host CPU and MangoBoost NVMe/TCP full offloading. The results demonstrate that Mango StorageBoost significantly outperforms BlueField-3 NVMe/TCP and even shows higher performance than BlueField-3 NVMe/RDMA.

Additionally, we also ran the Llama3-8B checkpointing workload on the same setup (this result is not included in v2.0, we will submit the checkpointing results in the next round). Figure 8 shows the comparison of checkpointing performance for different storage configurations. The setup used only one client and did not fully saturate the given bandwidth. The I/O pattern of MLPerf Storage v2.0 checkpointing is not optimized for throughput; shorter storage system latency is crucial for achieving higher throughput on a single client. The results demonstrate that Mango StorageBoost significantly outperformed BlueFIeld-3 (NVMe/TCP) and achieved near local SSD performance.

The superior performance advantage directly translates into TCO reduction. Figure 9 shows the TCO reduction of MangoBoost DPU compared to BlueField-3; as more storage is deployed, the TCO gap widens. For more details, please check our 

, which provides further analysis on throughput and cost breakdown.

Mango StorageBoost™ Traditional Storage, JBoF, JBoD and EBoF Solutions.

Mango StorageBoost™ is a high-performance vendor-agnostic easy-to-use solution that allows AI systems running on any host and GPUs to scale their ML workloads.

Solution 1: Mango StorageBoost™- NVMe/TCP Initiator (NTI).

Mango StorageBoost™- NVMe/TCP Initiator (NTI) is a high-performance NVMe-oF initiator solution that unlocks the full potential of storage disaggregation. By offloading the entire NVMe/TCP stack into hardware, NTI delivers unmatched full-duplex line-rate performance with zero CPU consumption.

Mango StorageBoost™- NVMe/TCP Initiator (NTI) integrates seamlessly into NVMe/TCP initiator servers. By exposing the DPU as a standard NVMe PCIe device, existing storage systems can leverage NTI without the need for any SW modifications. Furthermore, NTI can connect to any NVMe/TCP target servers over a standard TCP/IP network.

Solution 2: Mango StorageBoost™- NVMe/TCP Target (NTT).

Mango StorageBoost™ - NVMe/TCP Target is a groundbreaking NVMe-oF solution, powering the storage system with outstanding performance. It enables disaggregated storage servers to be connected to the network through the standard TCP/IP protocol stack. By fully offloading and accelerating both the TCP/IP and NVMe-oF processing, it entirely eliminates the CPU intervention from the I/O path.

On top of Mango NetworkBoost™ - TCP/IP, NTT decapsulates the TCP packets into NVMe-oF commands and translates them into standard NVMe commands. By utilizing PCIe peer-to-peer technology, NTT directly sends those commands to the peer NVMe SSDs without any CPU consumption. Additionally, its modular and flexible architecture also enables more dynamic customer-driven storage systems such as data reduction and reliability.

Solution 3: Mango StorageBoost™- GPU Storage Boost (GSB).

Mango StorageBoost™- GPU Storage Boost (GSB) is an essential software solution designed to dramatically enhance the efficiency of GPU data movement, enabling unprecedented performance capabilities. By establishing a direct data path for DMA (Direct Memory Access) transfers between GPU memory and storage, GSB eliminates the need for bounce buffers through the CPU, and bypasses CPU involvement entirely. It facilitates peer-to-peer communication within GPU systems, regardless of whether the storage is local or remote. Furthermore, GSB offers a nearly identical syntax and semantic structure to traditional POSIX File APIs, making integration straightforward. This allows for efficient, filesystem-based data management with minimal adaptation, enabling users to harness the full power of the GPU systems without sacrificing ease of use.

When combined with Mango StorageBoost™- NVMe/TCP Initiator (NTI), GSB enables highly efficient GPU data movement, even over remote storage systems based on the TCP/IP transport layer, without any CPU involvement. Typically, handling TCP/IP protocols and the NVMe/TCP stack requires CPU intervention and memory copying, which introduces latency and reduces overall efficiency. NTI addresses these challenges by offloading these tasks to dedicated hardware, allowing the host to leverage the DMA engine as if it were directly connected to an NVMe PCI device. NTI effectively removes the performance and efficiency constraints associated with CPU resources during remote storage access, while GSB eliminates similar limitations for GPU data movement. By utilizing both solutions together, users can unlock unparalleled synergies, dramatically improving data flow efficiency and reducing system bottlenecks.

With the three solutions combined, MangoBoost DPU not only can be used for a traditional remote block storage, but can be used to manage headless storage network including Just-a-bunch-of-Flash (JBoF), Just-a-bunch-of-Disk (JBoD), and Ethernet-bunch-of-Flash (EBoD) to further lower the TCO of high-performance storage system.

Interested in how our DPU solution can deliver higher performance and lower your TCO, try our solutions today by contacting us at 

MangoBoost Demonstrates Record-Breaking Performance in AI Training Storage Solution with MLPerf® Storage v2.0 

MangoBoost, a provider of cutting-edge system solutions designed to maximize AI data center efficiency, has set a new industry benchmark with its latest MLPerf Training v5.0 submission. The company’s Mango LLMBoost™ AI Enterprise MLOps Software together with Mango GPUBoost™ RoCEv2 RDMA solution has demonstrated unparalleled performance on AMD Instinct™ MI300X GPUs, delivering the first-ever MLPerf multi-node results on AMD GPUs.

RoCEv2 RDMA solution on AMD Instinct™ MI300X servers.

First-ever multi-node MLPerf submission using AMD Instinct™ GPUs

, achieving the fastest Llama2-70B-LoRA fine-tuning on AMD hardware at just 

Customizable RoCEv2-based DPU-powered communication stack

 ensures seamless multi-node scalability—purpose-built for high-throughput AI and cloud infrastructure.

 (95–100% efficiency) on additional models like Llama2-7B and Llama3.1-8B, validating real-world scalability beyond benchmark tuning.

Model-agnostic, hardware-flexible, and deployment-ready

 across public cloud or on-premise—MangoBoost enables enterprise-grade training anywhere.

End-to-End Software-Hardware Stack for Performance at Scale

At the heart of this achievement is MangoBoost’s integrated platform, illustrated in Figure 1:

, featuring patent-pending technology for auto-tuning, model parallelism, batch scheduling, and memory management.

, enabling customizable, low-latency, high-throughput communication across nodes using RDMA over converged Ethernet.

First-Ever Multi-Node MLPerf Submission on AMD GPUs

Llama2-70B-LoRA fine-tuning across four AMD Instinct™ MI300X nodes (32 GPUs)

. This breakthrough enabled us to achieve:

The fastest Llama2-70B-LoRA fine-tuning time on AMD GPUs

 using MangoBoost’s end-to-end AI stack

 that demonstrate the maturity of AMD MI300X for real-world AI workloads

first-ever multi-node MLPerf Training result on AMD Instinct™ GPUs

high-performance large-scale AI training is now possible beyond vendor-locked platforms

, our AMD MI300X runs—spanning 8, 16, and 32 GPUs—exhibit 

scaling behavior comparable to prior NVIDIA H100 submissions in MLPerf Training v4.1

MI300X can deliver competitive multi-node performance

 backed by our software-hardware co-design.

This milestone was also recognized by industry leaders:

"

I'm excited to see MangoBoost's first MLPerf Training results, pairing their LLMBoost AI Enterprise MLOps software with their RoCEv2-based GPUBoost DPU hardware to unlock the full power of AMD GPUs, demonstrated by their scalable performance from a single-node MI300X to 2- and 4-node MI300X results on Llama2-70B LoRA. Their results underscore that a well-optimized software stack is critical to fully harness the capabilities of modern AI accelerators.

We congratulate Mangoboost on their MLPerf 5.0 training results on AMD GPUs and are excited to continue our collaboration with them to unleash the full power of AMD GPUs. In this MLperf Training submission, MangoBoost has achieved a key milestone in demonstrating training results on AMD GPUs across 4 nodes (32 GPUs). This showcases how the AMD Instinct™ MI300X GPUs and ROCm™ software stack synergize with MangoBoost's LLMBoost™ AI Enterprise software and GPUBoost™ RoCEv2 NIC.

, Fellow, AI Performance Design Engineering, AMD

As our first multi-node MLPerf Training submission on AMD Instinct™ MI300X, this result already demonstrates strong performance and scalability. With continued tuning and deeper hardware-software co-optimization, we expect even stronger results in future rounds.

Demonstrating Near-Linear Scaling on Llama2-7B and Llama3.1-8B

MangoBoost has benchmarked additional large language models—including 

 of its LLMBoost™ platform. Unlike our MLPerf Training v5.0 submission, these results are from 

, as MLPerf currently does not support these two models.

As shown in Figure 3, both models exhibit 

 as GPU count increases, confirming that our 

MLOps software and RDMA communication stack

 can efficiently orchestrate training across a variety of cluster sizes and workloads.

architecture-aware scheduling and optimized RDMA communication

, adapting well to diverse real-world training scenarios.

MangoBoost’s LLMBoost™ platform is production-ready, scalable, and optimized for modern AI training workloads—regardless of model size or structure.

Mango GPUBoost™ RDMA: Unlocking Multi-Node Training at Scale

Multi-node training isn’t just about adding more GPUs—it's about ensuring those GPUs communicate 

 that outpace standard RNICs in the market.

Up to 3.2Tbps RDMA (RoCEv2) throughput in total (up to 400Gbps per NIC)

Peer-to-peer communication between RNIC and GPU (i.e., ROCmRDMA support)

Customized congestion control and load balancing

Workload-specific optimization hooks using RoCEv2 extended headers

High connection scalability: Maintains line-rate throughput at thousands of QPs—3.4× higher than standard RNICs under heavy load

MangoBoost’s RDMA stack ensures that your training isn’t bottlenecked by the network—one of the biggest pain points in distributed deep learning today.

Collaboration with AMD: Unlocking the Full Potential of MI300X GPUs

MangoBoost’s ground-breaking MLPerf performance was made possible through deep collaboration with our partner 

 with industry-leading compute density and massive memory bandwidth.

Together, AMD’s ROCm platform and MangoBoost’s LLMBoost stack deliver an 

AI training and finetuning solution that is fast, scalable, and easy to deploy

—whether on a single node or across a multi-node cluster. MangoBoost’s GPUBoost RoCEv2 RDMA solutions further pushes the performance and scalability of the GPUs, and allows training across multiple MI300X nodes.

Train Anywhere with Mango LLMBoost™: Model-Agnostic, Hardware-Flexible, and Deployment-Ready

Modern AI infrastructure demands flexibility—not lock-in. MangoBoost’s LLMBoost AI Enterprise MLOps Software is built to support 

enterprise-grade LLM training wherever you need it

, whether that’s on public cloud, private datacenters, or hybrid environments.

MangoBoost supports a broad range of open and custom LLM architectures out of the box:

Whether you're training from scratch or fine-tuning with adapters, MangoBoost’s stack adapts to your model with minimal configuration.

Our full-stack platform is optimized for multiple GPU backends, including:

 — deeply optimized with ROCm and RDMA acceleration

 — compatible via standard CUDA/NCCL support

Future support for other AI accelerators via our abstraction layer

 and can make hardware choices based on price, performance, or availability.

☁️ Cloud & On-Premise Deployment Ready

MangoBoost offers seamless deployment modes:

: Preconfigured Docker images for AWS, Azure, and GCP

: Hardware-aware deployment scripts and RDMA stack support

Blog

Ready for
a boost?

Blog

Pioneering Multinode Heterogeneous Inference: MangoBoost and AMD, in Collaboration with OEM Partners, Sets Record Llama2-70B Performance in MLPerf Inference v5.1

MangoBoost Demonstrates Record-Breaking Performance in AI Training Storage Solution with MLPerf® Storage v2.0

MangoBoost Sets a New MLPerf Training v5.0 Standard for Multi-nodes Llama2-70B-LoRA on AMD Instinct™ MI300X GPU!

MangoBoost Sets The Highest MLPerf Inference v5.0 Result in History for Llama2-70B Offline!

MangoBoost Demonstrates Mango GPUBoost RDMA™ with MLPerf Training v4.1 on a Multi-Node System Powered by AMD MI300X GPUs

MangoBoost Demonstrates Turn-Key Storage Solution for AI Training with MLPerf® Storage v1.0

MangoBoost Demonstrates a Turn-Key LLM Inference Serving Solution on AMD Instinct™ MI300X GPUs with the MLPerf® Inference v4.1 Llama Benchmark

Ready for a boost?

Ready for
a boost?