MangoBoost Launches Mango Inference, Emerging as a Key Partner Powering AMD's Enterprise AI Ecosystem

Key Highlights

• Mango Inference is live: A serverless inference service (LLM-as-a-service) that gives developers production-ready access to leading open models, including GLM 5.3/5.2, Kimi K3, DeepSeek V4 and MiniMax M3, through OpenAI and Anthropic-compatible APIs. New accounts receive $10 in free credits and 30% off model usage.
• Fully on AMD Instinct™ GPUs: Every token is served on AMD Instinct™ GPUs and AMD ROCm™ software, an open stack from model to infrastructure.
• Measured performance: On the same AMD Instinct™ MI355X GPU, Mango Inference delivers 2.1x to 2.8x the throughput of open-source serving engines at equal latency, with no loss in answer quality. On the Respan gateway, it ranks first or second on 30-day average latency for GLM-5.2, DeepSeek-V4-Pro and MiniMax-M3.
• Lower effective token price for agents: A tiered KV cache keeps prompt prefixes warm for about one hour, several times longer than typical providers, so agent sessions that resume hit the cache instead of paying full price for the whole context.
• Self-operated infrastructure, with data residency options: Mango Inference runs in MangoBoost’s own AI data center on AMD Instinct™ GPU clusters that MangoBoost operates itself, with zero data retention, and regional deployment options for customers that have data residency requirements.
• Backed by MLPerf® experience: The team behind Mango Inference has published MLPerf® Inference results with AMD across four consecutive rounds, including the largest multi-region cluster ever submitted.
• Featured by AMD: AMD has published a blog introducing Mango Inference, and MangoBoost is featured in AMD’s Enterprise AI partner ecosystem as an inference service provider.

Open Models, Production Ready

MangoBoost is proud to announce the launch of Mango Inference, a serverless inference service that makes leading open models available through a single endpoint with competitive, usage-based pricing.

Open-weight models are closing in on the frontier. Model families such as Kimi, DeepSeek, GPT-OSS and MiniMax now approach the quality of closed models on many enterprise workloads, at a fraction of the token price. They also give teams something closed APIs cannot: full control over the data path, the deployment location and the model itself, with no vendor lock-in. As the quality gap narrows, the questions teams ask are shifting toward serving efficiency, deployment speed, cost control and infrastructure choice.

Running an open model well is still hard. Taking a single model to production involves more than 20 separate tasks: GPU procurement, drivers, containers and deployment, model weights and license review, quantization, tensor and pipeline parallelism, KV cache tuning, batch scheduling, request routing, autoscaling, monitoring, logging, rate limits, usage metering, tenant isolation and on-call. Mango Inference removes that work. MangoBoost manages the models, the GPUs and the data center infrastructure, so development teams can focus on their applications instead of model-serving operations.

[Press event: Mango Inference was announced at <A New Era of AI Inference, Powered by MangoBoost, 10 October 2026, Grand InterContinental Parnas, Seoul, South Korea>. Read the press release]

The launch comes alongside a blog from AMD introducing Mango Inference, and MangoBoost is now featured in AMD’s Enterprise AI partner ecosystem.

What Developers Get with Mango Inference

Mango Inference is designed so that teams can move from sign-up to their first production request in minutes, using the tools and client libraries they already have.

• One endpoint, many models: Access every model in the catalog through a single endpoint and switch models with a one-line configuration change.
• Familiar APIs: Mango Inference provides OpenAI-compatible and Anthropic-compatible APIs, so existing clients and SDKs work with minimal changes. Coding agents including Claude Code, OpenCode and OpenHands are supported.
• Application-ready features: Streaming and reasoning, plus tool calling when the model supports it.
• No GPUs to manage, no idle capacity to pay for: Prepaid accounts, billed by token. Every response reports token usage, and cached input is billed at a lower rate, which lowers costs for agentic workloads that resend long system prompts.
• One management console: Create API keys, browse the model catalog with context length and availability, test models in the Playground and track usage in one place.

Why Mango Inference: Performance, Efficiency and Data Sovereignty

Many services can host an open model. Mango Inference differentiates on three axes that decide whether an open model is actually cheaper and better to run as a service: performance per GPU, efficiency of the KV cache that agentic workloads depend on, and sovereignty over where customer data is processed.

1. Performance: more tokens per GPU, faster responses, same answer quality

Mango Inference is served by Mango LLMBoost™, MangoBoost’s inference engine, running on AMD Instinct™ MI355X GPUs with disaggregated prefill and decode. We measured it against open-source SGLang and vLLM on the same MI355X GPU, using an 8K-token input and 1K-token output workload that is typical of agentic traffic. At equal latency, LLMBoost delivers 2.1x to 2.8x the output tokens per GPU (2.8x on GLM, 2.6x on DeepSeek-V4-Pro, 2.1x on MiniMax-M3). At equal throughput, it answers 3.2x to 4.3x faster. The same measurements place MI355X with LLMBoost above open-source engines running on NVIDIA B300 (InferenceX results).

Faster than peers on a live gateway. Benchmarks measure a controlled setup. Gateways measure what customers experience. On Respan, an LLM routing gateway that lists Mango Inference alongside other providers, 30-day average request latency from live traffic places Mango Inference first or second on GLM-5.2, DeepSeek-V4-Pro and MiniMax-M3.

No loss in model quality. Speed is only useful if the answers stay right. Mango Inference serves each model in a precision chosen to keep accuracy at the top of the field. On GPQA-Diamond, a graduate-level science benchmark, Mango Inference scores at or near the top of all listed providers for every model, including providers that serve the model in FP8 or an undisclosed precision.

2. Efficiency: a KV cache that stays warm, so agents pay less per token

Inference workloads are shifting to context-heavy traffic. Across OpenRouter, the average prompt grew about 4x in 2025, from roughly 1.5K to more than 6K tokens, and reasoning models went from almost none of the tokens served in early 2025 to the majority by year-end (OpenRouter State of AI 2025). Coding agents and multi-step agents resend the same long context on every turn.

That makes the KV cache a cost structure, not a technical detail. When a session resumes, the prior context is billed at the cached-input rate if the provider still holds the prefix in cache. If the prefix has been evicted, the full context is resent at list price, which costs 5 to 10 times more than a cache hit. How long a provider keeps a prefix alive therefore sets the effective token price.

Mango Inference applies MangoBoost’s full stack to keep the hit rate high:

• Tiered KV cache: Prefix retention extends across three tiers, from GPU HBM to pooled node DRAM to storage on Mango Kesar™ servers. Prefixes are shared across nodes over a DPU-based high-speed network, and frequently used prefixes are protected from eviction.
• Cache-aware routing: Each request is routed to the GPU that already holds its prefix. KV location and GPU load are considered together, so both hit rate and tail latency improve. The result is fewer redundant GPU computations and higher inference efficiency.

In practice, Mango Inference keeps a prefix cache alive for about one hour after the last request. Typical providers keep it for 10 to 30 minutes, and providers without a DRAM pool for well under a minute. A session that resumes after 20 or 45 minutes still hits the cache on Mango Inference and is billed at the cached rate.

3. Data sovereignty: your data processed on infrastructure we operate

Growing data-control requirements make where inference runs a procurement question. Sending prompts to an overseas third-party API can constitute a cross-border transfer of personal data. Data held abroad is subject to foreign courts and preservation orders, regardless of what a contract says about retention. Export controls or sanctions can cut off an overseas API without notice. And storing data on overseas servers is enough to exclude a vendor from many public-sector tenders and regulated-industry security reviews.

Mango Inference serves open-weight models from GPU clusters that MangoBoost operates itself, rather than from rented third-party clouds. The launch cluster is in Seoul, Korea. For Korean customers, prompts and outputs are processed in country under Korean jurisdiction, with guaranteed deletion, and the service is free from external disruption. For customers with data residency or latency requirements elsewhere, including sovereign AI programs, regional deployment options are available.

Privacy by default. Mango Inference does not store prompts or outputs (zero data retention). The console supports role-based access control, multi-factor authentication and logging of admin actions. See our policies and user agreement.

Built on MangoBoost’s Full-Stack AI Infrastructure

Mango Inference is the service layer of a stack that MangoBoost builds end to end. Owning every layer is what makes the performance and efficiency results above possible.

• Mango Inference (service layer): OpenAI- and Anthropic-compatible APIs, the management console and usage-based billing. Customers never manage GPUs and pay only for what they use.
• Mango LLMBoost™ (inference optimization layer): The inference engine, optimized with disaggregated prefill and decode on AMD Instinct™ GPUs. This is where the multiples in throughput, per-user speed and latency come from.
• Mango DPU and servers (infrastructure layer): Mango Alphonso™ GPU servers built on AMD Instinct™ MI355X and MI350P, Mango Kesar™ AI storage servers that hold KV cache and model weights, and Mango BoostX™, our in-house DPU that offloads network and storage.

Together these layers form Mango AI Factory, MangoBoost’s own full-stack AI data center, which powers Mango Inference operations. Because the DPU, the storage tier and the inference engine are designed together, capabilities such as cross-node KV cache sharing and cache-aware routing are built in rather than bolted on.

Fully on AMD Instinct™, Proven Across Four Rounds of MLPerf® Inference

Mango Inference runs entirely on AMD Instinct™ GPUs and the open-source AMD ROCm™ software stack. Customers get an open stack from model to infrastructure: open-weight models, standard APIs, AMD Instinct™ GPUs and ROCm™ software. This gives enterprises and sovereign AI programs more infrastructure choice and reduces dependence on a single closed model or infrastructure provider.

The service is built and operated by the same team behind our MLPerf® results on AMD Instinct™ GPUs. MLPerf® Inference, from MLCommons, is the industry-standard benchmark suite for AI inference. Working with AMD, MangoBoost has submitted results on AMD Instinct™ GPUs in four consecutive rounds:

• MLPerf Inference v5.0 (April 2025): Four AMD Instinct™ MI300X nodes delivered the first multi-node MLPerf® Inference result on AMD Instinct™, at approximately 103,000 tokens per second on Llama 2 70B Offline, the highest recorded result at the time. (read more)
• MLPerf Inference v5.1 (September 2025): Four MI300X nodes and two MI325X nodes served Llama 2 70B as one system at approximately 160,000 tokens per second, the first MLPerf® cluster to combine GPU generations. (read more)
• MLPerf Inference v6.0 (April 2026): MI355X capacity in the United States and MI300X and MI325X systems in Korea formed the first multi-region MLPerf® cluster spanning three GPU generations, with 95.5% scaling efficiency on Llama 2 70B across two continents. (read more)
• MLPerf Inference v6.1 (September 2026): A 32-GPU deployment of AMD Instinct™ MI300X and MI355X GPUs across four sites on two continents served GPT-OSS-120B through one endpoint at up to 97% scaling efficiency. It was the largest multi-region cluster ever submitted to MLPerf® and included the first prefill/decode-disaggregated submissions on AMD Instinct™ GPUs. (read more)

Our MLPerf Inference v6.1 write-up also reports throughput per dollar on AMD Instinct™ GPUs, derived from the published MLPerf® results. The engineering experience from these submissions, from multi-region routing to prefill/decode disaggregation, is the same engineering that runs Mango Inference today.

Our MLPerf® results are the product of joint engineering work with AMD. AMD has published a blog introducing Mango Inference, and MangoBoost is featured in AMD’s Enterprise AI partner ecosystem as an inference service provider. We will continue working with AMD to bring more open models and more AMD Instinct™ capacity to Mango Inference customers.

Model Catalog and Roadmap

Mango Inference launches with models chosen for the workloads customers run most, and the catalog is designed to keep up as new open-weight models are released.

• Available now: GLM 5.3 and 5.2 for coding and agents, DeepSeek V4 for reasoning and math, Kimi K3 (long context), and MiniMax M3 for ultra-long context and large batches.
• Coming soon: Gemma 4 31B (lightweight, on-device class), MiMo V2.5 Pro (reasoning-focused, lightweight), Qwen3.6 27B and 35B (multilingual, general purpose) and DeepSeek-V4.1 Flash (real-time chat, high-volume processing).

Our internal automation is designed to bring new open models onto AMD Instinct™ GPUs quickly as they are released, so customers can adopt new models without changing their integration. Model requests are welcome through the support channels below.

Try Mango Inference Today

Getting started takes three steps and a few minutes:

1. Get an API key. Sign up at inference.mangoboost.io and create an API key from the console. It takes about 30 seconds, with no waiting for approval.
2. Select and validate a model. Pick an open-weight model, tune parameters in the Playground and check output quality. Playground credits are included.
3. Integrate the code. Point your OpenAI- or Anthropic-compatible client at the Mango Inference endpoint. Change only the base URL and the model name, about three lines, and existing code runs as is. See the Quick Start guide.

New accounts receive $10 in free credits to try the model catalog. [Launch offer: 30% off all models for 30 days. Available in all regions.]

* Offer terms are subject to change or withdrawal at any time without prior notice.

Visit us at SC26 to see Mango Inference live.

For support or model requests, email support@mangoboost.io, use the Support menu in the Mango Inference console, or join the Mango Inference Discord community.

AMD, AMD Instinct, ROCm and combinations thereof are trademarks of Advanced Micro Devices, Inc. MLPerf is a registered trademark of MLCommons. Other names are for identification purposes only and may be trademarks of their respective owners.

MangoBoost: Contact Us