IT Operations Automation
Baseten needed to orchestrate a global, multi-cloud GPU fleet for increasingly large reasoning models while improving latency, throughput, and cost efficiency enough to keep real-time AI inference practical for enterprise customers.
Reported impact
Baseten now serves five times as many requests on the same number of GPUs, improved price-performance for frontier reasoning models by up to 225 percent, reduced latency by up to 38 percent for large LLMs, can provision thousands of GPUs in under five minutes, and boosted throughput by over 60 percent for Writer’s Palmyra models.
Baseten built its inference platform around a unified global GPU pool and then upgraded the serving layer with NVIDIA Blackwell hardware and NVIDIA’s open inference stack. The company aggregates GPUs from more than 10 cloud providers across dozens of regions, uses its multi-cloud capacity management layer to provision thousands of GPUs in under five minutes, and abstracts away regional latency differences for customers. On Google Cloud A4 VMs and NVIDIA HGX B200 systems, Baseten combines NVIDIA CUDA, NVIDIA Dynamo, and NVIDIA TensorRT-LLM with open-source engines such as SGLang to run frontier reasoning models like DeepSeek-R1, Llama 4, and gpt-oss. That architecture supports disaggregated serving, LLM-aware routing, KV-cache offload, and custom model compilation, so Baseten can serve public model APIs and dedicated customer deployments with much higher throughput and lower latency than its previous generation.
Customer traffic hits Baseten’s global GPU pool, the inference stack routes requests to Blackwell-backed endpoints, Dynamo applies distributed serving and cache optimizations, TensorRT-LLM and SGLang execute the selected model efficiently, and the platform returns lower-latency results from public or dedicated deployments.
At the core of this success was Basten’s integration of NVIDIA Dynamo, a low-latency distributed inference serving platform that supports advanced inference optimization techniques such as disaggregated serving, LLM-aware routing, and KV cache offloading to storage.
Baseten can now serve five times as many user requests for custom models with the same number of GPUs.
Baseten developed multi-cloud capacity management (MCM), with the capability to provision thousands of GPUs within less than five minutes.
Baseten reduced latency for serving the largest LLMs like DeepSeek-R1 by up to 38%.
These efforts have boosted throughput by over 60% for Writer’s Palmyra LLMs.
Review how this reference was scored and trace each normalized claim back to its documented source.
Scoring methodology
Customer Reference Value measures usefulness as a customer reference. Evidence Confidence measures the quality and traceability of available evidence, not independent verification.
Traceability
Field coverage shows which facts each source supports. Dates are shown only when documented by the source.
The company profile reports '165 employees'.
Related
NVIDIA Sample Feb 2, 2026 NVIDIA NVIDIA needed to move employee-built chatbots and agents out of shadow-AI pilots and onto a unified, secure, high-performance platform that could support hundreds of governed AI workflows across engineering, software… from 15 hours to one hour NVIDIA AI Blueprint for Retrieval-Augmented Generation (RAG) · NVIDIA AI Enterprise · NVIDIA GPU Operator 
Google Cloud Sample Archived · Oct 16, 2026 Alta Alta needed a transactable Google Cloud Marketplace path for its AI and technology spend management platform without pulling its product team off core forecasting work, while also helping finance and IT leaders… Saved 125 engineering and 60 project management hours Cloud Pub/Sub · Cloud Run · Cloud SQL 
Alibaba Cloud Sample Oct 12, 2026 ECLICKTECH (易点天下) ECLICKTECH needed to industrialize AI-driven advertising operations, but media-delivery workflows were still long, manual, and heavily dependent on senior operators for anomaly diagnosis and creative adaptation. As more… AI covers about 50% of core execution work AdClaw · AgentCore · AgentLoop 