Baseten needed to offer customers a cloud inference platform that could scale from sporadic traffic to high-volume real-time demand while keeping cold starts, infrastructure costs, and operational complexity low.
Reported impact
Baseten reduced cold starts from up to five minutes to roughly 5-10 seconds, delivered 30-60x faster startup behavior, and boosted one customer's LLM inference performance by 2x while keeping the platform elastic enough for real-time workloads.
Baseten built its managed inference platform on NVIDIA GPU instances in AWS and paired that compute layer with NVIDIA TensorRT-LLM to speed up large-model serving. Customers can bring their own models or deploy pretrained ones through Baseten's open-source Truss framework, which packages the models and routes them onto autoscaling GPU infrastructure. NVIDIA A100 and A10-class instances supply the accelerated runtime, while TensorRT-LLM provides optimizations such as tensor parallelism to improve LLM throughput and cost efficiency. NVIDIA AI Enterprise gives Baseten access to the production-grade TensorRT-LLM stack, and the platform's autoscaling controls adjust replicas as traffic changes. Together, those pieces let Baseten turn model deployment into an on-demand service with fast startup times and less manual infrastructure tuning.
Customers package models with Truss or choose pretrained ones, Baseten deploys them onto autoscaling EC2 GPU instances, TensorRT-LLM serves the optimized inference path on A100 or A10 hardware, and results are returned through the managed platform as traffic expands or contracts.
Baseten leverages NVIDIA GPUs and NVIDIA® TensorRT™-LLM to provide machine learning infrastructure that's high-performance, scalable, and cost-effective.
Baseten's infrastructure, powered by NVIDIA GPUs on AWS EC2 instances, only takes 5-10 seconds to get the model up and running.
This is an incredible speedup on cold starts, which previously took up to five minutes—a speedup of 30-60X.
Using the TensorRT-LLM feature of tensor parallelism served on AWS, Baseten boosted inference performance for a customer's LLM deployment by 2X through their open-source Truss framework.
With Baseten, organizations have what they need to deploy and serve ML models performantly, scalably, and cost-effectively for real-time applications.
Review how this reference was scored and trace each normalized claim back to its documented source.
Scoring methodology
Customer Reference Value measures usefulness as a customer reference. Evidence Confidence measures the quality and traceability of available evidence, not independent verification.
Traceability
Field coverage shows which facts each source supports. Dates are shown only when documented by the source.
Related
NVIDIA Sample Sep 22, 2026 Sluicebox Sluicebox needed to turn fragmented bills of materials, supplier records, PDFs, CAD files, and emissions inventories into timely product carbon assessments, but manual life-cycle-analysis work was slow, expensive, and… 51-80% lower inference costs Lucy · NVIDIA Nemotron 3 Ultra · carbon graph NVIDIA Sample Sep 15, 2026 Lambda Lambda needed to raise token throughput and improve AI-cloud economics, but fixed rack and facility power budgets were leaving GPU capacity stranded and limiting how many nodes could run safely at once. +17% Hybrid Training Throughput Dynamic Power Software · NVIDIA DSX AI Factory Platform · NVIDIA DSX MaxLPS NVIDIA Sample Sep 1, 2026 Onetera Onetera needed to help more than 90 municipalities turn fragmented regulations, zoning maps, permit rules, and legacy documents into fast resident-facing services without imposing heavy IT overhead or expensive manual… 1.7x increase in data throughput GPT OSS 20B · Government Intelligence Agent · LangGraph 