Perplexity needed a production-grade inference stack for pplx-api that could serve open-source LLMs with lower latency, lower first-token delay, and lower operating cost while still scaling elastically across large GPU fleets.
Reported impact
Perplexity achieved up to 3.1x lower latency and up to 4.3x lower first-token latency, cut serving cost by 4x, saved about $600,000 per year by re-pointing external API references to pplx-api, and doubled throughput when moving comparable workloads from A100 to H100 GPUs with FP8 precision.
Perplexity built pplx-api on NVIDIA-accelerated AWS infrastructure so developers could call open-source language models through a low-latency API instead of managing their own inference clusters. The serving layer runs on Amazon P4d and P5 instances backed by NVIDIA A100 and H100 GPUs, and NVIDIA TensorRT-LLM applies optimizations such as FlashAttention, masked multi-head attention, quantization, and tensor parallelism to improve both context processing and token generation. NVIDIA AI Enterprise supplies the production-ready software base around those optimizations, while AWS-integrated Kubernetes scales the service across hundreds of GPUs. This combination gives Perplexity a faster and cheaper inference plane for real-time applications without sacrificing elasticity or uptime.
Developer API requests hit pplx-api, where TensorRT-LLM-optimized model servers run on A100 and H100 GPU fleets in AWS. Kubernetes scales those services beyond hundreds of GPUs, and the optimized inference path returns tokens with lower latency and lower first-token delay than Perplexity's prior external API routing model.
By leveraging NVIDIA H100 GPUs and FP8 precision on Amazon P5 instances, they can cut their latency in half and boost throughput by 200 percent compared to NVIDIA A100 GPUs.
Perplexity harnesses the power of NVIDIA's hardware and software to solve this challenge.
Perplexity was able to lower costs by 4X by simply switching their external inference-serving API references to call pplx-api, resulting in savings of $600,000 per year.
Pplx-api can achieve up to 3.1X lower latency and up to 4.3X lower first-token latency relative to other deployment platforms.
Review how this reference was scored and trace each normalized claim back to its documented source.
Scoring methodology
Customer Reference Value measures usefulness as a customer reference. Evidence Confidence measures the quality and traceability of available evidence, not independent verification.
Traceability
Field coverage shows which facts each source supports. Dates are shown only when documented by the source.
The public company page includes numberOfEmployees value 1408 and a PostalAddress with streetAddress 115 Sansome St, addressLocality San Francisco, and addressCountry US.
Related
NVIDIA Sample Sep 22, 2026 Sluicebox Sluicebox needed to turn fragmented bills of materials, supplier records, PDFs, CAD files, and emissions inventories into timely product carbon assessments, but manual life-cycle-analysis work was slow, expensive, and… 51-80% lower inference costs Lucy · NVIDIA Nemotron 3 Ultra · carbon graph NVIDIA Sample Sep 15, 2026 Lambda Lambda needed to raise token throughput and improve AI-cloud economics, but fixed rack and facility power budgets were leaving GPU capacity stranded and limiting how many nodes could run safely at once. +17% Hybrid Training Throughput Dynamic Power Software · NVIDIA DSX AI Factory Platform · NVIDIA DSX MaxLPS NVIDIA Sample Sep 1, 2026 Onetera Onetera needed to help more than 90 municipalities turn fragmented regulations, zoning maps, permit rules, and legacy documents into fast resident-facing services without imposing heavy IT overhead or expensive manual… 1.7x increase in data throughput GPT OSS 20B · Government Intelligence Agent · LangGraph 