IBM × IBM — IT Operations Automation

IBM

IT Operations Automation

IBM Research's Big AI Models team needed to keep scarce GPUs available for a growing set of LLM experiments and inference workloads, but overprovisioned Kubernetes instances were wasting expensive accelerator capacity and forcing engineers to spend time manually monitoring and resizing resources.

Reported impact

Business outcome

IBM increased idle GPU availability from three to sixteen, doubled throughput without hurting latency, and projected that full automation would eliminate the need for thirteen GPUs, while also freeing engineers from constant manual performance monitoring.

Solution

IBM's Big AI Models team deployed IBM Turbonomic to automate resource management for its internal generative-AI environment. The BAM platform supports engineering teams testing and refining LLM projects on Kubernetes-based inference services backed by more than 100 NVIDIA A100 GPUs. Turbonomic continuously evaluates real-time utilization data, recommends optimal actions, and can automatically scale instances up or down so workloads get the right amount of GPU-backed capacity without overprovisioning. That let the team convert manual monitoring into policy-driven optimization, free idle GPUs for other users, and time-share infrastructure instead of reserving dedicated capacity. The resulting setup preserved performance while increasing throughput and cutting the number of GPUs required in a fully automated environment, turning hybrid-cloud optimization software into a practical control plane for high-demand AI infrastructure.

Data flow

Runtime utilization telemetry from BAM's Kubernetes-hosted LLM services is analyzed by Turbonomic, which recommends or triggers scaling changes so GPU capacity is released from overprovisioned instances and reassigned to other active workloads.

Solution architecture

3 components · 2 layers

Compute
- NVIDIA A100 GPUs Provide the accelerated compute pool that BAM shares across model testing and inference workloads.
Orchestration
- IBM Turbonomic Analyzes runtime demand and automates scaling decisions to optimize BAM's GPU-backed LLM services.
- Kubernetes Runs the LLM inference instances that are scaled up and down based on Turbonomic recommendations.

Architecture clues

The BAM environment runs LLM services on Kubernetes and shares a pool of NVIDIA A100 GPUs.
The team adopted time-sharing to reuse GPUs across multiple workloads.
Turbonomic uses real-time data and can execute automated scaling actions.

Evidence from the source

Now, with scaling and sharing, IBM BAM team showed 13 fewer GPUs will be necessary in a full automation environment.

To optimize their GPU resources and manage their LLM Kubernetes instances, the IBM BAM team deployed IBM Turbonomic

With Turbonomic, the IBM BAM team was able to create a scalable and agile infrastructure

the team show the ability to increase the idle GPU resources from 3 to 16 (5.3 times)

IBM

Business outcome

Solution

Data flow

Solution architecture

Architecture clues

Evidence from the source

How scores are calculated

Sources & provenance

IBM customer story

Date evidence

Similar use cases