Your GPU Is Bored: A Practical Utilization Playbook

You launch a training job on an expensive GPU. The fans spin up, the model fits, and the dashboard shows a repeating pattern with a burst of activity, a stretch of nothing, another burst, another pause.

The GPU is not slow. The GPU is waiting.

Perhaps a data worker is decoding the next batch. Perhaps Python is launching hundreds of tiny operations. Perhaps a well-intentioned logging call is forcing every queued CUDA operation to finish. Whatever the cause, the result is the same: you are paying for elapsed time while the accelerator has no useful work to execute.

This guide presents a repeatable way to find that lost time. The examples use PyTorch and NVIDIA tooling, but the reasoning applies to most accelerator-backed training and inference systems.

Watch the video walkthrough

First, define what “utilization” means

The GPU-Util value in nvidia-smi is the percentage of a recent sample period during which at least one kernel was executing. It answers “Was the GPU doing anything?”, not “Was the GPU doing useful work efficiently?” A tiny kernel can make the device look busy without using most of its compute capacity.

Memory utilization is easy to misread too. In nvidia-smi, it describes time spent reading or writing device memory, it’s not the percentage of VRAM allocated. For deeper analysis, Nsight Systems can expose SM activity, Tensor Core activity, PCIe or NVLink throughput, memory bandwidth, and the CPU-GPU timeline.

Track all three. A change that raises GPU activity but lowers samples per second is not an optimization. Neither is a throughput win that breaks convergence or pushes serving latency beyond its service-level objective.

Why a GPU goes idle

Most low-utilization problems fall into one of four groups:

1. Starvation: Data preparation or transfer cannot deliver the next batch in time.
2. Underfilled work: Batches or kernels are too small to occupy the device effectively.
3. Synchronization: The host, another device, or a network collective makes the GPU wait.
4. Insufficient demand: An inference service does not have enough concurrent work for a full GPU.

The shape of the timeline points to the relevant group.

Profile before changing code

Start with a coarse view to confirm that an idle-time problem exists, but not to diagnose it:

nvidia-smi dmon -s u -d 1

Then capture a representative window with Nsight Systems. Avoid profiling a cold start alone: compilation, cache population, and initial data reads can dominate it.

nsys profile \  --trace=cuda,nvtx,osrt \  --output=gpu-training-trace \  python train.py

For a model-level view, PyTorch Profiler can rank operators and export a trace for TensorBoard or Chrome:

import torchfrom torch.profiler import ProfilerActivity, profile, record_functionactivities = [ProfilerActivity.CPU, ProfilerActivity.CUDA]with profile(activities=activities, record_shapes=True) as prof:    with record_function("train_step"):        loss = train_one_step(model, batch)print(    prof.key_averages().table(        sort_by="self_cuda_time_total",        row_limit=15,    ))prof.export_chrome_trace("trace.json")

Ask three questions when reading the trace:

• Where are the empty regions on the GPU timeline?
• What is the CPU doing immediately before each gap?
• Are there many short kernels, long copies, explicit synchronizations, or communication collectives?

That evidence determines which lever to pull.

Fix the input pipeline when the GPU is starved

A training step that computes for 40 ms and then waits 120 ms for input is a pipeline problem. Faster math can make it worse by reaching the wait sooner.

PyTorch data loading can overlap CPU preparation with GPU execution by using worker processes. Pinned host memory, combined with a non-blocking device copy, can also reduce host-side transfer stalls. Tune these values on the actual machine—there is no universally correct worker count.

from torch.utils.data import DataLoaderloader = DataLoader(    dataset,    batch_size=microbatch_size,    shuffle=True,    num_workers=8,          # Benchmark this value    pin_memory=True,    persistent_workers=True,    prefetch_factor=2,)for inputs, targets in loader:    inputs = inputs.to("cuda", non_blocking=True)    targets = targets.to("cuda", non_blocking=True)    loss = train_one_step(model, inputs, targets)

Common improvements include:

• Move expensive decoding or augmentation out of the main training process.
• Cache deterministic preprocessing instead of repeating it every epoch.
• Store many tiny samples in a format that supports larger sequential reads.
• Place data close to the compute and watch storage throughput and CPU saturation.
• Bucket variable-length sequences to reduce padding and erratic step times.
• Check NUMA placement on multi-socket hosts as the closest CPU and memory path can matter.

Pinned memory is a finite host resource, not a magic switch. Benchmark it, monitor system memory, and use it with non_blocking=True when you want the host to enqueue copies without synchronizing after each transfer.

Use the right numerical format

Mixed precision can reduce memory traffic and unlock hardware acceleration for supported operations. On recent accelerators, BF16 is often a convenient training format because it retains FP32’s exponent range. FP16 commonly needs gradient scaling to protect small gradients from underflow.

Here is the current PyTorch AMP pattern for FP16:

import torchscaler = torch.amp.GradScaler("cuda")for inputs, targets in loader:    optimizer.zero_grad(set_to_none=True)    with torch.autocast(device_type="cuda", dtype=torch.float16):        predictions = model(inputs)        loss = loss_fn(predictions, targets)    scaler.scale(loss).backward()    scaler.step(optimizer)    scaler.update()

For BF16 on compatible hardware, autocast is usually used without loss scaling:

with torch.autocast(device_type="cuda", dtype=torch.bfloat16):    predictions = model(inputs)    loss = loss_fn(predictions, targets)loss.backward()

Do not assume the result will be numerically equivalent. Compare loss curves and task metrics, watch for NaN or infinite values, and keep numerically sensitive operations in a safer format when necessary.

Increase useful work per launch

Small batches produce small matrix operations and can leave execution resources unused. Increase the microbatch size until throughput stops improving, memory becomes the constraint, or model quality changes. Record samples per second at each point instead of stopping when the utilization gauge looks full.

Gradient accumulation is useful when the desired effective batch size does not fit in memory:

import torchaccumulation_steps = 4assert len(loader) % accumulation_steps == 0optimizer.zero_grad(set_to_none=True)for step, (inputs, targets) in enumerate(loader, start=1):    with torch.autocast(device_type="cuda", dtype=torch.bfloat16):        predictions = model(inputs)        loss = loss_fn(predictions, targets) / accumulation_steps    loss.backward()    if step % accumulation_steps == 0:        optimizer.step()        optimizer.zero_grad(set_to_none=True)

The effective batch is microbatch_size × accumulation_steps for one process. But accumulation does not make each GPU operation larger, the microbatch still determines most kernel shapes. Its main benefit is fitting the desired effective batch while performing fewer optimizer steps.

If activation memory is the blocker, activation checkpointing can trade extra computation for a smaller memory footprint. That may permit a larger microbatch, but the recomputation can reduce step speed. Measure the complete trade-off.

Reduce launch and Python overhead

A model made of many small operations can spend meaningful time launching kernels and returning to Python. torch.compile can capture parts of the program and apply compiler optimizations such as operation fusion:

model = torch.compile(model)

Treat that line as an experiment, not a guaranteed speed-up. The first iterations include compilation overhead, dynamic shapes may trigger recompilation, and unsupported Python behavior can create graph breaks. Benchmark after warm-up, inspect graph-break logs, and confirm outputs before and after compilation.

Also look for accidental synchronizations. CUDA work is asynchronous from the host’s perspective, but operations that need a scalar or CPU copy must wait for queued device work. Repeated calls to .item(), .cpu(), print(cuda_tensor), or torch.cuda.synchronize() inside the hot path can turn an asynchronous pipeline into a serialized one. Aggregate metrics on the device and transfer them less often.

Treat distributed training as a pipeline

Adding GPUs only helps if the extra compute outweighs communication and imbalance. In data-parallel training, one slow rank can hold up every other rank at a collective operation.

Inspect per-rank traces and check for:

• Uneven batch shapes or preprocessing time
• Gradient communication that fails to overlap backward computation
• Network topology, link bandwidth, and incorrect interface selection
• Unnecessary barriers or device synchronization
• Data-loader contention when every rank reads from the same storage path

Report scaling efficiency alongside raw throughput. If one GPU processes 1,000 samples per second and four process only 2,400, the job is faster but much less efficient per GPU. That may still be the correct trade-off when time-to-result matters as it should be an explicit one.

When the model is too small for the GPU

Sometimes the workload is already efficient so it is simply not large enough to occupy the device. This is common in online inference, where requests arrive individually and latency matters.

For stateless models, dynamic batching can combine compatible requests into larger batches. The queue delay creates a direct latency-throughput trade-off, so tune it against a percentile latency objective rather than average latency alone. Multiple model instances can help too, though they compete for the same memory and may add no benefit after batching fills the device.

When independent workloads need stronger isolation, NVIDIA MIG can partition supported GPUs into isolated instances with dedicated compute and memory resources. CUDA MPS instead allows processes to share scheduling resources and overlap work, with different isolation characteristics. Choose based on workload compatibility, isolation, operational complexity, and the GPU models available, not merely on the desire to see a higher utilization number.

A repeatable optimization loop

The takeaway

Low GPU utilization is not one problem. It is a symptom shared by slow input pipelines, small kernels, synchronization, distributed imbalance, and insufficient request concurrency. That is why random tuning produces random results.

Start with the timeline. Find the largest avoidable wait. Fix that constraint, then measure the whole workload again—because the reward for removing one bottleneck is discovering the next one.

Your GPU does not need a busier dashboard. It needs a steadier supply of useful work.