New GPUs promise big gains. The messy part is getting your stack to actually run faster without breaking builds, accuracy, or distributed training. Here’s a practical list of the gotchas you’ll hit when moving from, say, Ampere or Hopper to Blackwell, and what to do about each.

1) Driver and CUDA mismatches

This is the number one time sink during upgrades.

The issue: Toolkits and drivers have minimum and compatible ranges. Miss them and you’ll see build errors or cryptic runtime failures.

Fix: Check NVIDIA’s compatibility matrix, then pin versions in code and image metadata. CUDA 11 and later default to minor-version compatibility, and current tables call out the minimum driver per major CUDA release. If you need newer toolkits with older drivers, lean on CUDA Enhanced Compatibility, but still verify the exact range.

2) Incorrect nvcc targets and surprise JIT

Compiling for the wrong SM wastes time or leaves performance on the table.

The issue: Shipping only PTX forces the driver to JIT on first run. Shipping only old cubins can make kernels ineligible on newer GPUs.

Fix: Build fatbins with native cubins for your new architecture plus a PTX fallback. Example: include sm_80, sm_90, and compute_52 PTX for portability. NVIDIA’s docs and blog show the pattern and how to inspect contents with cuobjdump. For Blackwell specifically, target compute capability 10.0 in your -gencode flags.

3) Numeric behavior changes

Speed ups can change math unless you control the knobs.

The issue: Ampere enabled TF32 tensor cores by default for many FP32 GEMMs. Hopper adds FP8 via Transformer Engine. Either can shift accuracy if you don’t plan for it.

Fix: Decide precision per workload. If you rely on strict FP32, disable TF32 in your DL framework. If you want the speed, keep it on and assert acceptable tolerances. For transformer training or long-context inference on Hopper and newer, evaluate FP8 with NVIDIA’s Transformer Engine to cut memory and boost throughput.

4) Kernel regressions from resource layout differences

New SMs, cache, and memory paths change what “fast” means.

The issue: A kernel tuned on A100 might underperform on H100 or B200 due to different shared memory limits, scheduling, or new copy engines.

Fix: Re-profile, don’t assume. Use Nsight Compute’s Occupancy Calculator and guided analysis to retune block sizes, registers, and shared memory. On Hopper and newer, learn TMA to move multi-dimensional tiles asynchronously instead of manual loops or cp.async. It’s often a free win once you wire it in.

5) Library version drift

Linking against the wrong cuDNN or cuBLAS can nuke performance or even correctness.

The issue: Older libraries may lack kernels for new SMs or have feature toggles that behave differently.

Fix: Upgrade to the library releases validated for your CUDA major and GPU family, then read the support matrix and release notes for caveats. Keep an eye on cuDNN and Transformer Engine notes for Blackwell-specific paths and narrow-precision behaviors.

Need help with cloud migration? Do have a look on this article: Cloud Migration Guide: Strategies, Types & Best Practices

6) Distributed training stalls

Your communication stack needs a refresh too.

The issue: NCCL defaults aren’t always optimal on new interconnects or topologies. Upgrades to NVLink/NVSwitch generations or multi-node configs expose fresh bottlenecks.

Fix: Match NCCL to the platform generation, validate firmware, and test with NCCL tests. NVIDIA publishes tuning guidance and even tuner plugins to pick protocols and CTA counts for you. Bake those settings into your job recipes.

7) Memory pressure and batch sizing

New precision modes tempt you to crank the batch until it tips over.

The issue: FP8 or BF16 lets you fit more, but activation size, optimizer state, and sequence length can still blow up HBM.

Fix: Treat memory like a budget. Enable activation checkpointing and gradient accumulation first, then size batch or context. Add per-step memory metrics to your training loop so you can see real headroom before and after the move. (No special vendor doc needed, just discipline.)

8) MIG and MPS changes

Concurrency tools evolve across generations.

The issue: Teams expect old sharing behavior. MIG and MPS have different trade-offs and per-architecture quirks.

Fix: Use MIG to hard-partition A100/H100/B200 class parts for multi-tenant inference or mixed small jobs. Use MPS to time-slice execution when processes need to share a device without strict memory isolation. Read the current MIG guide for supported layouts and the MPS docs for per-process share controls introduced since CUDA 11.2.

9) IO and storage path assumptions

Faster GPUs starve if your data path is still CPU-bound.

The issue: Old pipelines copy through host memory, burning CPU and capping throughput.

Fix: Where it fits, enable GPUDirect Storage so DMA runs directly between NVMe or parallel filesystems and GPU memory. Start with the official getting-started guide and measure with the provided samples before migrating an entire pipeline.

10) Launch overhead at scale

Newer schedulers and runtimes make tiny kernels look expensive.

The issue: Many small launches waste CPU time and add jitter. This gets louder when the GPU gets faster.

Fix: Use CUDA Graphs to capture and replay steady sequences. Frameworks expose this now, or you can wire it in directly for custom workloads. Graphs cut per-iteration overhead and stabilize latency.

11) Toolchain surprises inside containers

An image that ran for years might suddenly fail on a new base.

The issue: Old compilers, glibc, or host kernels don’t match new drivers and SDKs.

Fix: Standardize on NGC or vendor images for your CUDA major and GPU family. Pin base tags, record driver minimums in the Dockerfile label or README, and rebuild reproducibly whenever you bump toolkits. If you develop on Windows, follow the CUDA install guide exactly per Visual Studio version to avoid build breakage.

12) “It works, but it’s not faster”

This one hurts the most after the purchase order clears.

The issue: You moved, but your code or libraries are not exercising new hardware features.

Fix: Confirm you’re actually using tensor cores at the intended precision. In PyTorch, control TF32 via the documented flag. In training stacks, enable FP8 paths via Transformer Engine where appropriate. Recheck that you compiled with the new SM target so kernels dispatch on the best code path, not a legacy fallback.

A sane migration checklist

Print this and work top to bottom.

• Map target GPUs to compute capability and pick -gencode settings that include native cubins for the new SM plus PTX fallback. Verify with cuobjdump.
• Pick a CUDA major, then confirm driver minimums and library support matrices before you touch production.
• Upgrade cuBLAS/cuDNN/TE to versions that expose TF32 or FP8 on your hardware. Lock accuracy expectations up front.
• Re-profile kernels on the new cards. Use Nsight Compute and consider TMA-based copies on Hopper or newer.
• For multi-GPU, test NCCL throughput and latency, then apply tuner guidance or plugins. Bake results into environment defaults.
• Decide your sharing model. Use MIG for isolation, MPS for cooperative time-slicing. Document who gets what.
• Fix the data path. Pilot GPUDirect Storage on one pipeline and measure.
• Reduce launch overhead. Capture steady loops with CUDA Graphs.

Final thought

New architectures won’t auto-accelerate old habits. If you align drivers, rebuild with the right targets, embrace the precision modes you intend to run, and re-profile the hot spots, you’ll get the speed you paid for without weeks of mystery regressions.