← BACK TO LESSONPARALLEL HORIZONSREAD // 03

LESSON READING // 03

MULTI-GPU SCALING

HISTORICAL ANCHORMULTI-GPU → P100 / NVLINK / DGX-1READ TIME · ABOUT 60 MIN

When one GPU cannot hold the workload, explicitly decide what each GPU owns, what it computes locally, and when results must be exchanged.

Write down answers as you go; the estimate includes code tracing and practice.

PLAY THE INTERACTIVE LESSON
COURSE PACINGABOUT 60 MINUTES
  1. 01ORIENT + OBJECTIVES05 MIN
  2. 02MENTAL MODEL08 MIN
  3. 03DEEP DIVES15 MIN
  4. 04CODE TRACE12 MIN
  5. 05HISTORY + ECOSYSTEM08 MIN
  6. 06PRACTICE + REVIEW12 MIN
ACCELERATED COMPUTING CAPABILITY MAPINDEPENDENT LEARNING COMPANION · NOT AN NVIDIA COURSE OR CERTIFICATION
PATH // 03
SCALINGMULTI-GPUGPU COMMUNICATION
THE OFFICIAL PATH EMPHASIZES

Scale a single-GPU application across devices while including partitioning, communication, and topology in one performance model.

HOW THIS LESSON CONNECTS

The game establishes capacity, shard ownership, and collective semantics; the lab exposes ranks, topology, and All-Reduce time.

SUGGESTED PREREQUISITES
  • Lessons 00–02
  • One working single-GPU kernel
  • Basic distributed-rank concepts
PRACTICE TOOLS
  • NCCL
  • nvidia-smi topo -m
  • Nsight Systems
  • MPI or torchrun
01 · 05 MIN

ORIENT + OBJECTIVES

REMEMBER THIS FIRST

Multi-GPU programming is about ownership, balance, and necessary communication—not the number of cards.

BY THE END, YOU SHOULD BE ABLE TO

  1. 01

    Distinguish replication, data sharding, model sharding, and memory oversubscription.

  2. 02

    Define explicit, balanced ownership for a multi-GPU workload.

  3. 03

    Select All-Reduce, Reduce, All-Gather, or related semantics from consumer needs.

  4. 04

    Analyze scaling through compute, communication, topology, and stragglers.

02 · 08 MIN

BUILD THE MENTAL MODEL

  1. 01
    SHARD

    Split model or data into similarly sized pieces that fit each GPU.

  2. 02
    LOCAL

    Each rank computes only on its shard, extending independent work as far as correctness allows.

  3. 03
    COMMUNICATE

    Use a collective only when the next step truly requires global state.

03 · 15 MIN

CONCEPT DEEP DIVES

01

MEMORY DOES NOT MERGE ITSELF

Four 16 GB GPUs are not one transparent 64 GB allocation. The program still defines shards, reachability, and communication.

02

THE SLOWEST RANK SETS THE BARRIER

Even if every shard fits, uneven compute or communication creates a straggler at synchronization points.

03

ALL-REDUCE IS A SEMANTIC

It reduces inputs from every rank and returns the same result to every rank; the library may choose the concrete algorithm and topology.

PART 01

CAPACITY IS FIRST AN OWNERSHIP PROBLEM

If 24 GB of state cannot fit in 16 GB, more threads or streams cannot help. A multi-GPU design must say which parameters, activations, or samples each rank owns, what remains replicated, and who updates it. Only explicit ownership proves per-GPU memory dropped.

Standard data parallelism replicates a model while splitting batch, so it does not necessarily solve an oversized model. Tensor and pipeline parallelism shard width or depth. Unified memory supplies addressing and migration, not a free fusion of every GPU's memory.

PAUSE AND REASONCan ordinary data parallelism fit a model larger than one GPU?

REFERENCE ANSWERUsually not: every rank keeps the full model. Model-state sharding or tensor/pipeline parallelism is needed.

PART 02

COLLECTIVE CHOICE FOLLOWS CONSUMER NEEDS

Work backward from the next step. Use Reduce when only a root needs the sum, All-Reduce when all ranks need it, and All-Gather when every rank needs all shards concatenated. The wrong semantic moves unnecessary data or computes the wrong operation.

NCCL collectives enqueue on CUDA streams. Buffer lifetime, dependencies, communicator membership, and call ordering across ranks must align. Otherwise consumers may read incomplete data or ranks may wait indefinitely.

PAUSE AND REASONAll ranks need only one reduced shard each. What may fit better than All-Reduce?

REFERENCE ANSWERReduce-Scatter, which reduces and distributes distinct result shards without replicating the full output.

PART 03

THE SLOWEST PATH SETS SCALING EFFICIENCY

Four GPUs might ideally quarter local compute, but real time includes communication, synchronization, launch cost, and imbalance. With 12/6/4/2 GB shards, every rank waits for the 12 GB straggler at a collective.

Topology changes paths across NVLink, PCIe switches, and nodes. Libraries can be topology-aware, but applications still choose groups, shard shapes, and communication frequency. Strong scaling eventually makes local work too small relative to coordination.

PAUSE AND REASONWhy can adding a fifth underutilized GPU make the job slower?

REFERENCE ANSWERIt adds coordination and communication while shrinking local work; overhead can exceed saved compute.

04 · 12 MIN

PUT IT BACK INTO CODE

PROGRAM MODELCompute local gradients, then reduce only when required
01local = shard(dataset, rank, world_size);02grad = backward(local);03ncclAllReduce(grad, grad, count, ncclFloat,04              ncclSum, communicator, stream);05optimizer.step(grad);

TRACE THE CODE

  1. 01
    SHARDEACH RANK LOADS ITS OWN DATA

    Sharding should be deterministic, intentionally disjoint or replicated, and balanced by actual cost.

  2. 02
    LOCALEXTEND THE COMMUNICATION-FREE REGION

    Local backward work touches rank-owned state before global coordination.

  3. 03
    REDUCEMATCH GLOBAL SEMANTICS

    Every replicated optimizer needs the same gradient sum, so All-Reduce matches this example.

  4. 04
    STREAMCOMMUNICATION HAS A DEPENDENCY BOUNDARY

    The optimizer reads only after completion, enforced through stream order or events.

REF · 03

KNOWLEDGE ATLAS

OPTIONAL REFERENCE · REVISIT AS NEEDED · OUTSIDE THE CORE 60 MINUTES

CORE VOCABULARY

Rank
A logical distributed participant, often bound to one GPU; a rank number is not inherently a physical device number.
Shard
The data, parameters, or activations owned by a rank under an explicit partitioning rule.
Collective
A communication operation called in matching order by all ranks, such as All-Reduce or All-Gather.
Topology
The real GPU connection graph across PCIe, NVLink, switches, or networks.
Straggler
The last rank to reach a synchronization point because of imbalance, path differences, or system noise.
EXAMPLE

WORKED EXAMPLE

  1. 01
    CHECK CAPACITY

    Both 14/10 and 12/12 stay below 16 GB per GPU, so both fit.

  2. 02
    CHECK BALANCE

    A 14/10 split gives one rank more work; 12/12 better matches a synchronous workload.

  3. 03
    ADD COMMUNICATION

    If two-GPU compute takes 200 ms plus 50 ms communication, total time is 250 ms and speedup is 1.6×.

RESULT

Scaling efficiency is 400 ÷ (2 × 250) = 80%, short of the ideal 2× speedup.

WHY IT MATTERS

Fitting in memory is only the first test; balance, collectives, and topology determine whether scaling pays.

DIAGNOSTIC PLAYBOOK

01SYMPTOM

More GPUs produce little speedup

INSPECT FIRST
Work/rank, collective share, message size, and topology
EVIDENCE
Per-rank timelines, communication bandwidth, and scaling curves
02SYMPTOM

A collective hangs

INSPECT FIRST
Matching call order, count/type agreement, and failed ranks
EVIDENCE
Per-rank logs, NCCL debug output, and last successful collective ID
03SYMPTOM

The same GPU always delays the join

INSPECT FIRST
Shard size, input skew, throttling, NUMA, and link differences
EVIDENCE
Per-rank compute/communication distributions—not a global mean

HARDWARE → ECOSYSTEM

  1. 01PCIe MULTI-GPU

    Applications managed devices, peer access, and copies.

    Capacity could split, but communication became a bottleneck.
  2. 02NVLINK + SWITCH FABRICS

    GPU bandwidth and topology choices expanded.

    Parallel strategy had to understand physical adjacency.
  3. 03NCCL + FRAMEWORK PARALLELISM

    Libraries encapsulated collective algorithms and topology discovery.

    Developers select semantics and shards, then profile the real cost.
05 · 08 MIN

HARDWARE + ECOSYSTEM COORDINATE

2016 · PASCAL P100 / NVLINK

P100 brought HBM2 and first-generation NVLink into data-center GPU systems, changing the cost of local and GPU-to-GPU movement. NCCL made collectives such as All-Reduce reusable software primitives. Faster links did not erase topology.

06 · 12 MIN

PRACTICE + REVIEW

All four ranks need the summed gradient before updating. Which semantic fits directly?

  1. A. Keep only local gradients
  2. B. All-Reduce
  3. C. Replicate the full dataset and never communicate
REVEAL ANSWER
B

All-Reduce combines reduction and distribution so every participating rank receives the same global result.

02CAPACITY

Why does full replication of a 24 GB model fail on four 16 GB GPUs despite 64 GB total?

HINT

How much does each GPU own?

REFERENCE ANSWER

Each GPU still needs 24 GB. Aggregate capacity matters only after software shards ownership.

03BALANCE

Shards are 12/6/4/2 GB and all fit. What is the main synchronous-training risk?

HINT

When can the collective begin?

REFERENCE ANSWER

The 12 GB rank becomes a straggler while others wait. Rebalance by actual compute cost.

04SEMANTIC

Ranks hold distinct feature slices and each needs the full concatenated sequence. Which collective?

HINT

This is concatenation, not summation.

REFERENCE ANSWER

All-Gather, or a variable-size variant after exchanging lengths. All-Reduce would apply the wrong operation.

05ANALYZE

Efficiency drops from 8 to 16 GPUs. What evidence should you collect?

HINT

Separate compute, communication, waiting, and topology.

REFERENCE ANSWER

Compare local kernel time, collective time, rank arrival skew, message sizes, topology, group layout, and shard balance.

LAB · 40–60 MIN

OPTIONAL HANDS-ON LAB

OUTSIDE THE CORE 60 MINUTES · REQUIRES A SUITABLE CUDA / GPU ENVIRONMENT

LAB GOAL

VERIFY OWNERSHIP AND ALL-REDUCE ON TWO GPUS

Give ranks distinct input and produce the same global result while explaining path and synchronization cost.

PROCEDURE

  1. 01

    Print rank, device, local input, and local result.

  2. 02

    Run NCCL All-Reduce and validate every rank.

  3. 03

    Record nvidia-smi topo -m and identify the link domain.

  4. 04

    Sweep message sizes and separate compute, collective, and waiting.

EVIDENCE OF COMPLETION

Submit rank output, topology summary, and at least three message sizes with the current scaling limit.

STRETCH CHALLENGE

Replace All-Reduce with Reduce-Scatter + All-Gather and compare semantics, memory, and time.

OFFICIAL LEARNING ENTRY POINTSNCCL User GuideNCCL Tests
REF

GO DEEPER

01
NCCL User Guide · Collective OperationsSemantics of All-Reduce and other collectives
OFFICIAL SOURCE · OPENS IN A NEW TAB02
NVIDIA Pascal Architecture In-DepthP100, HBM2, and NVLink context
OFFICIAL SOURCE · OPENS IN A NEW TAB