← BACK TO LESSONPARALLEL HORIZONSREAD // 05

LESSON READING // 05

RESOURCE ISOLATION

HISTORICAL ANCHOR2020 · AMPERE MIGREAD TIME · ABOUT 60 MIN

When online inference shares a GPU with batch work, establish a hardware boundary first, then balance latency and throughput with bounded batching and asynchronous feeding.

Write down answers as you go; the estimate includes code tracing and practice.

PLAY THE INTERACTIVE LESSON
COURSE PACINGABOUT 60 MINUTES
  1. 01ORIENT + OBJECTIVES05 MIN
  2. 02MENTAL MODEL08 MIN
  3. 03DEEP DIVES15 MIN
  4. 04CODE TRACE12 MIN
  5. 05HISTORY + ECOSYSTEM08 MIN
  6. 06PRACTICE + REVIEW12 MIN
ACCELERATED COMPUTING CAPABILITY MAPINDEPENDENT LEARNING COMPANION · NOT AN NVIDIA COURSE OR CERTIFICATION
PATH // 05
PROFILINGSYSTEM BOTTLENECKSAI INFRASTRUCTURE BRIDGE
THE OFFICIAL PATH EMPHASIZES

Extend profiling from kernels to end-to-end pipelines and locate queueing, contention, and movement bottlenecks.

HOW THIS LESSON CONNECTS

The game separates streams, MIG, and batching; the lab judges resource policy through tail latency, not average throughput alone.

SUGGESTED PREREQUISITES
  • Lesson 02
  • Basic service-latency concepts
  • P50/P95/P99 distinction
PRACTICE TOOLS
  • Nsight Systems
  • Triton Model Analyzer (optional)
  • MIG-capable GPU (optional)
01 · 05 MIN

ORIENT + OBJECTIVES

REMEMBER THIS FIRST

Concurrency expresses independence, isolation creates boundaries, and scheduling controls queues inside those boundaries.

BY THE END, YOU SHOULD BE ABLE TO

  1. 01

    Separate concurrency, priority, resource isolation, and service scheduling.

  2. 02

    Explain which compute and memory paths MIG GPU instances isolate.

  3. 03

    Design maximum batch, queue window, and double buffering from a latency budget.

  4. 04

    Choose observability signals for a shared-GPU experiment.

02 · 08 MIN

BUILD THE MENTAL MODEL

  1. 01
    CONTEND

    Two streams submit independently but still share SMs, cache, and memory bandwidth.

  2. 02
    ISOLATE

    MIG creates GPU instances from supported profiles, bounding compute and memory paths.

  3. 03
    FEED

    Inside an instance, bound batch size and wait time, then use buffers to reduce copy/compute bubbles.

03 · 15 MIN

CONCEPT DEEP DIVES

01

A STREAM IS NOT QOS

Stream priority is a scheduling hint, not a hard reservation of SMs, L2, or memory bandwidth.

02

MIG USES DISCRETE PROFILES

GPU instances come from hardware-supported combinations, not arbitrary percentage sliders.

03

ISOLATION STILL LEAVES A QUEUE

Dynamic batching trades among maximum batch, queue delay, request priority, and latency objectives.

PART 01

INDEPENDENT SUBMISSION DOES NOT CREATE A RESOURCE BOUNDARY

CUDA streams give workloads independent command flows, yet kernels still compete for SMs, registers, L2, and memory bandwidth. Stream priority can influence pending work, but it does not reserve a fixed share or necessarily preempt a running large kernel.

Start with the objective: security separation, fault containment, or predictable interference? Different goals may call for processes, MPS, time slicing, MIG, or separate GPUs. One mechanism's guarantee cannot be projected onto every layer.

PAUSE AND REASONCan a high-priority stream guarantee completion within 5 ms?

REFERENCE ANSWERNo. Priority is a scheduling hint; running kernels, contention, queueing, and model time still govern latency.

PART 02

MIG MAKES PROFILE-BASED INSTANCE BOUNDARIES

On supported GPUs, MIG combines compute and memory slices into GPU instances. Each instance receives isolated address and memory-system paths, reducing interference from another instance's cache or bandwidth behavior and defining clearer QoS and fault boundaries.

Profiles are discrete hardware-supported combinations. Each smaller instance has less capacity and compute, so model fit and overload must be revalidated. MIG also does not replace container, identity, or application authorization controls.

PAUSE AND REASONCan MIG split a GPU into arbitrary 37% and 63% instances?

REFERENCE ANSWERNo. Configurations must use supported, layout-compatible profiles.

PART 03

BATCHING SPENDS LATENCY BUDGET TO BUY THROUGHPUT

Dynamic batching shares one execution across requests and can improve matrix shapes, but the first request waits for the batch to form. Maximum queue delay directly consumes end-to-end latency budget; derive it from the target P95/P99 rather than maximizing batch size.

Double buffering overlaps input movement for one slot with compute in another, but cannot remove model compute. Highly variable request shapes may need separate queues, ragged batching, or priority policy to prevent one request from dragging the batch.

PAUSE AND REASONWhy is a larger maximum batch not always better?

REFERENCE ANSWERIt may increase throughput but also queue time, memory use, and one-batch execution time, worsening tail latency.

04 · 12 MIN

PUT IT BACK INTO CODE

PROGRAM MODELBound queue time, then form microbatches and feed asynchronously
01while (serving) {02  batch = queue.take(max_batch, max_queue_delay);03  copy_async(slot[next], batch);04  infer_async(slot[current]);05  swap(current, next);06}

TRACE THE CODE

  1. 01
    BUDGETDEFINE max_queue_delay FIRST

    Queue time is one piece of end-to-end latency; leave budget for network, preprocessing, execution, and response.

  2. 02
    FORMDISPATCH ON SIZE OR TIME

    A batch leaves when maximum size or oldest-request wait is reached.

  3. 03
    BUFFERSEPARATE COPY AND EXECUTION SLOTS

    One slot receives input while the other executes; completion must precede reuse.

  4. 04
    MEASURETRACK THROUGHPUT AND TAILS

    Average latency hides queue spikes. Observe queue, copy, compute, and P95/P99 by request class.

REF · 05

KNOWLEDGE ATLAS

OPTIONAL REFERENCE · REVISIT AS NEEDED · OUTSIDE THE CORE 60 MINUTES

CORE VOCABULARY

Throughput
Requests or samples completed per unit time; it does not reveal how long one request waited.
Tail Latency
High-percentile latency such as P95/P99, exposing slow requests and contention.
Queue Delay
Time waiting for a batch, stream, memory, or execution resource.
Dynamic Batching
Combining requests within a bounded waiting window to improve GPU utilization.
MIG
Hardware partitioning, introduced with Ampere, that assigns isolated compute and memory-system resources to instances.
EXAMPLE

WORKED EXAMPLE

  1. 01
    NORMAL LOAD

    L = λW = 80 × 0.10 = 8, so eight requests are in flight on average.

  2. 02
    CONTENTION

    If W rises to 0.25 s at the same arrival rate, L = 80 × 0.25 = 20.

  3. 03
    INTERPRET IT

    Twelve additional requests now wait or execute, increasing queue and memory pressure.

RESULT

At equal throughput, raising mean latency from 100 ms to 250 ms raises average in-flight work from 8 to 20.

WHY IT MATTERS

Little’s Law connects averages only. Record every request and inspect P95/P99 to understand experience.

DIAGNOSTIC PLAYBOOK

01SYMPTOM

Mean latency is fine; P99 explodes

INSPECT FIRST
Long batches, head-of-line blocking, background jobs, reclamation, and bursts
EVIDENCE
Per-request queue/copy/compute spans and the latency distribution
02SYMPTOM

Throughput drops after enabling MIG

INSPECT FIRST
Whether the profile fits the working set and whether one job needs full-GPU resources
EVIDENCE
Per-instance utilization, memory, bandwidth, and QoS under identical traffic
03SYMPTOM

A larger batch does not raise utilization

INSPECT FIRST
Pre/post-processing, movement, shape variation, memory, and CPU supply
EVIDENCE
An end-to-end timeline and stage queue lengths—not one GPU percentage

HARDWARE → ECOSYSTEM

  1. 01SINGLE-JOB OWNERSHIP

    Batch jobs optimized full-GPU throughput.

    Utilization was simple, but online multi-tenancy was weak.
  2. 02STREAM / MPS SHARING

    Multiple workflows submitted concurrently.

    Utilization rose alongside cache, bandwidth, and tail interference.
  3. 03MIG + SERVICE ORCHESTRATION

    Hardware isolation joined batching, scheduling, monitoring, and containers.

    The problem moved from kernels to QoS and capacity planning.
05 · 08 MIN

HARDWARE + ECOSYSTEM COORDINATE

2020 · AMPERE MIG

MIG introduced hardware-supported GPU instances on Ampere data-center GPUs, giving clients isolated compute and memory paths. Service software such as Triton made batching, queue policy, and model placement system-design concerns.

06 · 12 MIN

PRACTICE + REVIEW

Can two CUDA streams guarantee that a large batch will not interfere with an online request?

  1. A. Yes, each stream reserves half the SMs
  2. B. No, they can still compete for one resource pool
  3. C. Yes, if host priority is raised
REVEAL ANSWER
B

Streams enable potential concurrency but do not establish hard resource boundaries. Isolation and queue scheduling live at different layers.

02DISTINGUISH

Do separate processes, streams, and MIG instances provide the same isolation?

HINT

Compare address space, scheduling, and hardware resources.

REFERENCE ANSWER

No. Processes separate software address spaces, streams order commands, and MIG allocates isolated compute/memory paths. Platform details still matter.

03BUDGET

P99 target is 50 ms: network/preprocess 8, model 30, response 4. What queue budget remains?

HINT

Subtract, then reserve jitter margin.

REFERENCE ANSWER

A static 8 ms remains, but max_queue_delay should be below 8 ms to leave margin and must be load-tested.

04DEBUG

Double buffering overwrites slot[next] while the prior batch still uses it. Fix it.

HINT

Each slot needs a completion signal.

REFERENCE ANSWER

Associate an event/future with each slot and wait for that slot's copy and compute before reuse.

05EXPERIMENT

How would you test whether MIG stabilizes online latency?

HINT

Build shared versus isolated controls.

REFERENCE ANSWER

Hold model, traffic, and background batch load constant; compare P50/P95/P99, throughput, errors, memory, and bandwidth while recording profiles and intra-instance scheduling.

LAB · 35–50 MIN

OPTIONAL HANDS-ON LAB

OUTSIDE THE CORE 60 MINUTES · REQUIRES A SUITABLE CUDA / GPU ENVIRONMENT

LAB GOAL

EVALUATE SHARED GPU + BOUNDED BATCHING WITH TAIL LATENCY

Compare shared resources, bounded microbatching, and MIG when available under fixed traffic.

PROCEDURE

  1. 01

    Generate repeatable online requests and background batch load.

  2. 02

    Capture shared-GPU throughput, P50/P95/P99, and a timeline.

  3. 03

    Add max_batch, max_queue_delay, and double buffering under identical traffic.

  4. 04

    Repeat with an explicit MIG profile only when supported.

EVIDENCE OF COMPLETION

Submit throughput/tail-latency results, a timeline, and a statement separating isolation from scheduling.

STRETCH CHALLENGE

Use separate long- and short-request queues to study head-of-line blocking and fairness.

OFFICIAL LEARNING ENTRY POINTSNsight Systems DocumentationMIG User Guide
REF

GO DEEPER

01
Multi-Instance GPU User Guide · IntroductionMIG isolation and QoS boundaries
OFFICIAL SOURCE · OPENS IN A NEW TAB02
Triton Inference Server · Dynamic BatcherMaximum batch, queue delay, and scheduling
OFFICIAL SOURCE · OPENS IN A NEW TAB