← BACK TO LESSONPARALLEL HORIZONSREAD // 07

LESSON READING // 07

RACK-SCALE INFERENCE

HISTORICAL ANCHOR2024+ · BLACKWELL ERA / NVFP4 / NVL72READ TIME · ABOUT 60 MIN

Combine four-bit quantization, prefill/decode placement, KV-cache routing, and failure recovery into a rack-scale inference system with explicit state boundaries.

Write down answers as you go; the estimate includes code tracing and practice.

PLAY THE INTERACTIVE LESSON
COURSE PACINGABOUT 60 MINUTES
  1. 01ORIENT + OBJECTIVES05 MIN
  2. 02MENTAL MODEL08 MIN
  3. 03DEEP DIVES15 MIN
  4. 04CODE TRACE12 MIN
  5. 05HISTORY + ECOSYSTEM08 MIN
  6. 06PRACTICE + REVIEW12 MIN
ACCELERATED COMPUTING CAPABILITY MAPINDEPENDENT LEARNING COMPANION · NOT AN NVIDIA COURSE OR CERTIFICATION
PATH // 07
AI INFRASTRUCTURE BRIDGEINFERENCE WORKFLOWSNETWORKING AWARENESS
THE OFFICIAL PATH EMPHASIZES

Extend accelerated applications into workflows, networking, and AI infrastructure so analysis covers compute, movement, and orchestration.

HOW THIS LESSON CONNECTS

This systems synthesis goes beyond CUDA basics: NVFP4 addresses representation, prefill/decode addresses phases, and KV routing/migration addresses networked state.

SUGGESTED PREREQUISITES
  • Lessons 03, 05, and 06
  • KV-cache concepts
  • Basic routing and health checks
PRACTICE TOOLS
  • Dynamo or TensorRT-LLM (optional)
  • Nsight Systems
  • Prometheus / request logs
  • NIXL / RDMA (advanced)
01 · 05 MIN

ORIENT + OBJECTIVES

REMEMBER THIS FIRST

Compression needs scales, placement needs phase awareness, and recovery needs a map of state.

BY THE END, YOU SHOULD BE ABLE TO

  1. 01

    Explain NVFP4 E2M1 values, E4M3 microblock scales, and the FP32 tensor scale.

  2. 02

    Compare aggregated and disaggregated prefill/decode serving on benefit, cost, and workload fit.

  3. 03

    Design cache-, load-, and topology-aware routing decisions.

  4. 04

    Separate worker, token, KV-cache, endpoint, and retry state for recovery.

02 · 08 MIN

BUILD THE MENTAL MODEL

  1. 01
    COMPRESS

    NVFP4 stores E2M1 values, shares an E4M3 scale per 16 values, and adds an FP32 tensor scale for global range.

  2. 02
    PLACE

    Prefill processes prompts and creates KV cache; decode uses that cache to emit tokens. Their scaling pressures differ.

  3. 03
    RECOVER

    Worker health, emitted tokens, KV cache, endpoint discovery, and retries are separate state boundaries.

03 · 15 MIN

CONCEPT DEEP DIVES

01

LOW BITS NEED LOCAL RANGE

With only a global scale, outliers squeeze other values. Microblock scales use E2M1's range more effectively.

02

DISAGGREGATION ADDS A KV PATH

Independent prefill/decode scaling requires explicit KV transfer or exposure; benefits must be benchmarked on real traffic.

03

REQUEST STATE IS NOT KV CACHE

Migration can preserve emitted tokens and retry metadata, while KV survival depends on backend and fault type.

PART 01

TWO-LEVEL SCALING SEPARATES LOCAL AND GLOBAL RANGE

E2M1 represents very few values, so a raw cast creates heavy saturation and rounding. NVFP4 uses an E4M3 scale per 16 consecutive values, limiting each outlier's effect to a microblock, then adds an FP32 tensor scale for global range beyond the micro-scales.

Finer scales add metadata, quantization logic, and layout constraints. Quality still depends on distributions, calibration or training, and operator support. Four-bit storage can reduce movement without becoming a validation-free replacement.

PAUSE AND REASONWhy add an FP32 tensor scale beyond the E4M3 scale per 16 values?

REFERENCE ANSWERE4M3 micro-scales have limited range; the FP32 level covers the tensor's global scale and avoids scale overflow.

PART 02

PREFILL AND DECODE HAVE DIFFERENT RESOURCE CURVES

Prefill processes prompt tokens in larger matrix work and creates KV cache. Decode emits small numbers of tokens repeatedly while reading accumulated KV state, making concurrency, output length, and cache capacity central. Aggregated workers are simple, but long prompts can interfere with ongoing decode.

Disaggregation independently sizes the pools and parallelism, but adds KV transfer to the critical path. It wins only when phase isolation and scaling outweigh movement and orchestration. Benchmark real prompt/output distributions against an aggregated baseline.

PAUSE AND REASONIs disaggregation always faster for short prompts, short outputs, and low single-node load?

REFERENCE ANSWERNo. KV transfer and routing add fixed cost, so a simple aggregated worker may be better.

PART 03

RECOVERY STARTS WITH A STATE MAP

A streaming request can include the original prompt, sampling parameters, emitted tokens, random state, KV location, current worker, and retry count. Some state lives at the frontend, some in cache services, and some only in failed VRAM. Recovery must name the source of each item.

Request migration can move emitted-token and request metadata to a healthy worker. KV may survive remotely, require transfer, or need recomputation. Health checks must remove bad endpoints, and retry limits prevent failure storms and duplicate output.

PAUSE AND REASONDo saved emitted tokens guarantee cost-free decode recovery?

REFERENCE ANSWERNo. The new worker still needs matching KV cache; without it, the system must transfer or recompute prefill.

04 · 12 MIN

PUT IT BACK INTO CODE

PROGRAM MODELRoute prefill, carry KV metadata into decode, then migrate safely
01prefill = router.pick_prefill(prompt, cache_overlap)02kv_meta = prefill.run(prompt)03decode = router.pick_decode(load, topology)04stream = decode.resume(kv_meta, emitted_tokens)05if decode.unhealthy():06  migrate(request_state, healthy_worker, retry_limit)

TRACE THE CODE

  1. 01
    PREFILLSELECT BY CACHE OVERLAP + LOAD

    Prefix overlap can save work, but queue depth and topology remain part of estimated completion time.

  2. 02
    KV METAPASS LOCATION AND FORMAT

    Metadata names blocks, endpoints, model version, dtype, and layout so decode can receive compatible state.

  3. 03
    DECODEBALANCE LOAD AND TOPOLOGY

    The emptiest worker may still be poor if the KV path crosses an expensive link.

  4. 04
    MIGRATEMOVE REQUEST STATE WITH A RETRY LIMIT

    Remove the bad endpoint, preserve emitted position, recover KV, and prevent duplicated stream fragments.

REF · 07

KNOWLEDGE ATLAS

OPTIONAL REFERENCE · REVISIT AS NEEDED · OUTSIDE THE CORE 60 MINUTES

CORE VOCABULARY

NVFP4
A 4-bit floating recipe for Blackwell Tensor Core paths, using fine local scales plus a higher-level scale to manage range.
Prefill
Processes the prompt and produces state and KV cache; it is typically more compute intensive.
Decode
Reuses KV cache to generate tokens iteratively and is often constrained by memory capacity, bandwidth, and concurrency.
KV Cache
Stored key/value state per layer that prevents recomputing the full context for every new token.
TTFT / ITL
Time to first token and inter-token latency, capturing startup experience and ongoing generation cadence.
EXAMPLE

WORKED EXAMPLE

  1. 01
    SUBSTITUTE

    2 × 32 × 8 × 128 × 4,096 × 2 = 536,870,912 bytes.

  2. 02
    CONVERT CAPACITY

    That is about 512 MiB per request; eight equal requests need about 4 GiB before overhead.

  3. 03
    BOUND TRANSFER

    At an effective 100 GB/s, the ideal lower bound for 512 MiB is about 5 ms; reality is slower.

RESULT

Context and concurrency quickly amplify KV capacity and movement before fragmentation or metadata enters the count.

WHY IT MATTERS

Before splitting prefill and decode, size the state and use request traces plus an aggregated baseline to prove transfer pays.

DIAGNOSTIC PLAYBOOK

01SYMPTOM

High TTFT while prefill GPUs are idle

INSPECT FIRST
Queueing, routing, cache misses, transfer path, and network fallback
EVIDENCE
A request-ID trace from queue → prefill → transfer → decode
02SYMPTOM

ITL degrades rapidly with concurrency

INSPECT FIRST
Decode batch, KV capacity/bandwidth, fairness, and output length
EVIDENCE
ITL by concurrency/output bucket, KV use, and batch duration
03SYMPTOM

Disaggregated throughput loses to baseline

INSPECT FIRST
KV bytes, fabric bandwidth, cross-domain path, and pool ratio
EVIDENCE
Same-traffic comparison plus the actual transfer protocol and bandwidth

HARDWARE → ECOSYSTEM

  1. 01AGGREGATED INFERENCE

    Prefill, KV state, and decode shared one worker.

    The path was simple but phases could not scale independently.
  2. 02PAGED KV + CACHE-AWARE ROUTING

    KV became block-managed and routing considered locality plus active load.

    Scheduling became central to avoiding repeated prefill and memory waste.
  3. 03RACK-SCALE DISAGGREGATION

    Prefill/decode pools move KV through paths such as NIXL.

    Compute, memory, fabric, discovery, and recovery jointly determine experience.
05 · 08 MIN

HARDWARE + ECOSYSTEM COORDINATE

2024–NOW · BLACKWELL AND RACK-SCALE SYSTEMS

Blackwell introduced FP4 acceleration paths and finer-grained scaling; NVL72 organizes multiple compute trays into a rack-scale NVLink domain. Software such as Dynamo and TensorRT-LLM handles routing, cache movement, parallel configuration, and recovery.

06 · 12 MIN

PRACTICE + REVIEW

A decode worker fails after emitting three tokens. Which two state categories must recovery distinguish?

  1. A. Emitted tokens and KV cache
  2. B. CSS and fonts
  3. C. Block count and CPU core count
REVEAL ANSWER
A

Tokens prevent duplicate output, while KV cache may need rebuilding or retransmission. They are not the same recovery object.

02SCALING

With a few large outliers, which more strongly compresses ordinary small values: one global scale or microblock scales?

HINT

Which values set the whole range?

REFERENCE ANSWER

One global scale. Outliers control every value's range, while microblock scales localize their influence.

03ARCHITECT

Prefill is idle and decode saturated. Does adding prefill workers help?

HINT

Identify the bottleneck phase.

REFERENCE ANSWER

Usually not. Scale or optimize decode, routing, batching, parallelism, or KV capacity instead.

04ROUTE

Worker A has 80% prefix cache but high load; B has no cache and is idle. Always choose A?

HINT

This is a multi-objective estimate.

REFERENCE ANSWER

No. Compare saved prefill work with queue delay and KV/topology cost; route by estimated completion, not one score.

05FAILURE

Decode fails after three tokens. List at least five recovery steps.

HINT

Endpoint, request, token cursor, KV, retry.

REFERENCE ANSWER

Remove the endpoint; freeze request/token cursor; choose a healthy worker; locate, transfer, or rebuild KV; restore sampling/retry state; deduplicate output; fail clearly at the retry limit.

LAB · 50–90 MIN

OPTIONAL HANDS-ON LAB

OUTSIDE THE CORE 60 MINUTES · REQUIRES A SUITABLE CUDA / GPU ENVIRONMENT

LAB GOAL

BUILD AN OBSERVABLE PREFILL→KV→DECODE REQUEST TRACE

Compare aggregated and disaggregated serving across queue, prefill, KV transfer, decode, and recovery state.

PROCEDURE

  1. 01

    Capture aggregated baseline TTFT, ITL, total latency, and KV use by prompt/output length.

  2. 02

    When supported, separate pools and log workers, cache overlap, KV bytes, and transfer time.

  3. 03

    Inject a decode-worker failure and record discovery removal, token cursor, KV handling, and retries.

  4. 04

    Compare short/long prompt and output groups rather than one average.

EVIDENCE OF COMPLETION

Submit one complete trace, an aggregated/disaggregated table, and a recovery-state map; a deterministic simulator is acceptable without serving GPUs.

STRETCH CHALLENGE

Design a routing score combining cache overlap, queue delay, and transfer cost, then test three counterexamples.

OFFICIAL LEARNING ENTRY POINTSDynamo Disaggregated ServingNVIDIA Training Learning Paths · May 2026
REF

GO DEEPER

01
Transformer Engine · FP8 / FP4 PrimerNVFP4 E2M1 and two-level scaling
OFFICIAL SOURCE · OPENS IN A NEW TAB02
NVIDIA Dynamo · Disaggregated ServingPrefill, KV transfer, and decode flow
OFFICIAL SOURCE · OPENS IN A NEW TAB03
NVIDIA Dynamo · Request MigrationWorker failure and partial request recovery
OFFICIAL SOURCE · OPENS IN A NEW TAB