Training and inference: Two faces of AI compute

There are two major workloads that make up AI compute: training and inference. Training looks back. It digests a frozen body of past text, images and code, and distills it into weights. Inference looks forward. It takes those weights and, one token at a time, produces something that did not exist a moment ago.

Same model, same matrices, same multiply-accumulate at the bottom of it all. Yet the two faces ask the silicon to perform nearly opposite tasks.

Training wants arithmetic, and lots of it. Inference wants bytes delivered on time. Treating them as one market has cost the industry a decade of misplaced benchmarks.

Let’s look at each separately, then at what each requires of the processor running it.

The face that looks back: Training

Training is a loop repeated trillions of times, and every iteration has three stages:

1. Forward pass. A batch of token sequences flows through the neural network. Every layer is a large matrix-matrix multiply (GEMM): each accelerator (a single GPU such as an NVIDIA B300) processes thousands of tokens in parallel, and across the cluster a single training step covers millions. The intermediate results, the activations needed to calculate gradients later, must be kept.
2. Backward pass. The model’s predictions are compared with the true next tokens, yielding a single error score, the loss. Its gradient is then propagated back through the same layers. Each layer computes two more GEMMs: one for the gradient with respect to its input, the other for the gradient with respect to its weights. That is why the backward pass costs roughly twice the forward pass.
3. Optimizer step. Gradients from every replica of the model are averaged across the cluster, then the optimizer, typically Adaptive Moment Estimation (Adam) or a variant such as AdamW, updates every weight.

The arithmetic works out to about six 6 FLOPs per parameter per training token: two forward, four backward. Multiply by tens of billions of parameters and tens of trillions of tokens, and frontier training runs land in the range of 10²⁵ to 10²⁶ FLOPs.

Memory is the second issue. In one common mixed-precision Adam configuration, each parameter can require about 16 bytes of state: a BF16 weight and gradient (2 bytes each), an FP32 master weight (4), and two FP32 optimizer moments (4 each). A 70-billion-parameter model therefore needs about 1.1 terabyte (TB) before a single activation is stored. No single device holds that. Training is, by construction, a distributed problem.

A key feature of training is batching. Processing many tokens together lets the processor reuse model weights across a large amount of computation, raising arithmetic intensity—the number of FLOPs performed per byte moved. The large matrix operations that dominate much of training can therefore make effective use of compute throughput. Adding chips can increase training capacity and speed, but the gains depend on how quickly they exchange data and synchronize their work.

The face that looks forward: Inference

Inference omits the backward pass and the optimizer. For the main dense linear operations, a forward pass takes roughly two FLOPs per parameter per token, a third of training. That makes it sound like a lighter version of the same job. It is not. Look closer and inference has two faces of its own.

Prefill processes the prompt. All input tokens are known in advance, so they move through the network in parallel, as GEMMs. A sufficiently large prompt or batch can make its matrix operations compute intensive, making prefill resemble a forward training pass: compute-bound, high arithmetic intensity. Along the way, the model writes the keys and values of every token, in every layer, into the key-value (KV) cache that later tokens will use for attention. Prefill strongly affects time to first token.

Decode generates the answer, one token at a time. Each new token cannot be selected until the preceding step is complete, so there is nothing to parallelize along the sequence. Every layer collapses into a matrix-vector multiply (GEMV), reusing weights across relatively little work. To produce one token, the processor must stream all the weights and the entire KV cache for that sequence out of memory and start over.

Weight and cache traffic then dominate the time spent generating each token.

That is the crux. A current flagship accelerator can perform several hundred FLOPs in the time it moves one byte from high-bandwidth memory (HBM). Decode at small batch sizes supplies only one or two FLOPs per byte: each weight is fetched, used in a single multiply-add, and discarded. Meanwhile, the compute units are idle, waiting on memory.

Decode is memory-bandwidth bound, and in agentic workloads with long outputs, decode can reach around 90% of wall-clock time. At larger batches, weight reuse improves and the balance can shift, though not for the KV cache, which each sequence reads in full. Long generated answers can make decode dominate a request’s end-to-end time, but the share depends on prompt length, output length, batch size, and serving system. See block diagram below.

In a large language model (LLM) inference workflow, prefill processes the prompt in parallel and writes the KV cache in one pass; decode generates one token at a time, reading the full cache on every step. The time split is illustrative. Source: VSORA

The KV cache adds another constraint as context grows. In each layer, every cached token contributes a key and a value for each KV head. Assuming the full context remains resident at a fixed precision, the cache size per sequence is:

KV bytes = 2 x L x H_kv x d_head x S x b

Whereas L represents layers, H_kv key-value heads, d_head head dimension, S sequence length, and b bytes per element. For a Llama-3-70B-class model (80 layers, 8 KV heads, head dimension 128, BF16) that is about 320 KB per token.

At a 128k-token context, one conversation holds roughly 42 GB of cache, more than half a model’s worth of weights at FP8, for a single user.

Three workloads, side by side

Prefill sits closer to training than to decode. The real divide runs between batched, parallel math and sequential, memory-starved math, as shown in the table below.

What training asks of the silicon

A training processor is judged by how much of its rated arithmetic it turns into useful work across an entire cluster. Five pressures shape it:

1. Dense matrix throughput. Tensor or systolic arrays sized for large GEMMs fed from on-chip SRAM with enough reuse to stay busy. This is the ground where peak TFLOPS on the datasheet actually mean something.
2. Memory capacity, not only bandwidth. Weights, gradients, optimizer states, and activations must sit close to the compute. HBM stacks per package keep growing for this reason, and activation recomputation trades FLOPs for bytes when they don’t fit.
3. Every step ends with a gradient all-reduce, and tensor or pipeline parallelism adds traffic inside each layer. Scale-up links (NVLink-class) and scale-out fabrics (InfiniBand or Ethernet) often decide utilization more than the cores do. A cluster that spends 40% of each step waiting on collectives has thrown away 40% of its silicon.
4. Gradients span a huge dynamic range. BF16 won because it keeps FP32’s 8-bit exponent; FP16 needed loss scaling to avoid underflow. FP8 training works, but only with per-tensor or per-block scaling and a high-precision master copy of the weights.
5. Reliability at scale. A run on tens of thousands of chips for weeks will see failures. So, checkpointing, fast restart, and silent-data-corruption detection become architectural features, not operational afterthoughts.

Training has no user waiting for the next token, but time still matters: a stalled step delays the entire run. Sustained throughput, utilization, and time to completion are the measures that count.

What inference asks of the silicon

An inference processor is judged by cost and energy per token delivered within a latency target. That changes the priorities almost completely.

1. Bandwidth per FLOP. Decode throughput is set by how fast weights and KV cache reach the arithmetic units. A chip with half the TFLOPS and twice the effective bandwidth can win outright. This is the memory wall, and it’s where rated peak and sustained performance part ways.
2. KV cache capacity and management. The cache, not the model, becomes the binding constraint on how many users a chip serves at long context. Paging, compression, quantizing the cache itself, and offloading to cheaper tiers are now first-order design problems.
3. The batching dilemma. Batching many users together restores arithmetic intensity, because weights are fetched once and reused across requests. But every user in the batch waits for the slowest step. Operators trade throughput against per-user latency, and the hardware must make that trade cheap: fine-grained scheduling, continuous batching, and fast context switches.
4. Aggressive precision. Inference tolerates far lower precision than training. FP8 is routine, FP4 and INT4 weights are increasingly common, and each halving of bits nearly doubles effective bandwidth. The processor must execute these formats natively, with scaling factors handled in hardware.
5. Energy and TCO. Training is a capital expense paid once per model. Inference is an operating expense paid on every token, for the life of the product. Watts per token, not peak watts, determine which company makes money.
6. Utilization and predictability. Request lengths vary wildly, and traffic is bursty. Deterministic, well-scheduled architectures keep the pipeline full; architectures tuned for large uniform GEMMs spend much of decode underused.

Prefill complicates the picture: it’s compute-bound, like training, and it sets time to first token. A good inference chip must handle both regimes, even if decode dominates the bill.

Can one chip wear both faces?

The GPU’s answer has been yes. It’s a superb training engine, and its software ecosystem made it the default for inference too. But a die sized for training GEMMs carries a large compute budget that decode cannot use. At batch sizes that keep latency acceptable, much of that silicon sits idle while HBM does the real work. You Engineers pay for both faces and use one.

The industry is responding on three fronts:

1. Specialized inference silicon. Designs that raise bandwidth per FLOP by putting far more memory close to compute: SRAM-heavy dataflow chips, wafer-scale parts, multi-tier memory hierarchies, and architectures built around sustained rather than peak efficiency. Each trades differently between capacity and bandwidth, and each hits a different cliff when models or contexts outgrow their fast memory.
2. Disaggregated serving. Split prefill and decode onto different pools of hardware, each sized for its own bottleneck, and ship the KV cache between them. This admits openly that inference itself is two workloads.
3. Software that closes the gap. Speculative decoding, which lets a small model draft tokens that the big one verifies in a single batched pass, turns part of decode back into prefill-like math. Quantization and cache compression shrink the bytes to move.

None of this makes the GPU obsolete for training. It does end the assumption that the best training chip is automatically the best inference chip.

Choosing which face to serve

For 10 years, AI hardware was built for the face that looks back. Training was where the prestige was, and peak TFLOPS was the number on the slide. That made sense when models were trained often and served rarely.

The ratio has flipped. A frontier model is trained once and then queried billions of times, increasingly by agents that read long contexts and think out loud before answering. The economics of AI now live in the forward-looking face, and that face is starved for bytes, not FLOPs.

The challenge for processor architects is to deliver tokens economically within a latency target while still handling the bursts of parallel computation that begin each request.

Lauro Rizzatti is a business development executive with VSORA, a technology company offering silicon semiconductor solutions that redefine performance. He is a noted chip design verification consultant and industry expert on hardware emulation.

Related Content

• Chiplets Are The New Baseline for AI Inference Chips
• The next AI frontier: AI inference for less than $0.002 per query
• Purpose-built AI inference architecture: Reengineering compute design
• As AI Moves from Training to Inference, Optics Moves Closer to the Chip
• The truth about AI inference costs: Why cost-per-token isn’t what it seems

The post Training and inference: Two faces of AI compute appeared first on EDN.