Tensor Parallelism

ELI5 - The Vibe Check

Some models are too big to fit on a single GPU, full stop, no amount of quantization saves you. Tensor parallelism solves this by slicing individual weight matrices into pieces and spreading them across multiple GPUs, so each one holds only a fraction of every layer and they constantly compare notes mid-computation. It's less like giving each GPU its own task and more like several people jointly holding one enormous whiteboard.

Real Talk

Tensor parallelism splits individual weight matrices within each transformer layer across multiple GPUs, requiring high-bandwidth interconnects (like NVLink) since every forward pass needs frequent all-reduce communication between devices. It differs from pipeline parallelism, which assigns whole layers to different GPUs, and from data parallelism, which replicates the full model and splits the batch. Frameworks like Megatron-LM and DeepSpeed implement it for training and serving models too large for a single device.

When You'll Hear This

"The 70B model needs tensor parallelism across 4 GPUs just to load the weights." / "Without NVLink, tensor parallelism communication overhead kills your throughput."