GPU Scheduling
ELI5 - The Vibe Check
GPUs are expensive and scarce, so you can't just let every job grab one whenever it feels like it. GPU scheduling is the traffic cop deciding which training run or inference request gets which GPU, for how long, and what happens when everyone wants one at the same time. Get it wrong and you've got a $30,000 card sitting idle while three teams argue in Slack.
Real Talk
GPU scheduling is the allocation of GPU resources across competing training and inference workloads, handled by orchestrators like Kubernetes device plugins, Slurm, Ray, or cloud-native schedulers such as Kubeflow. Strategies include time-slicing, MIG (Multi-Instance GPU) partitioning, bin-packing by memory footprint, and priority queues, all aimed at maximizing utilization of a resource that is both costly and supply-constrained.
When You'll Hear This
"We moved to MIG partitioning so small inference jobs stop hogging a whole A100." / "The scheduler is queuing training jobs behind higher-priority inference traffic."
Related Terms
Auto Scaling
Auto scaling is when the cloud automatically adds more servers when traffic spikes and removes them when it drops, so you're not paying for idle machines a...
GPU (Graphics Processing Unit)
A GPU was originally built for rendering graphics in games, but turns out it's also perfect for AI.
Inference
Inference is when the AI actually runs and generates output — as opposed to training, which is when it's learning.
Load Balancer
A load balancer is like a traffic cop for servers.
Model Serving
Model serving is the infrastructure that takes a trained AI model and makes it available as a fast, reliable API.