Loss Function
ELI5 — The Vibe Check
The loss function is the AI's score of how badly it's doing. After every prediction, the loss function computes a number: 0 means perfect, high means terrible. The goal of training is to make this number as small as possible. Common choices include cross-entropy for classification and mean squared error for regression. The loss is what gradient descent is trying to minimize.
Real Talk
A loss function (or cost function) quantifies the discrepancy between model predictions and ground truth labels. It must be differentiable with respect to model parameters to enable backpropagation. Common examples include cross-entropy loss (classification), MSE/MAE (regression), and perplexity (language modeling). Choice of loss function directly shapes model behavior.
When You'll Hear This
"The training loss plateaued — try a different optimizer." / "Cross-entropy loss is standard for classification."
Related Terms
Backpropagation
Backpropagation is how errors flow backwards through a neural network during training.
Gradient Descent
Gradient Descent is how an AI learns — it's the algorithm that nudges the model's weights in the right direction after each mistake.
Overfitting
Overfitting is when your model gets TOO good at the training data and becomes useless on new data.
Training
Training is the long, expensive process where an AI learns from data.
Underfitting
Underfitting is the opposite of overfitting — the model hasn't learned enough and does badly on BOTH the training data AND new data.