Gradient Boosting

ELI5 - The Vibe Check

Gradient boosting builds a model the way you'd improve a rough first draft: you write it, look specifically at what it got wrong, then write a second pass that focuses on fixing exactly those mistakes, then a third pass fixing what's still wrong, and so on. Each new tiny model is trained specifically to correct the errors of everything that came before it, and the whole chain added together becomes surprisingly accurate.

Real Talk

Gradient boosting builds an ensemble of weak learners, typically shallow decision trees, sequentially, where each new tree is trained to predict the residual errors of the combined ensemble so far, using gradient descent to minimize a loss function. Unlike random forests, which build trees independently and average them, boosting builds trees dependently, each one correcting the last, which tends to produce higher accuracy but is more prone to overfitting without careful tuning.

When You'll Hear This

"Gradient boosting squeezed out another 3 points of accuracy over the random forest, but it took way more tuning to get there." / "Watch the validation loss closely with gradient boosting, it'll happily overfit if you let it run too many rounds."