Training-Serving Skew
ELI5 - The Vibe Check
Training-serving skew is when the code that prepared your data during training doesn't quite match the code that prepares data when the model is actually running live, so the model gets subtly different inputs than it learned on. Maybe training computed an average over 30 days but the live system computes it over 7, and now the model is confidently wrong in a way that's infuriating to debug because nothing technically broke.
Real Talk
Training-serving skew occurs when there's a discrepancy between how features are computed or preprocessed during model training versus during live inference, causing the model to receive inputs that don't match the distribution it was trained on. It's commonly caused by duplicated feature logic living in two separate codebases, and is a major reason production teams invest in feature stores that guarantee the exact same transformation code runs in both training and serving.
When You'll Hear This
"The model tanked in production despite great offline metrics, classic training-serving skew, the feature pipelines didn't match." / "We moved feature computation into the feature store specifically to kill training-serving skew for good."
Related Terms
Data Drift
Data drift is when the real world quietly stops looking like the world your model was trained on.
Feature Engineering
Feature engineering is the unglamorous, hugely important work of turning raw, messy real-world data into the specific columns a model can actually learn fr...
Machine Learning (ML)
Machine Learning is teaching a computer by showing it thousands of examples instead of writing out every rule.
Model Monitoring
Model monitoring is refusing to treat we deployed the model as the finish line.