Schema Evolution

ELI5 - The Vibe Check

Your data's shape never actually stays still: new fields show up, old ones get renamed, types change, and yet you've got old data sitting around in the old shape while new data arrives in the new shape. Schema evolution is the set of rules and tooling that lets both versions coexist peacefully instead of every change becoming a breaking migration that takes the whole pipeline down.

Real Talk

Schema evolution is the ability to change a data schema, adding, removing, or modifying fields, while remaining compatible with data written under previous versions of that schema. Formats like Avro and Protocol Buffers, and table formats like Iceberg, support this natively through compatibility rules (backward, forward, or full compatibility), so producers and consumers don't need to be upgraded in lockstep.

Show Me The Code

-- Backward-compatible schema change: add a nullable column with a default
ALTER TABLE orders ADD COLUMN discount_code TEXT DEFAULT NULL;

When You'll Hear This

"The schema registry blocked the deploy because the change wasn't backward compatible." / "Iceberg's schema evolution let us add a column without rewriting every existing file."