Migrating the transformation layer without stopping the business

Transformation layers accrete: five years of models, three naming conventions, one departed architect. When teams decide to migrate (to a new framework, or onto the current engine generation, or from stored procedures into version control at all), the estate's tangle convinces them to attempt it as one heroic rewrite. Heroic rewrites of transformation layers have a completion rate we would politely call aspirational.

The method that ships: migrate in lineage order, behind a reconciliation gate, retiring as you go.

Lineage order, consumer first. Pick one consequential output (a finance mart, the churn model's features) and migrate its full lineage: sources, staging, marts. One vertical slice proves the pattern, trains the team and delivers a consumer running on the new layer by week two or three. Repeat by slice. Never migrate "all staging models" as a horizontal phase; horizontal phases deliver nothing until they deliver everything.

The reconciliation gate. Old and new layers both build during the overlap; an automated diff compares outputs on every run. A slice cuts over when its diff has been empty for an agreed window, which converts arguments about correctness into a report nobody has to attend. The same parallel-run discipline that governs warehouse cutovers applies one layer up.

Retire in the same commit. The old model is deleted, or at minimum de-scheduled, when its replacement cuts over. Estates where old and new run indefinitely converge on paying twice and trusting neither. The architectural alternatives differ on many things, but every credible one assumes the legacy version actually leaves.

Slice selection decides the schedule, so pick by consumer, not by dependency graph elegance. The first slice should be a mart one vocal team reads daily: small enough to migrate in two weeks, visible enough that its success recruits the next team, and owned by consumers who will actually check the parallel-run report. Deep shared staging layers migrate last, not first, despite every engineer's instinct; they have the most dependents, which makes them the worst place to learn the new engine's quirks and the best place to arrive once the pattern is boring.

The steering-committee version: value ships from week two, risk is bounded per slice, and the estate shrinks monotonically. Rewrites promise none of those three.