Migrations
Inventorying a legacy ETL estate without archaeology burnout
Before modernizing twenty years of jobs, you need a map that fits on a wall. A triage method that inventories by consequence instead of by job count.
The Cloud Practice2 min readMigrations
Legacy ETL estates resist inventory by volume: eight hundred jobs, four generations of tooling, comments in the dialect of engineers who left in 2019. Teams that start by cataloging every job burn out at job two hundred with nothing decided. The workable method inventories by consequence, not by count.
Start from the outputs someone would miss. The finance close, the regulatory feed, the tables the current orchestrators actually schedule for consumers who exist. Work backward from those into the jobs that produce them. This typically touches a fraction of the estate and covers most of the consequence.
Sort everything touched into three lanes. Retire: outputs nobody could name a reader for, jobs whose targets have not been queried in two quarters (the logs know). Migrate as-is: correct, boring jobs whose only sin is running on old tooling; wrap them, schedule them on the new platform, refactor never or later. Rebuild: the jobs where business logic, failure rate or opacity make them liabilities. The rebuild lane should be the shortest list, because rebuilds are where estimates go to double.
The untouched remainder gets a tripwire, not a review. Jobs reached by no consequential output stay running with alerting on their consumers. If nothing reads them for another quarter, they join the retire lane with evidence attached. This is how you avoid spending a year reviewing jobs whose correct disposition was silence.
One warning about the people part. The engineer who has kept the legacy estate alive for a decade reads an inventory exercise as an audit of their life's work, and without care the map's most important source goes quiet. Name them the inventory's lead, not its subject: they know which jobs lie, which schedules exist for reasons the documentation never captured, and which "temporary" fix from 2019 is now structural. Every accurate map we have helped produce had that person's fingerprints on it. The ones drawn around them were confident, tidy and wrong.
The deliverable is one wall-sized map: consequential outputs, their lineage, and a lane per job. In our assessments this map is usually the artifact clients keep longest, because every future modernization argument points at it instead of at recollection.