A Database Migration Rollback That Went Wrong (and the Process We Built After)

The migration itself wasn't the problem. It was a normal one — adding a status column to a subscriptions table, deployed, tested, working fine for two weeks. The problem was rolling it back three weeks later for an unrelated reason, without first checking what had been built on top of that column in the meantime. This is the sequence of what actually happened, and the checklist that exists now specifically because that check didn't happen the first time.

Why the rollback happened in the first place

A separate migration, deployed a few days after the status column, introduced a bug affecting an unrelated table's index that caused a measurable slowdown on a frequently-run query. The fix required rolling back several recent migrations to get back to a known-good schema state before reapplying a corrected version — a reasonable, fairly common recovery approach. The rollback targeted every migration deployed in the prior three weeks, including, without anyone specifically flagging it, the status column migration from two weeks earlier that had nothing to do with the actual bug being fixed.

What nobody checked before running it

In the two weeks between the status column shipping and the rollback, a small internal feature — an admin dashboard widget showing subscription status breakdowns — had been built directly on top of that column, deployed, and was actively in use by the support team to triage tickets. The rollback plan was reviewed for whether it would break the database schema in an internally inconsistent way (it wouldn't — the migrations were written to roll back cleanly in isolation), but nobody checked whether any application code deployed since those migrations had come to depend on the columns being removed. That's a meaningfully different question, and the review process at the time only asked the first one.

// The rollback dropped this column cleanly, exactly as designed:
Schema::table('subscriptions', function (Blueprint $table) {
    $table->dropColumn('status');
});

// ...which the dashboard widget, deployed two weeks later, had come to depend on:
$breakdown = Subscription::query()
    ->select('status')
    ->groupBy('status')
    ->selectRaw('count(*) as total')
    ->get(); // now throws: Unknown column 'status'

What actually broke, and how it was noticed

The rollback ran cleanly with no errors of its own — dropping a column that exists is not, by itself, a failure. The admin dashboard widget started throwing a database error on every load about four minutes later, caught first by an internal error-tracking alert rather than by anyone testing the rollback's actual downstream effects, since the rollback's own test coverage checked schema state, not every piece of application code that might reference the changed table. The support team lost access to the subscription-status breakdown for roughly 25 minutes — not a customer-facing outage, but a real, avoidable disruption to an internal tool people were actively relying on, entirely because a rollback three weeks after the fact didn't account for what had been built in between.

The immediate fix, and why it wasn't as simple as re-running the original migration

Re-adding the status column was straightforward. Restoring the actual data in it was not: the column had held real, meaningful values for two weeks — active, cancelled, past-due — and rolling back the migration had dropped that data along with the column, not preserved it somewhere recoverable. The two weeks of status changes had to be reconstructed from a secondary audit log that happened to track subscription state changes for a separate purpose, which existed by coincidence rather than by design, and that reconstruction wasn't perfectly complete — a handful of subscriptions had their status changed and changed back within a short window in a way the audit log's granularity didn't fully capture, and those specific records needed manual review against billing-provider data to get right.

What the pre-rollback checklist actually contains now

1. List every table and column touched by the migrations being rolled back — not just the one causing the original problem, since a rollback targeting "the last three weeks of migrations" touches everything in that window, whether related to the actual issue or not.
2. Search the codebase for every reference to those tables and columns deployed since the migration in question, using a straightforward grep across the application code, not just a schema-level review — this is the single step that would have caught the dashboard widget dependency before the rollback ran.
3. Check whether any of the affected columns hold data written after the migration ran, not just schema structure — a column that's held real values for any length of time needs a backup or export taken before a rollback drops it, every time, without exception.
4. Prefer a forward-fix migration over a rollback whenever the affected migration is more than a few days old — write a new migration that corrects the specific problem, rather than reaching for a rollback that silently un-does everything since, including changes unrelated to the actual bug.
5. If a rollback is still the right call, take a full backup of every affected table's data immediately before running it, even when the rollback is expected to be clean — "expected to be clean" was also the assessment before this incident.

The rule that generalized best: rollbacks scoped by time, not by relevance

The deeper mistake wasn't skipping a data backup — it was treating "roll back the last three weeks of migrations" as a single unit of work scoped by when the migrations happened to run, rather than by which migrations were actually relevant to the problem being fixed. A rollback scoped that broadly will, by definition, touch things unrelated to the actual issue, and the review process needs to treat every one of those unrelated changes as a real risk worth checking individually, not as harmless collateral swept up in service of fixing the one thing that mattered. Since this incident, a broad time-scoped rollback is treated as a last resort specifically because of this risk, with a narrowly targeted forward-fix migration as the default first option whenever the affected migration isn't from the last day or two.

How this connects to environment and secrets discipline generally

This incident sits in the same category of problem covered in .env management across local, staging, and production without leaking secrets — both are cases where a change that looks contained and low-risk in isolation (rotating a config value, rolling back a migration) turns out to have downstream dependencies that aren't visible from the change itself, only from actually searching the codebase for what depends on it. The specific fix is different in each case, but the underlying discipline is the same: before touching something that's been live long enough for other code to build on it, search for what actually depends on it now, not what depended on it at the moment it was first deployed.

What actually changed in the deploy process, concretely

Beyond the checklist itself, migration rollbacks now require a second reviewer specifically checking application code for downstream dependencies — a distinct sign-off from the schema-level review that already existed — before a rollback touching anything older than 48 hours is allowed to run against production. It's a small amount of added friction for a rollback that, in the vast majority of cases, genuinely is as clean and low-risk as it looks. The cost of that friction is real but small; the cost of the 25 minutes this incident actually caused, plus the manual data-reconstruction work afterward, was larger, and the checklist exists specifically because the second cost is the one that's easy to underestimate until it's already happened.