Migrating data between systems without losing it

The migration nobody remembers correctly

A decade ago this article was about one specific chore: taking an old standalone bbPress forum and folding it into WordPress as a plugin. bbPress itself barely matters anymore. What’s still true is the shape of the problem, because every data migration — forum software, a homegrown CRM, a legacy billing table moving into a new schema — fails in the same handful of ways, and the fixes are the same handful of disciplines.

The bbPress case is a good small example precisely because it’s low stakes: nobody’s money moves and nobody’s account gets locked out if a row goes missing. That made it a forgiving place to learn what a migration needs, and those requirements don’t get more forgiving once the stakes go up — they get less.

Idempotency: running it twice should be safe

The original bbPress upgrade path was a one-shot script: point it at your database, run it once, hope it worked. If it failed halfway — timed out, hit a malformed row, ran out of memory — the safe move was unclear, because running it again risked processing already-migrated rows a second time and corrupting them further.

A migration written to be idempotent avoids this entirely: each unit of work checks whether it has already been done before doing it again.

-- Bad: unconditionally inserts, breaks on rerun (duplicate key or duplicate row)
INSERT INTO wp_posts (post_title, post_content, post_type)
SELECT topic_title, topic_text, 'forum_topic' FROM bb_topics;

-- Idempotent: safe to run as many times as needed
INSERT INTO wp_posts (post_title, post_content, post_type, legacy_topic_id)
SELECT topic_title, topic_text, 'forum_topic', topic_id
FROM bb_topics
WHERE topic_id NOT IN (SELECT legacy_topic_id FROM wp_posts WHERE legacy_topic_id IS NOT NULL);

The legacy_topic_id column is the important part: it’s a durable foreign-key-style link back to the source system, and it’s what makes “have I already migrated this row” an answerable question instead of a guess. Keep that column (or a mapping table, if you’d rather not touch the target schema) even after the migration finishes — it’s what makes every later step in this list possible.

Reversibility: plan the way back before you need it

The failure mode the original bbPress migration was most exposed to was silent data loss on parent-child relationships — a subforum losing its reference to its parent forum during the version jump, with no error raised anywhere. Nobody found out until someone went looking for a forum that had quietly become orphaned.

The fix generalizes: never migrate in a way that discards the source data, and never cut over in a way you can’t undo. Concretely, that means:

• The source system stays untouched and queryable until the migration is verified — don’t delete the old bbPress tables the day of the upgrade, keep them for a defined retention window.
• The cutover itself is a flag flip, not a destructive operation — point reads at the new data, keep the old data reachable, and make “point reads back at the old data” a fast, tested operation, not a hypothetical one.
• Any transformation that can’t be inverted (a field that gets reformatted, values that get merged) gets the original value preserved alongside it, at least until verification passes.

Verification counts, not spot checks

“I looked at a few rows and it seemed fine” is not verification. What worked for the forum migration, and what works for anything larger, is a row-count and relationship-integrity comparison run automatically after the migration completes:

-- Every migrated topic must have a real forum to belong to
SELECT COUNT(*) FROM wp_posts
WHERE post_type = 'forum_topic' AND post_parent NOT IN (SELECT ID FROM wp_posts WHERE post_type = 'forum');

-- Row counts must match between source and target, not just "looks close"
SELECT (SELECT COUNT(*) FROM bb_topics) AS source_count,
       (SELECT COUNT(*) FROM wp_posts WHERE post_type = 'forum_topic') AS target_count;

The first query is the one that would have caught the orphaned-subforum problem immediately instead of whenever a user happened to notice. The second is cheap insurance against the more mundane failure — a batch that silently dropped rows because of a timeout or a WHERE clause that excluded more than intended. Neither query is sophisticated. Both are the difference between finding a problem in a test run and finding it from a support ticket three weeks after cutover.

Cutover: a rehearsed event, not a one-time script run

The original article’s step-by-step — download this package, drop it in that folder, log in and watch for a confirmation message — reads today like exactly what it was: a manual runbook with no dry run and no rollback built in. The improvement isn’t more automation for its own sake, it’s treating cutover as a rehearsed event:

1. Run the migration against a copy of production data, not a synthetic fixture — real data has the malformed rows and edge cases a fixture won’t.
2. Run the verification queries and require them to pass before scheduling the real cutover.
3. Do the real migration during a window where the source system can be temporarily read-only, so nothing writes to it mid-migration.
4. Keep the source system’s data available and the rollback path tested for a defined period after cutover — a week is typical for something this size — not deleted the moment the migration script exits successfully.

What generalizes

None of this is forum-specific, which is the actual point. Idempotent units of work, a durable link back to source records, a rollback path that’s tested rather than assumed, and automated verification that checks counts and relationships rather than eyeballing a sample — that combination is what separates a migration that’s boring from one that generates a support ticket a month later from someone who just discovered their data went missing during a system upgrade nobody told them about.

---

Originally published in 2015 and updated for 2026.