LootLocker Says a Replication Process Recovered Normally After a Crash, With No Visible Signal That Data Had Been Dropped

On May 8, 2026 LootLocker published a postmortem on its own blog, headed "Postmortem: Data Loss During Database Migration." The company was moving from a large managed database provider to a custom MySQL deployment running with a European infrastructure partner. On April 28, 2026, during that migration, it "discovered that a small window of data written shortly before the migration had not been replicated to the new database." The company says it is publishing the postmortem in full so that customers "can see exactly what happened, what we got wrong, and what we are changing as a result." Parts of it follow, not the whole.

A Short Crash, and No Visible Signal

The root cause, it says, was traced to a crash eight days earlier. A temporary, one-off replication process copied data to the new database for the length of the migration, and it held a database connection using credentials. On April 20 those credentials were rotated as part of an unrelated configuration fix, and the rotation caused the process to crash. "It recovered automatically within minutes, but the data batches that were in flight at the moment of the crash were dropped and were not replayed."

"The loss went undetected until migration day," the postmortem says. "The replication process continued running normally after recovering, so there was no visible signal that anything had been dropped." The way replication offsets and timezones interacted, it adds, meant the affected window also included a small amount of data from the previous day.

A process that is running looks healthy. Here the running process was what hid the loss, and no monitoring or alerting was watching the temporary sync's health or its replication lag.

Contributing Factors, and the Recovery

The postmortem lists the contributing factors. There was no monitoring or alerting on the health or replication lag of the temporary sync process. There was no step in the runbook before cutover requiring a check of the sync's health and the data's integrity. The credential rotation was treated as a low-risk configuration change, "with no downstream impact assessment for processes that might be holding the credential." And the source database was deleted shortly after cutover, before data integrity in the new database had been confirmed.

That last decision shaped the recovery. Data anomalies were detected in production after cutover, and service was degraded "for approximately 3 hours from cutover until normal behaviour was restored." Normal service was restored first, and the root cause was found that evening. The original database, by then deleted, was needed as the source to replay from, so a restore was requested from the previous provider. Once it was restored, the affected records were replayed over the following two days.

What the Company Says It Learned, and What It Cannot Rule Out

"The failure was a silent secondary effect of an unrelated change made eight days earlier," the postmortem says. "The insidious part was that the process did recover and continue working, so there was no obvious signal that a small window of data had been dropped." It says a data integrity check before cutover "would have caught this before traffic resumed," and that choosing a live migration to minimize disruption "worked against us," because it left "no clean point at which to verify integrity before traffic resumed."

The changes it lists include these. Future changes of comparable risk will be made in a scheduled maintenance window rather than as live cutovers. A checklist before cutover will be required for changes of this kind, including verification that any replication or sync process is live and caught up. Credential rotations affecting service accounts used by long-running processes will need an explicit review of their downstream impact. And source systems will not be deleted until integrity in the target has been independently verified after cutover.

On the impact, it adds: "We cannot fully exclude the possibility that some additional data written during the window of service instability was affected."