DanubeData Says Its Monitoring Caught a Storage Outage in Three Minutes and Told Nobody

DanubeData published a postmortem on August 18, 2026, about an outage the day before. One of its storage servers lost power at 00:32 UTC on August 17. Object storage, the container registry, and serverless and static-site deployments were unavailable for "eight hours and fifteen minutes," the company says, while running applications kept serving and no data was lost. Parts of the postmortem follow, not the whole.

Detected, and Seen by Nobody

The company's status page detected the failure and opened an incident at 00:35 UTC, three minutes in. The postmortem says it "was correct, it was public, and it stayed up for the entire outage." Nobody saw it, the company says. It found out at 08:18 UTC, "when an engineer opened the status page for an unrelated reason and asked why so many services were red." The server stayed off until an engineer pressed its power button by hand at 08:22 UTC.

Further on, the postmortem says that "before this incident, a platform incident reached one person, by email." It adds, "At 00:35 on a Sunday morning, an email is not an alarm."

Among the lessons it lists: "Detection and notification are different systems, and we had only built one." And: "Monitoring that nobody is woken by is a record, not an alarm."

The Storage Layout and a Near Miss

The outage was total, the postmortem says, because of where copies of the data were allowed to live. Customer objects were split into six pieces, any four of which can rebuild the object, but the storage cluster spread those pieces across individual disks "without requiring them to be on different machines." When one server went away, it took on average one and a half of every object's six pieces with it, and for a large number of objects "it took two or three." The company says the obvious fix, six pieces on six different machines, could not be done on the day because it was running four storage machines. "This was never one configuration flag away from correct. It was a capacity decision that had been quietly deferred behind a default."

A small set of internal bookkeeping records, kept as two copies rather than split into pieces, made it worse. For a handful of them, including the one the storage gateways read at startup, both copies were on the failed server. In its lessons, the company says that six copies, and six copies allowed to share a machine, "are entirely different guarantees." "We had configured the second while believing we had the first."

There was also a near miss. As the cluster began rebuilding the missing copies onto the remaining machines, its projected path ended at about 97% full, past the threshold where the cluster stops accepting writes entirely. At the observed rebuild rate that point was "around five days away," and the company had not suppressed the rebuild. Its structural lesson is that at 71% utilization, a four-node cluster cannot absorb the permanent loss of a node.

What Changed, and What Was Still Open

Two things shipped within hours of recovery, the postmortem says. From then on, an incident on the public status page would place a voice call to on-call, with an SMS alongside it, and the company says the pager is deliberately outside its normal notification system, because "A page must not be suppressible." A watchdog was set to probe each server every minute, able to press a dead machine's power button unattended, and "Four independent signals must agree" before anything happens.

The postmortem also listed what was not fixed as of August 18. Until data and bookkeeping copies are guaranteed to sit on different machines, it said, losing one machine will still interrupt object storage; the watchdog "shortens that outage; it does not prevent it." Capacity headroom was too thin to absorb the permanent loss of a node. On-call was one phone number. And the root cause of the power loss was unresolved, by the company's own choice, pending recurrence.