IONOS Says a Drift Between Its Central Repository and Live Hardware Settings Went Undetected Before a Configuration Rollout

In a root cause analysis published on its status page on February 25, 2026, IONOS Cloud says a configuration deployment during scheduled maintenance rebuilt device settings from a repository that lacked settings present only on the live hardware, and that, although the change was validated under a four-eyes principle, the rendered output did not provide enough visibility into the discrepancy.

IONOS Cloud posted a root cause analysis on its own status page on February 25, 2026, under the incident heading "Network service degraded." On February 24, it says, during a scheduled maintenance window intended to improve network resiliency in its FRA region, "a configuration deployment led to a loss of connectivity for public services." Internal monitoring detected the outage shortly after the start of the maintenance, once the change was applied. By 20:55 UTC a manual fix was applied to the affected devices, and the initially affected services were restored by 21:00 UTC. Parts of the report follow, not the whole.

The maintenance, the report says, "was initially planned to have only a few seconds of service disruption; however, due to the issues described, it caused a significantly higher impact."

The status page keeps that evening's entries too. The earliest, posted at 20:39 UTC, spoke of "sporadic connection issues and substantial delays on packet delivery." At 21:05 UTC an entry said a fix had been implemented and customers should see traffic and connectivity return to normal. At 21:44 UTC the issue was marked resolved, with the root cause analysis to follow on the same page.

Settings the Repository Did Not Have

The incident was caused, the report says, by "a configuration state drift" between the company's central software repository and the live hardware settings in a production environment, a drift "that went undetected before the rollout." The settings involved were Forward Error Correction (FEC) settings, which the report describes as "parameters that allow different brands of networking hardware to communicate reliably."

"Unlike the staging environments, the production environment had unique configurations for settings required for multi-vendor hardware interoperability," the report says. The change was validated under what the report calls a four-eyes principle. Even so, it says, "the rendered output did not provide enough visibility into the unexpected discrepancy. This caused the difference to go unnoticed."

The automated deployment, the report says, performed what it calls a full rebuild of the device configuration. "Because the repository did not contain the specific FEC settings (the discrepancy), it omitted them during the rebuild." Once the new configuration was pushed, "the lack of FEC parity caused the physical links between mismatched devices to fail, dropping traffic for all public services in the region."

Corrective Actions

The corrective actions the report lists are each to be completed within the first quarter of 2026. The first is "a full audit of all production devices," to find and resolve any drift where live settings differ from the central repository. The second is an automated pre-flight check that compares the intended configuration against the running configuration, "to clearly flag any potential omissions before a change is finalized." That, the report says, "increases the visibility of drifts and unexpected discrepancies and reduces the surface area for human error."

Sources

IONOS Cloud, on its status page. Network service degraded (incident report and root cause analysis). 25 February 2026. incident report

More from this shelf: Deployments and changes

October 26, 2026

Deployments and changes

Control D Says Its Tests Did Not Check That Data Saved by Its API Stayed in the Format Its DNS Software Reads

In an incident report of October 20, 2026, Control D says an API update of September 16 began saving a setting in a format its DNS software could not read, and that the first save of that kind, 25 days later, stopped DNS for its paid plans at every location for about 54 minutes.

October 23, 2026

Deployments and changes

Surveypal Says a Production Release Was Deployed but Only Partially Applied, and Automations for Submitted Events Did Not Trigger Correctly

In an incident report posted on its status page on April 13, 2026, Surveypal said a production release of April 10 "was deployed but only partially applied." Part of its infrastructure kept running an older version, automations for submitted events did not trigger correctly, and some customers experienced missing automation triggers. The release was fixed on April 13.