On October 11, 2026, from 16:40 to 17:35 UTC, Control D's DNS service for paying customers "stopped answering queries at every location," the company says in its incident report of October 20. "Almost everything online starts with a DNS lookup," the report says. Its free service stayed online, and the company says no query was answered incorrectly and no customer settings or data were changed or exposed. "This was our own error," it says. Parts of the report follow, not the whole.
From the September Update to October 11
On September 16 the company released an API update that added validation for an optional per-domain override in its services, the ready-made entries in a profile for things like streaming platforms. As part of the update, the report says, "the API began saving the override as an integer. Our DNS software expects it as a string."
The new code, the report says, only ran when a domain was saved with an explicit override, such as allow or block. "Ordinary additions to a service did not touch it, and services were edited in the following weeks without incident."
At 16:40 UTC on October 11, a member of the team made a routine change in the company's admin tools to resolve a customer request, adding four domains to one service with an explicit allow override. It was, the report says, "the first save of that kind in the 25 days since the September 16 update." The API stored the four overrides as numbers and told every DNS server for paid plans to reload that service.
None of them could read the new values. Because of an error-handling flaw, the DNS software stopped instead of logging and skipping them. A safeguard moved traffic toward healthy locations, "but every location had the same problem, so there was nowhere healthy to send it." The software restarted automatically. It was designed not to start if it could not read all of its configuration, so that it would never serve a half-loaded filtering policy, and so every restart failed on the same stored values until they were corrected.
The cause was traced to the 16:40 change and the September update at 17:16. At 17:30 the affected service was saved again with the four overrides removed, and service was fully restored at every location by 17:35.
What the Checks Did Not See
Under "Contributing conditions," the report says: "Our tests did not check that data saved by our API stayed in the format our DNS software reads." Configuration written by the API "was not checked against our DNS software's own reader before it was published." Updates to the DNS software itself are rolled out gradually; configuration changes are not, "so this one reached every DNS server for paid plans at the same moment."
The first automated alerts, at 16:46, "reported individual servers and locations failing. None of them said that DNS for paying customers was down." The company says it was too slow to tell customers what was happening. It had no incident status page, and its network status page, the report says, "still showed most locations as healthy," while many customers learned from community channels that the problem was the company's.
The API's output and the DNS software's input were two sides of one agreement, and no test held the two together. Each side could pass its own checks while the pair was broken.
What It Is Changing
The report gives each fix with its status. A DNS software update that sets aside and logs any service entry it cannot read, instead of stopping, is done, and so is an API fix that stores overrides as text again. A test that saves service changes "through the real code path and checks the exact stored format," with regression tests of malformed input, is written and being added to its automated test pipelines; run against the code that was live on October 11, it says, "these tests reproduce the failure." Checking configuration against the DNS software's own reader before publishing is planned. Rolling out configuration changes gradually is in design.
On alerting, it plans to page on-call when answered DNS queries for paying customers drop to zero in any region for more than two minutes, among other triggers, and says, "Any of these would have paged us within two to four minutes on October 11." A status page that runs independently of its own infrastructure is being set up.
Which of a system's own reports would still read healthy while the thing it reports on has stopped?
Sources
- Control D. Incident Report: Control D DNS Outage on October 11, 2026. 20 October 2026. incident report