[dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

[Viktor Dukhovni]

On Tue, Jul 11, 2023 at 10:24:21PM +0000, Wessels, Duane wrote:

> Last week, during a migration of one of our DNS resolution sites in
> Singapore, from one provider to another, we unexpectedly lost
> management access and the ability to deliver changes and DNS updates
> to the site. Following our standard procedure, we disabled all transit
> links to the affected site. Unfortunately, a peering router remained
> active, which was not immediately obvious to our teams due to the lack
> of connectivity there.

Thanks for the PM details, much appreciated.

> Over the weekend, this caused an issue that may have affected the
> ability of some internet users in the region to reach some .com and
> .net domains, as DNSSEC signatures on the site began expiring. The
> issue was resolved by powering off the site’s peering router, causing
> the anycast route announcement to be withdrawn and traffic to be
> directed to other sites.

I should note that DNSSEC was not the only fallout from outdated zone
files.  Some delegations had stale NS records, for which the outdated
nameservers were already returning REFUSED (or outdated answers).

Consequently, some users unlucky enough to have switched providers or
moved to new NS hosts at the same a provider after the site was cut off
from updates also observed some issues, whether or not DNSSEC happened
to be involved.

-- 
    Viktor.