[dns-operations] OpenDNS, Google, Nominet - New delegation update failure mode
dougb at dougbarton.email
Thu Apr 2 19:56:27 UTC 2020
I redelegated shopdisney.co.uk this morning. I can see that all of the
Nominet authorities are returning the correct new NS set, however I have
a number of reports of resolution failures. There are resolvers from
OpenDNS, Google, Virgin, O2, and others that are not finding any name
servers at all, and refusing to re-query. This is causing address record
resolution failures for users behind those resolvers.
What is odd to me is that earlier this week we cross-pollinated the old
and new zone files with both the old and new sets of name servers. I
have seen situations in the past where cutting cleanly from one set of
name servers to a completely different set has caused problems, so we
take this extra step of updating the zones so that no matter what point
in the process we're at the resolving name servers will always have at
least one good set to query. It's always worked for me in the past.
What's even more strange is that we also did shopdisney.it this morning,
having done the same preparation, and it's solid as a rock. It's only
the CO.UK name that is failing. When querying OpenDNS or Google directly
I get the same result when it fails:
dig @184.108.40.206 shopdisney.co.uk ns
; <<>> DiG 9.10.6 <<>> @220.127.116.11 shopdisney.co.uk ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 1587
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;shopdisney.co.uk. IN NS
;; Query time: 501 msec
;; SERVER: 18.104.22.168#53(22.214.171.124)
;; WHEN: Thu Apr 02 12:28:46 PDT 2020
;; MSG SIZE rcvd: 45
The flags are the same for the OpenDNS servers.
Has anyone seen this happen before? I've seen plenty of cases where
resolvers have hung onto the old NS set for too long (following the
parent TTL instead of the child), which is why I have been adding both
sets of name servers to both zones in advance of the redelegation. But I
have literally never seen a case where a resolver not only has no NS
records, but also will not re-query.
My first thought was that Nominet withdrew the delegation for a short
period, and the resolvers have a negative cache entry, but when doing
the UAT this morning I happened to catch the exact point at which they
changed. In serial number 1308977661 they had the old NS set, and in
1308977662 they had the new one. So that doesn't seem to be the problem.
If anyone from OpenDNS and/or Google can take a look at a resolver that
is failing for shopdisney.co.uk and tell me what's in the logs I would
deeply appreciate it. Since I can't figure out what happened, I'm not
sure how to mitigate it for the next change.
In the past I've taken the intermediate step of also updating the parent
delegation to include both NS sets, which I plan to do for the next set
of updates just to be on the safe side, but given this fun new failure
mode it's not clear to me that even doing that will insulate us.
Any thoughts/help/advice welcome,
More information about the dns-operations