[dns-operations] OpenDNS, Google, Nominet - New delegation update failure mode

[Puneet Sood]

Hi Doug,

Google Public DNS resolution is working now.

Google Public DNS is “parent-centric”—meaning that it only uses the
name servers that are returned in the referral responses from the
parent zone name servers, and does not make NS queries to this child
zone. So updating the parent delegation to include both NS sets will
help with Google Public DNS resolution.

-Puneet

$ dig @8.8.4.4 shopdisney.co.uk ns

; <<>> DiG 9.11.5-P4-5.1+build2-Debian <<>> @8.8.4.4 shopdisney.co.uk ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8397
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;shopdisney.co.uk.              IN      NS

;; ANSWER SECTION:
shopdisney.co.uk.       167     IN      NS      a12-66.akam.net.
shopdisney.co.uk.       167     IN      NS      a18-64.akam.net.
shopdisney.co.uk.       167     IN      NS      a28-65.akam.net.
shopdisney.co.uk.       167     IN      NS      a1-127.akam.net.
shopdisney.co.uk.       167     IN      NS      a9-66.akam.net.
shopdisney.co.uk.       167     IN      NS      a13-67.akam.net.

;; Query time: 20 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Thu Apr 02 16:08:57 EDT 2020
;; MSG SIZE  rcvd: 178

On Thu, Apr 2, 2020 at 4:01 PM Doug Barton <dougb at dougbarton.email> wrote:
>
> Howdy,
>
> I redelegated shopdisney.co.uk this morning. I can see that all of the
> Nominet authorities are returning the correct new NS set, however I have
> a number of reports of resolution failures. There are resolvers from
> OpenDNS, Google, Virgin, O2, and others that are not finding any name
> servers at all, and refusing to re-query. This is causing address record
> resolution failures for users behind those resolvers.
>
> What is odd to me is that earlier this week we cross-pollinated the old
> and new zone files with both the old and new sets of name servers. I
> have seen situations in the past where cutting cleanly from one set of
> name servers to a completely different set has caused problems, so we
> take this extra step of updating the zones so that no matter what point
> in the process we're at the resolving name servers will always have at
> least one good set to query. It's always worked for me in the past.
>
> What's even more strange is that we also did shopdisney.it this morning,
> having done the same preparation, and it's solid as a rock. It's only
> the CO.UK name that is failing. When querying OpenDNS or Google directly
> I get the same result when it fails:
>
> dig @8.8.4.4 shopdisney.co.uk ns
> ; <<>> DiG 9.10.6 <<>> @8.8.4.4 shopdisney.co.uk ns
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 1587
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 512
> ;; QUESTION SECTION:
> ;shopdisney.co.uk.              IN      NS
>
> ;; Query time: 501 msec
> ;; SERVER: 8.8.4.4#53(8.8.4.4)
> ;; WHEN: Thu Apr 02 12:28:46 PDT 2020
> ;; MSG SIZE  rcvd: 45
>
> The flags are the same for the OpenDNS servers.
>
> Has anyone seen this happen before? I've seen plenty of cases where
> resolvers have hung onto the old NS set for too long (following the
> parent TTL instead of the child), which is why I have been adding both
> sets of name servers to both zones in advance of the redelegation. But I
> have literally never seen a case where a resolver not only has no NS
> records, but also will not re-query.
>
> My first thought was that Nominet withdrew the delegation for a short
> period, and the resolvers have a negative cache entry, but when doing
> the UAT this morning I happened to catch the exact point at which they
> changed. In serial number 1308977661 they had the old NS set, and in
> 1308977662 they had the new one. So that doesn't seem to be the problem.
>
> If anyone from OpenDNS and/or Google can take a look at a resolver that
> is failing for shopdisney.co.uk and tell me what's in the logs I would
> deeply appreciate it. Since I can't figure out what happened, I'm not
> sure how to mitigate it for the next change.
>
> In the past I've taken the intermediate step of also updating the parent
> delegation to include both NS sets, which I plan to do for the next set
> of updates just to be on the safe side, but given this fun new failure
> mode it's not clear to me that even doing that will insulate us.
>
> Any thoughts/help/advice welcome,
>
> Doug
> _______________________________________________
> dns-operations mailing list
> dns-operations at lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations