[dns-operations] DNS cookies in a mixed resolver anycast environment

Fri May 31 14:55:50 UTC 2019

Hi,

During the period, the oldest encounter and one of the most critical was a 17 year old Authoritative Servers running Windows DNS. They have now fixed this, it took around 6 months for them. I believe they were not alone. 

Just because 99.9% looks OK in statistics, does not mean that it really work in real life scenarios. Businesses and Government organs still think that "DNS is old and easy service, we do not need to update". 

Even if and when we reach out, there are instances that does not listen and still think it is our fault. 

So when Google, Q9, Cloudflare, etc. starts to do DNS Cookies, then it will be easier to enable it again. 

The favorite quote we always get: "It works on google, but not on your service. It is not a problem in our side. Please fix!".

(It has been 1.5 years ago since we tried it, so it might be a bit different today)

BR,
Bjorn Hellqvist
Senior System Expert (Internet, DNS & Automation)
Telia Company
Solna, Sweden

> -----Original Message-----
> From: Mark Andrews <marka at isc.org>
> Sent: den 31 maj 2019 16:21
> To: Hellqvist, Björn <bjorn.hellqvist at teliacompany.com>
> Cc: Patrik Lundin <patrik at sigterm.se>; dns-operations at dns-oarc.net
> Subject: Re: [dns-operations] DNS cookies in a mixed resolver anycast
> environment
> 
> Very few authoritative servers got it so bad as to break lookups. Echoing of
> the option doesn’t cause a problem. Returning FORMERR doesn’t cause a
> problem unless you the zone is also signed and you are validating.
> 
> There where a few that blocked queries but they got fixed relatively quickly.
> There where a few badly configured load balancers that returned
> NXDOMAIN but those too got fixed quickly.  You just had to contact the
> owners.
> 
> The fallback code for EDNS no response and handling plain EDNS FORMERR
> addressed most of the issues.
> 
> The world wasn’t 100% ready but it was ready enough.  I added about 20
> exceptions from when we first released BIND 9.10.0 to now for zones that
> where badly broken.
> 
> If one waited until authoritative servers where 100% ready you would never
> turn improve anything.
> --
> Mark Andrews
> 
> > On 31 May 2019, at 20:26, Hellqvist, Björn
> <bjorn.hellqvist at teliacompany.com> wrote:
> >
> > Hi,
> >
> > Just to respond with our experience regarding DNS Cookie, which we tried
> enabling in autumn 2017.
> >
> > At that time we figured out that the "world" was not ready for that feature
> and we got quite some complaints both by our customers and owners of
> domains.
> >
> > The problem was that the Authoritative parts were not responding
> properly to queries using DNS Cookie. Several important domains stopped
> working for our customers.
> >
> > Ever since then we have had to disable DNS Cookies upstream to the
> Authoritative servers.
> >
> > BR,
> > Bjorn Hellqvist
> > Senior System Expert (Internet, DNS & Automation) Telia Company Solna,
> > Sweden
> >
> >> -----Original Message-----
> >> From: dns-operations <dns-operations-bounces at dns-oarc.net> On
> Behalf
> >> Of Mark Andrews
> >> Sent: den 31 maj 2019 10:15
> >> To: Patrik Lundin <patrik at sigterm.se>
> >> Cc: dns-operations at dns-oarc.net
> >> Subject: Re: [dns-operations] DNS cookies in a mixed resolver anycast
> >> environment
> >>
> >> Well having different implementations serving the same client is
> >> always a dangerous idea, whether is is different vendors or not.
> >> Resolvers need to track server characteristics.
> >>
> >> * Does the server support EDNS?
> >> * Does EDNS buffer size X work?
> >> * Does the server support DNS COOKIE?
> >> * ...
> >>
> >> RFC 7873 had BIND’s algorithms described so that other
> >> implementations could implement them if they were intending to be
> >> configured in a anycast scenario with BIND.  Doing so reduces the
> >> number of BAD COOKIE responses and subsequent fallback to TCP.  DNS
> >> COOKIE clients will handle different algorithms and/or different shared
> secrets.
> >>
> >> As for discarding replies without DNS COOKIEs once the recursive
> >> server learns that the server supports DNS COOKIEs there is only a
> >> issue if the operator has configured a anycast server cluster for a
> >> given client with servers that both support and don’t support DNS
> >> COOKIE.  The client thinks it is seeing spoofed responses when it see
> >> non-DNS COOKIE responses.  If you are configuring a anycast server
> >> it’s just one more thing you need to ensure is consistent across the server
> set for a given client.
> >>
> >> Having mixed support is much more of a issue for local anycast
> >> instances than it is for global anycast instances.
> >>
> >> Mark
> >>
> >>> On 31 May 2019, at 5:11 pm, Patrik Lundin <patrik at sigterm.se> wrote:
> >>>
> >>> Hello!
> >>>
> >>> I have been trying to figure out how to best deal with DNS cookies
> >>> in an environment where you are running multiple resolver
> >>> implementations. From what I can tell, out of BIND, Knot Resolver,
> >>> PowerDNS Recursor and Unbound only BIND is currently implementing
> >>> cookie support. Knot seemed to have done so previously, but as of
> >>> 3.0.0 the cookie support was removed
> >>> (https://www.knot-resolver.cz/2018-08-20-knot-resolver-3.0.0.html)
> >>> because of some ongoing work in the IETF DNSOP.
> >>>
> >>> Reading RFC 7873 it states "If the client is expecting the response
> >>> to contain a COOKIE option and it is missing, the response MUST be
> >>> discarded.", which leads me to believe that having a anycast cluster
> >>> of a set of BIND servers where cookies are enabled together with a
> >>> set of servers where the cookies are not supported would be a bad
> >>> thing, causing clients to discard answers.
> >>>
> >>> Yet, when looking up how one would go about to disable the sending
> >>> of cookies in responses to clients for BIND, the documentation for
> >>> "answer-cookie"
> >>> (https://ftp.isc.org/isc/bind9/cur/9.15/doc/arm/Bv9ARM.ch05.html)
> >>> states the following:
> >>>
> >>> "answer-cookie no is intended as a temporary measure, for use when
> >>> named shares an IP address with other servers that do not yet
> >>> support DNS COOKIE. A mismatch between servers on the same address
> >>> is not expected to cause operational problems, but the option to
> >>> disable COOKIE responses so that all servers have the same behavior
> >>> is provided out of an abundance of caution. DNS COOKIE is an
> >>> important security mechanism, and should not be disabled unless
> >>> absolutely
> >> necessary."
> >>>
> >>> If clients are instructed to discard replies where the cookie are
> >>> missing, how can this not cause operational problems? Am I missing
> >>> something?
> >>>
> >>> On a related note, given that a set of BIND servers are already
> >>> having the default cookies enabled, what is the expected fallout of
> >>> setting "answer-cookie no" if this turns out to be the favorable
> >>> approach in this case?
> >>>
> >>> Regards,
> >>> Patrik Lundin
> >>> _______________________________________________
> >>> dns-operations mailing list
> >>> dns-operations at lists.dns-oarc.net
> >>> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
> >>> dns-operations mailing list
> >>> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
> >>
> >> --
> >> Mark Andrews, ISC
> >> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> >> PHONE: +61 2 9871 4742              INTERNET: marka at isc.org
> >>
> >>
> >> _______________________________________________
> >> dns-operations mailing list
> >> dns-operations at lists.dns-oarc.net
> >> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
> >> dns-operations mailing list
> >> https://lists.dns-oarc.net/mailman/listinfo/dns-operations