[dns-operations] 0x20 breakage *over TCP* from CDNS servers affecting 35 TLDs

[Zenon Mousmoulas]

Hello again.

As of today this issue now affects only the following TLDs and NS currently in the root zone:

ac.			172800	IN	NS	ns-a3.ac.
bn.			172800	IN	NS	ns1.bnnic.bn.
io.			172800	IN	NS	ns-a3.io.
sh.			172800	IN	NS	ns-a3.sh.
tm.			172800	IN	NS	ns-a3.tm.
tm.			172800	IN	NS	ns-a4.tm.

Let me say thank you to whoever worked towards getting this fixed.

Regards,
Z.

January 28, 2019 8:09 PM, "Zenon Mousmoulas" <zmousm at noc.grnet.gr> wrote:

> Hello.
> 
> We have observed that DNS servers operated by CommunityDNS (CDNS) break DNS 0x20, but this happens
> *only* for queries over TCP.
> 
> One might argue that 0x20, as a mitigation strategy, only has any practical value for queries and
> responses over UDP. Yet neither the DNS 0x20 I-D explicitly differentiates TCP from UDP, nor any
> implementation that I know of. Unbound resolver, for example, once use-caps-for-id is enabled,
> observes 0x20 for all responses, regardless the transport. Beyond that, the fact that CDNS servers
> respect 0x20 when queried over UDP makes me think this is perhaps an oversight.
> 
> User reports for Let's Encrypt (LE) issuance failures initially drew our attention to this issue.
> LE enforce 0x20 in unbound resolvers employed by ACME servers. As of recent they also cap EDNS
> buffer size to 512 bytes, effectively pushing more queries to TCP. Tests showed that DNS requests
> over TCP hitting a particular TLD server lead to SERVFAIL, due to both 0x20 and fallback mechanism
> breakage. This is also more likely to happen when the delegation chain points to NS in the affected
> TLD.
> 
> We reported this to the affected TLD. I later wrote to CDNS myself when I realized this affects
> more TLDs, and I also reported it to LE:
> 
> https://community.letsencrypt.org/t/widespread-servfail-problem-related-to-dns-0x20/83812/5
> 
> The spread of this breakage and the fact that I could not find any previous discussion on the
> matter prompted me to also post to this list.
> 
> As of today this issue affects the following TLDs and NS currently in the root zone:
> 
> ac. 172800 IN NS ns-a1.ac.
> ac. 172800 IN NS ns-a3.ac.
> am. 172800 IN NS ns-cdn.amnic.net.
> be. 172800 IN NS x.ns.dns.be.
> bn. 172800 IN NS ns1.bnnic.bn.
> brussels. 172800 IN NS x.nic.brussels.
> bs. 172800 IN NS ns36.cdns.net.
> ch. 172800 IN NS g.nic.ch.
> dm. 172800 IN NS ns34.cdns.net.
> fi. 172800 IN NS e.fi.
> gr. 172800 IN NS gr-c.ics.forth.gr.
> hu. 172800 IN NS ns-com.nic.hu.
> io. 172800 IN NS ns-a1.io.
> io. 172800 IN NS ns-a3.io.
> li. 172800 IN NS g.nic.li.
> lt. 172800 IN NS c.tld.lt.
> lu. 172800 IN NS k.dns.lu.
> lv. 172800 IN NS c.nic.lv.
> mo. 172800 IN NS ns17.cdns.net.
> my. 172800 IN NS ns30.cdns.net.
> ng. 172800 IN NS ns1.nic.net.ng.
> ph. 172800 IN NS ph.communitydns.net.
> pl. 172800 IN NS h-dns.pl.
> scb. 172800 IN NS c.nic.scb.
> sh. 172800 IN NS ns-a1.sh.
> sh. 172800 IN NS ns-a3.sh.
> si. 172800 IN NS g.dns.si.
> th. 172800 IN NS c.thains.co.th.
> tm. 172800 IN NS ns-a1.tm.
> tm. 172800 IN NS ns-a2.tm.
> tm. 172800 IN NS ns-a3.tm.
> tm. 172800 IN NS ns-a4.tm.
> ua. 172800 IN NS cd1.ns.ua.
> vlaanderen. 172800 IN NS x.nic.vlaanderen.
> vn. 172800 IN NS a.dns-servers.vn.
> xn--fzc2c9e2c. 172800 IN NS lk.communitydns.net.
> xn--mgbx4cd0ab. 172800 IN NS ns30.cdns.net.
> xn--mix891f. 172800 IN NS ns17.cdns.net.
> xn--qxam. 172800 IN NS gr-c.ics.forth.gr.
> xn--xkc2al3hye2a. 172800 IN NS lk.communitydns.net.
> xn--y9a3aq. 172800 IN NS ns-cdn.amnic.net.
> 
> Regards,
> Zenon Mousmoulas
> GRNET NOC