<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Hi, <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>For us nothing really interesting happened on the Flag day on our resolvers, since we did not update to a version that had removed these workarounds. Most of the calls to our help desk was regarding why their zones where marked with warnings. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>But we did encountered the problem with DNS cookies before the flag day, like 1.5 years earlier, when we introduced it to our platform. It took a while to figure out what happened since we only run 50% with it turned on and we had a load balancer in front. So it randomly worked and failed.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We concluded then that the world was not ready for DNS cookies. One that we contacted used a 14 year old Windows installation, they have now changed to have BIND on the fronting delegated DNS servers. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We did also actually turn off DNS cookies to the Authoritative on the servers to get rid of it. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=SV>BR,<o:p></o:p></span></p><p class=MsoNormal><span lang=SV>Bjorn Hellqvist<o:p></o:p></span></p><p class=MsoNormal><span lang=SV>Senior System Expert (Internet, DNS & Automation)<o:p></o:p></span></p><p class=MsoNormal>Telia Company<o:p></o:p></p><p class=MsoNormal>Solna, Sweden<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b>From:</b> dns-operations <dns-operations-bounces@dns-oarc.net> <b>On Behalf Of </b>Shumon Huque<br><b>Sent:</b> den 16 september 2019 20:21<br><b>To:</b> DNS Operations List <dns-operations@dns-oarc.net><br><b>Subject:</b> [dns-operations] Experiences with a post 2019 Flag Day Resolver<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Hi folks,<br><br>I haven't seen too much discussion here about operational experiences with post 2019 DNS Flag Day resolvers, so I thought I'd share ours. Would be interesting to hear from others on this topic.<br><br>We recently upgraded some of our resolvers to BIND 9.14.x. Soon after, we started getting complaints about numerous sites unable to be resolved (the response from the resolver to clients is SERVFAIL). We assumed this was related to post flag-day non-workarounds for broken authoritative servers. Hence, we expected these sites to also fail on software deployed by other flag day participants, namely Google Public DNS, Cloudflare, Quad9, OpenDNS, and recent versions of Unbound, PowerDNS, and Knot. But the sites resolved fine on all these platforms (actually I didn't get around to testing Knot yet, since the few minutes I devoted to figuring out how to use build tools unfamiliar to me like Ninja/Meson wasn't enough - will try to learn them later).<br><br>Some quick debugging revealed that this is because BIND sends outbound queries with DNS cookies by default, and none of the other implementations do. These non-resolving sites don't answer any queries with cookies. We tested sending a variety of other EDNS options to them, and the "non response" behavior is also the same. But all of them do respond to EDNS enabled queries containing no options.<o:p></o:p></p><div><p class=MsoNormal><br>(BIND of course has been using cookies by default in earlier versions too, but presumably they had the workaround behavior of retrying without them on non-response).<br><br>The proportion of these sites in comparison to the total population of zones that our resolvers talk to, is small, but not trivial. We have attempted to contact the zone owners in question as we discover them, pointing them to the various DNS compliance testing tools/sites. But this was getting to be burdensome enough that we ended up turning off outbound cookies ("send-cookie no;" in the global options).<o:p></o:p></p></div><div><p class=MsoNormal><br>Google Public DNS sends the EDNS Client Subnet option to authority servers that we run, and presumably to those broken servers too. We cannot observe the conversation between Google and the broken sites, but since they resolve, we assume that they might at least have a workaround to retry such sites without ECS (or maybe a dynamically maintained ECS blacklist is in use). Perhaps, a Google Public DNS operator can confirm or disconfirm this.<o:p></o:p></p></div><div><p class=MsoNormal><br>--<br>Shumon Huque<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div></div></div></div></div></body></html>