<div dir="ltr"><div dir="ltr">On Sun, Apr 14, 2019 at 9:59 AM Jon Reed <<a href="mailto:jreed@akamai.com">jreed@akamai.com</a>> wrote:</div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
On Sat, 13 Apr 2019, Shumon Huque wrote:<br>
<br>
> Now that you've deployed the fix, are you able to share how you <br>
> addressed the customer expectation problem that you describe? e.g. Did <br>
> you advise customers not to deploy CNAME records before they are fully <br>
> provisioned?<br>
<br>
The problem I described in the dnsop post wasn't our customers, it was our <br>
customers' customers. It put us in the position of having to tell a <br>
large SaaS provider that _their_ customers were doing the wrong thing, and <br>
they obviously were not very amenable to hearing that. In fact, in one <br>
case, they did move to a competitor that still had the incorrect wildcard <br>
matching behavior.<br></blockquote><div><br></div><div>Ah, thanks for that additional clarification .. it makes more sense now.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Did you deploy special case code in your nameservers for <br>
> these services to match wildcards above or parallel to the ENTs, while <br>
> providing NODATA responses to the ENTs? Or something else?<br>
<br>
Yes. We did an exhaustive survey of our zones, tried to infer customer <br>
intent, worked with our customers where we could, and when we couldn't, <br>
were forced to develop a new feature that operates in a similar manner as <br>
the one described. (I'm not sure how detailed I can get.)<br></blockquote><div><br></div><div>Got it. Thanks for confirming this level of detail at least.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> <br>
> At that time, Akamai and Cloudflare were the main problematic services in a survey of the Alexa top 1000 sites. I notified both of them well before my talk, and Cloudflare had already fixed their ENT<br>
> response behavior by the time of the talk. There were several Chinese CDN providers on the list too - I managed to contact a couple of them, and they informed me they were working on fixes, but I didn't<br>
> follow up and check on their status.<br>
<br>
I don't want to get into finger-pointing, but I think the wildcard <br>
matching issue we encountered was is subtler and harder to detect than the <br>
examples given in the slides.<br></blockquote><div><br></div><div>Yes, agreed (my survey on the other hand wasn't specifically looking</div><div>at the wildcard matching issue).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Consider the following zone in its entirety:<br>
<br>
<a href="http://example.com" rel="noreferrer" target="_blank">example.com</a> IN SOA [omitted for brevity]<br>
<a href="http://example.com" rel="noreferrer" target="_blank">example.com</a> IN NS <a href="http://ns1.example.com" rel="noreferrer" target="_blank">ns1.example.com</a>.<br>
*.<a href="http://example.com" rel="noreferrer" target="_blank">example.com</a> IN CNAME <a href="http://not-yet-provisioned.example.biz" rel="noreferrer" target="_blank">not-yet-provisioned.example.biz</a><br>
<a href="http://www.dev.example.com" rel="noreferrer" target="_blank">www.dev.example.com</a> IN A 192.168.0.1<br>
<br>
When we surveyed other providers at the start of this rollout in 2016, we <br>
found that a query for "<a href="http://dev.example.com" rel="noreferrer" target="_blank">dev.example.com</a>" would correctly return an ENT for <br>
nearly all of them, which is consistent with your results.<br>
<br>
However, for _all but one_ of the providers we surveyed, a query for <br>
"<a href="http://doesnotexist.dev.example.com" rel="noreferrer" target="_blank">doesnotexist.dev.example.com</a>" would return the CNAME <br>
<a href="http://not-yet-provisioned.example.biz" rel="noreferrer" target="_blank">not-yet-provisioned.example.biz</a>. That is not compliant with RFC 4592 -- <br>
the correct response is NXDOMAIN (there is no name, and there is no <br>
wildcard at the closest encloser, which is the ENT <a href="http://dev.example.com" rel="noreferrer" target="_blank">dev.example.com</a>).<br></blockquote><div><br></div><div>That's very interesting information. I'm pretty sure most of the open</div><div>source DNS implementations get this right though.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Further complicating the issue, for those providers that offered DNSSEC <br>
signing, when DNSSEC was enabled for such a zone, the wildcard matching <br>
behavior became correct (as it had to be, in order for negative proofs to <br>
work) and an NXDOMAIN was returned for "<a href="http://doesnotexist.dev.example.com" rel="noreferrer" target="_blank">doesnotexist.dev.example.com</a>".<br></blockquote><div><br></div><div>Yes, indeed. Although this is unmentioned in my OARC 2015 slides, I did mention</div><div>it during my actual talk, and speculated that implementers would necessarily have</div><div>to fix their ENT response behavior for the same reason you cite (NSEC/NSEC3 </div><div>nodata proofs), and this fact was confirmed by a couple of colleagues from other</div><div>DNS companies at the mic line.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I'd like to hear more about your original methodology for determining <br>
whether a provider correctly supported ENTs or not. The wildcard behavior <br>
is hard to infer, because knowledge of the zone contents is required. In <br>
our case, we actually provisioned a sample zone on other providers to test <br>
the wildcard behavior, and found that of the ones we tested, only Verisign <br>
did the wildcard behavior correctly, at least when we tested this in 2016.<br></blockquote><div><br></div><div>The goal of my survey was more limited than your more elaborate analysis.</div><div>I wasn't specifically looking at confirming correct ENT/wildcard behavior</div><div>across zone contents. I was simply trying to answer the question of whether </div><div>a qname minimizing resolver can successfully resolve (existing) names at the</div><div>apex (or at www.apex) of popular zones. In cases where the names were</div><div>aliased to CDNs/hosters, the qname minimization algorithm often uncovered</div><div>empty-non-terminals in the alias targets that returned NXDOMAIN and prevented</div><div>iterative resolution from proceeding.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
But my larger point was that explaining the concept of wildcards, closest <br>
enclosers, and empty-non-terminals to our customers was a NIGHTMARE. <br>
Customers choose cloud providers specifically so they _don't_ have to be <br>
DNS experts, and it's a non-starter to have a conversation along the lines <br>
of "Well yes, I know your zone works fine on $OTHER_PROVIDER, but you see <br>
there are actually hundreds of invisible records in your zone which are <br>
interfering with your wildcard matching."<br></blockquote><div><br></div><div>I understand your pain.</div><div><br></div><div>Not sure I have any specific suggestions, other than that the DNS community </div><div>could try to work collectively to ensure that all major DNS providers have protocol</div><div>compliant behavior.</div><div><br></div><div>If on the other hand, the DNS evolves in a direction where programmatically </div><div>generated responses are the norm, then many of these types of zone configuration</div><div>dilemmas could be worked around by pretending that wildcards don't exist (as an</div><div>actual DNS protocol element). This would be (somewhat) challenging for the</div><div>pre-computed signature model of DNSSEC, but fairly easy for some of the online</div><div>signers.</div><div><br></div><div>Shumon Huque</div><div><br></div></div></div>