[dns-operations] Akamai now works with ENT (Empty Non-Terminals)?

[Shumon Huque]

On Sun, Apr 14, 2019 at 9:59 AM Jon Reed <jreed at akamai.com> wrote:

> On Sat, 13 Apr 2019, Shumon Huque wrote:
>
> > Now that you've deployed the fix, are you able to share how you
> > addressed the customer expectation problem that you describe? e.g. Did
> > you advise customers not to deploy CNAME records before they are fully
> > provisioned?
>
> The problem I described in the dnsop post wasn't our customers, it was our
> customers' customers.   It put us in the position of having to tell a
> large SaaS provider that _their_ customers were doing the wrong thing, and
> they obviously were not very amenable to hearing that.   In fact, in one
> case, they did move to a competitor that still had the incorrect wildcard
> matching behavior.
>

Ah, thanks for that additional clarification .. it makes more sense now.

> Did you deploy special case code in your nameservers for
> > these services to match wildcards above or parallel to the ENTs, while
> > providing NODATA responses to the ENTs? Or something else?
>
> Yes.  We did an exhaustive survey of our zones, tried to infer customer
> intent, worked with our customers where we could, and when we couldn't,
> were forced to develop a new feature that operates in a similar manner as
> the one described.  (I'm not sure how detailed I can get.)
>

Got it. Thanks for confirming this level of detail at least.

>
> > At that time, Akamai and Cloudflare were the main problematic services
> in a survey of the Alexa top 1000 sites. I notified both of them well
> before my talk, and Cloudflare had already fixed their ENT
> > response behavior by the time of the talk. There were several Chinese
> CDN providers on the list too - I managed to contact a couple of them, and
> they informed me they were working on fixes, but I didn't
> > follow up and check on their status.
>
> I don't want to get into finger-pointing, but I think the wildcard
> matching issue we encountered was is subtler and harder to detect than the
> examples given in the slides.
>

Yes, agreed (my survey on the other hand wasn't specifically looking
at the wildcard matching issue).


> Consider the following zone in its entirety:
>
> example.com         IN    SOA    [omitted for brevity]
> example.com         IN    NS     ns1.example.com.
> *.example.com       IN    CNAME  not-yet-provisioned.example.biz
> www.dev.example.com IN    A      192.168.0.1
>
> When we surveyed other providers at the start of this rollout in 2016, we
> found that a query for "dev.example.com" would correctly return an ENT
> for
> nearly all of them, which is consistent with your results.
>
> However, for _all but one_ of the providers we surveyed, a query for
> "doesnotexist.dev.example.com" would return the CNAME
> not-yet-provisioned.example.biz.  That is not compliant with RFC 4592 --
> the correct response is NXDOMAIN (there is no name, and there is no
> wildcard at the closest encloser, which is the ENT dev.example.com).
>

That's very interesting information. I'm pretty sure most of the open
source DNS implementations get this right though.

Further complicating the issue, for those providers that offered DNSSEC
> signing, when DNSSEC was enabled for such a zone, the wildcard matching
> behavior became correct (as it had to be, in order for negative proofs to
> work) and an NXDOMAIN was returned for "doesnotexist.dev.example.com".
>

Yes, indeed. Although this is unmentioned in my OARC 2015 slides, I did
mention
it during my actual talk, and speculated that implementers would
necessarily have
to fix their ENT response behavior for the same reason you cite (NSEC/NSEC3
nodata proofs), and this fact was confirmed by a couple of colleagues from
other
DNS companies at the mic line.

I'd like to hear more about your original methodology for determining
> whether a provider correctly supported ENTs or not.  The wildcard behavior
> is hard to infer, because knowledge of the zone contents is required.  In
> our case, we actually provisioned a sample zone on other providers to test
> the wildcard behavior, and found that of the ones we tested, only Verisign
> did the wildcard behavior correctly, at least when we tested this in 2016.
>

The goal of my survey was more limited than your more elaborate analysis.
I wasn't specifically looking at confirming correct ENT/wildcard behavior
across zone contents. I was simply trying to answer the question of whether
a qname minimizing resolver can successfully resolve (existing) names at the
apex (or at www.apex) of popular zones. In cases where the names were
aliased to CDNs/hosters, the qname minimization algorithm often uncovered
empty-non-terminals in the alias targets that returned NXDOMAIN and
prevented
iterative resolution from proceeding.

But my larger point was that explaining the concept of wildcards, closest
> enclosers, and empty-non-terminals to our customers was a NIGHTMARE.
> Customers choose cloud providers specifically so they _don't_ have to be
> DNS experts, and it's a non-starter to have a conversation along the lines
> of "Well yes, I know your zone works fine on $OTHER_PROVIDER, but you see
> there are actually hundreds of invisible records in your zone which are
> interfering with your wildcard matching."
>

I understand your pain.

Not sure I have any specific suggestions, other than that the DNS community
could try to work collectively to ensure that all major DNS providers have
protocol
compliant behavior.

If on the other hand, the DNS evolves in a direction where programmatically
generated responses are the norm, then many of these types of zone
configuration
dilemmas could be worked around by pretending that wildcards don't exist
(as an
actual DNS protocol element). This would be (somewhat) challenging for the
pre-computed signature model of DNSSEC, but fairly easy for some of the
online
signers.

Shumon Huque
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20190414/679dcb01/attachment.html>