[dns-operations] CNAMEs pointing off into the weeds - inconsistent behavior from different recursive codebases

Wed Oct 9 02:06:49 UTC 2019

Dear Colleagues,

As part of refining an Ansible role for handling ACME dns-01 handshakes with LetsEncrypt, I ran into some odd intermittent failures which at first I hypothesized might be a bug in dnspython, which is the back-end for Ansible's lookup('dns' ...) call.

They turned out to be attributable to the fact that I had entries in named.conf on the jumphost pointing to more than one recursive nameserver technology stack and their behaviors were not identical.

The back story here here is that it makes sense to CNAME _acme-challenge.host.example.com records into a zone somewhere that allows updates.  That way I can insert the challenges I receive from the ACME server after I get them with minimal grief.  In the examples shown below, the dynamically updateable zone is "acme.seastrom.com".  I might add that I was slightly surprised that this works - it seems unaddressed in the ACME spec but kind of feels like a potential attack surface tparticularly since it works even to a non-child, non-same-origin (pedantically, not quite "out of baliwick" but YKWIM) zone.

Anyway, early in the execution of the role, I do a query to make sure the CNAME is there.  The qtype is CNAME.  The CNAME record is at that node.  But there is nothing at the node that the right hand side points to.

The expected behavior is that I would get back the CNAME record and NOERROR when asking for a CNAME even if it points somewhere that has no data.  Indeed this is what I get from a recent version of BIND, Power, Google DNS, and Windows 2016 Server.  It seems that Unbound is the outlier here in that it returns *both* an ANSWER section *and* NXDOMAIN, which is behavior I'm not sure I've ever seen before.

Qname minimization is in my standard Unbound config.  This behavior goes away if I turn off qname minimization and restart Unbound, despite the fact that tcpdump offers no evidence that there's actually an attempt being made to crawl the name returned in the right hand side of the original CNAME record regardless of whether qname minimization is on or off. I don't have a BIND-9.13 server handy to test out and see if it's a fundamental failing in my understanding of how qname minimization works, but in any event ANSWER + NXDOMAIN doesn't feel right to me.

Popping back up the stack to the code I was debugging, dnspython sees the NXDOMAIN and returns it as the status.  My role fails (intentionally - I want to bomb out when there is no CNAME).

Comments and thoughts solicited.

Thanks,

-r

unbound 1.6.0 (on armv7l raspberry pi / raspbian not that it probably matters)

maddog:letsencrypt-dns01-example-playbook rs$ dig _acme-challenge.funnel.seastrom.com. cname

; <<>> DiG 9.14.6 <<>> _acme-challenge.funnel.seastrom.com. cname
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 23091
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1400
;; QUESTION SECTION:
;_acme-challenge.funnel.seastrom.com. IN	CNAME

;; ANSWER SECTION:
_acme-challenge.funnel.seastrom.com. 300 IN CNAME _acme-challenge.funnel.seastrom.com.acme.seastrom.com.

;; AUTHORITY SECTION:
acme.seastrom.com.	300	IN	SOA	bifrost.seastrom.com. bug-dns.seastrom.com. 1204 1800 900 7776000 300

;; Query time: 376 msec
;; SERVER: 172.30.250.126#53(172.30.250.126)
;; WHEN: Tue Oct 08 16:58:48 EDT 2019
;; MSG SIZE  rcvd: 171

maddog:letsencrypt-dns01-example-playbook rs$ 

-=-=-=-=-

BIND 9.12.4-P2 (on x86-64 / SmartOS)

[root at a ~]# dig _acme-challenge.funnel.seastrom.com. cname

; <<>> DiG 9.10.1-P1 <<>> _acme-challenge.funnel.seastrom.com. cname
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17616
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_acme-challenge.funnel.seastrom.com. IN        CNAME

;; ANSWER SECTION:
_acme-challenge.funnel.seastrom.com. 300 IN CNAME _acme-challenge.funnel.seastrom.com.acme.seastrom.com.

;; Query time: 1 msec
;; SERVER: 192.148.252.11#53(192.148.252.11)
;; WHEN: Tue Oct 08 22:42:16 UTC 2019
;; MSG SIZE  rcvd: 131

[root at a ~]# 

-=-=-=-=-

powerdns 4.1.1 (unknown platform)

$ dig _acme-challenge.funnel.seastrom.com. cname

; <<>> DiG 9.10.3-P4-Ubuntu <<>> _acme-challenge.funnel.seastrom.com. cname
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23333
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_acme-challenge.funnel.seastrom.com. IN    CNAME

;; ANSWER SECTION:
_acme-challenge.funnel.seastrom.com. 300 IN CNAME _acme-challenge.funnel.seastrom.com.acme.seastrom.com.

;; Query time: 243 msec
;; SERVER: 10.181.35.100#53(10.181.35.100)
;; WHEN: Tue Oct 08 15:24:38 PDT 2019
;; MSG SIZE  rcvd: 119

$ 

-=-=-=-=-

Google DNS (non-open-source codebase and unknown platform)

[root at a ~]# dig @8.8.8.8 _acme-challenge.funnel.seastrom.com. cname

; <<>> DiG 9.10.1-P1 <<>> @8.8.8.8 _acme-challenge.funnel.seastrom.com. cname
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7043
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_acme-challenge.funnel.seastrom.com. IN        CNAME

;; ANSWER SECTION:
_acme-challenge.funnel.seastrom.com. 299 IN CNAME _acme-challenge.funnel.seastrom.com.acme.seastrom.com.

;; Query time: 2 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Oct 08 22:47:50 UTC 2019
;; MSG SIZE  rcvd: 119

[root at a ~]# 

-=-=-=-=-

Windows 2016 Server (presumably on x86)

[root at a ~]# dig @some_v6_addr::3 _acme-challenge.funnel.seastrom.com. cname

; <<>> DiG 9.10.1-P1 <<>> @some_v6_addr::3 _acme-challenge.funnel.seastrom.com. cname
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9399
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;_acme-challenge.funnel.seastrom.com. IN        CNAME

;; ANSWER SECTION:
_acme-challenge.funnel.seastrom.com. 299 IN CNAME _acme-challenge.funnel.seastrom.com.acme.seastrom.com.

;; Query time: 195 msec
;; SERVER: some_v6_addr::3#53(some_v6_addr::3)
;; WHEN: Wed Oct 09 00:34:27 UTC 2019
;; MSG SIZE  rcvd: 119

[root at a ~]# 