[dns-operations] Second set of eyes on www.brocade.com or lldns.net

[Frank Bulk]

My monitoring of HTTPv6 access to http://www.brocade.com has been showing
the site to go up and down.  Further investigation revealed that our
monitoring server is regularly not getting any AAAA for www.brocade.com.  

Several times over the weekend I checked all our DNS servers (three
corporate on Windows, two upstream, and seven ISP on ISC) and it's just two
of our ISP servers are showing this symptom. 

www.brocade.com has a CNAME to brocade.cust.lldns.net (LimeLight, a CDN
Brocade's website recently moved to), and that always works, but the CNAME
doesn't always return the AAAA.  Note that the A always returns.

I checked both www.brocade.com and brocade.cust.lldns.net against a few DNS
checking sites, but nothing popped up.  

Am hitting perhaps a faulty DNS server behind a load-balancer at
ns1.lldns.net and ns2.lldns.net, and our two server IPs are hashed to use
that from time to time?  Any other ideas?

============================================
DNS server: 96.31.0.5

; <<>> DiG 9.7.3 <<>> AAAA www.brocade.com @96.31.0.5
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10727
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;www.brocade.com.               IN      AAAA

;; ANSWER SECTION:
www.brocade.com.        274     IN      CNAME   brocade.cust.lldns.net.

;; AUTHORITY SECTION:
lldns.net.              2794    IN      SOA     lldns.net. root.lldns.net.
2010041901 600 300 604800 10

;; Query time: 0 msec
;; SERVER: 96.31.0.5#53(96.31.0.5)
;; WHEN: Mon Oct  5 17:10:50 2015
;; MSG SIZE  rcvd: 110

============================================
DNS server: 96.31.0.8

; <<>> DiG 9.7.3 <<>> AAAA www.brocade.com @96.31.0.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1753
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;www.brocade.com.               IN      AAAA

;; ANSWER SECTION:
www.brocade.com.        826     IN      CNAME   brocade.cust.lldns.net.

;; AUTHORITY SECTION:
lldns.net.              3582    IN      SOA     lldns.net. root.lldns.net.
2010041901 600 300 604800 10

;; Query time: 0 msec
;; SERVER: 96.31.0.8#53(96.31.0.8)
;; WHEN: Mon Oct  5 17:10:50 2015
;; MSG SIZE  rcvd: 110

============================================
DNS server: 96.31.0.5

; <<>> DiG 9.7.3 <<>> AAAA brocade.cust.lldns.net @96.31.0.5
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10590
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;brocade.cust.lldns.net.                IN      AAAA

;; AUTHORITY SECTION:
lldns.net.              2744    IN      SOA     lldns.net. root.lldns.net.
2010041901 600 300 604800 10

;; Query time: 0 msec
;; SERVER: 96.31.0.5#53(96.31.0.5)
;; WHEN: Mon Oct  5 17:11:40 2015
;; MSG SIZE  rcvd: 81

============================================
DNS server: 96.31.0.8

; <<>> DiG 9.7.3 <<>> AAAA brocade.cust.lldns.net @96.31.0.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40126
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;brocade.cust.lldns.net.                IN      AAAA

;; AUTHORITY SECTION:
lldns.net.              3532    IN      SOA     lldns.net. root.lldns.net.
2010041901 600 300 604800 10

;; Query time: 0 msec
;; SERVER: 96.31.0.8#53(96.31.0.8)
;; WHEN: Mon Oct  5 17:11:40 2015
;; MSG SIZE  rcvd: 81

============================================