[dns-operations] service-now.com DNSSEC broken?

[Michael Sinatra]

On 01/27/17 19:21, Casey Deccio wrote:
>
>> On Jan 27, 2017, at 8:03 PM, David <opendak at shaw.ca> wrote:
>>
>> On 2017-01-27 5:33 PM, Stephan Lagerholm wrote:
>>> 8.8.8.8 as well as Comcast DNS is servfailing without the +cd bit. But
>>> domain comes out clean in the verisign and dnsviz debuggers.
>>>
>>> Can anybody help me figure out what is wrong with it?
>>
>> https://puck.nether.net/pipermail/outages/2017-January/010036.html
>>
>
> Looks like it is intermittent:
>
> http://dnsviz.net/d/service-now.com/e/126998240/dnssec/ (looks good)
> http://dnsviz.net/d/service-now.com/e/127004107/dnssec/ (looks broken)
>
> (Both examples query 8.8.8.8)
>
> Perhaps the DS/DNSKEY inconsistency exists in some of the backend caches of Google and Comcast, but not everywhere.  Even when the correct records (i.e., matching DS/DNSKEY) are returned by 8.8.8.8, SERVFAIL is returned without the CD bit set in the request [1].

In the cases I queried, which led to my post to outages@, 8.8.8.8 
responded with different DS keytags to repeated queries from the same 
source host.  My conclusion is that some Google backend caches had 
expired the old DS record (keytag==30126) and some had not (as is to be 
expected).  Because repeated queries to 8.8.8.8 from the same source 
host create different cache responses, my assumption is that dnsviz 
can't guarantee a response from the same backend cache on two successive 
queries.  So, even if the DS/DNSKEY responses looked the same, I don't 
think we can know that they actually came from the same backend cache.

Doing cache dumps on our internal BIND 9.11 recursive server that was 
giving SERVFAILs shows the cached DS record with keytag of 30126, while 
the only KSK in the DNSKEY had tag 31893.  The TTL on the DS record in 
com is 86400 and the TTL on the DNSKEY record was 7200.  This feels like 
a simple failure to respect the DS record TTL--i.e. a botched KSK roll.

"Interestingly,"** it appears that there have been two recent KSK rolls 
for service-now.com.  For many months the primary (i.e. referenced in 
the parent DS and signing the ZSKs) KSK had tag 42762, with 30126 being 
published but not signing anything.  During the month of January, 30126 
was rolled in as primary around 1/13/17, only to be hastily (and 
improperly) rolled again 2 weeks later (making 31983 the primary).  Not 
sure if something went wrong with 30126 or if a 2-week KSK lifetime was 
intended.

michael

** I put "interestingly" in quotes because a friend of mine told me 
several years ago that whenever a tech person puts "interestingly" in a 
sentence, what follows is rarely actually interesting to normal people.