[dns-operations] Signing of the ARPA zone

[Simon Leinen]

Joe Abley writes:
> ARPA was added to the IANA ITAR shortly before 2200 Pacific time last
> night (2010-03-25 0500 UTC).

...and our automatic ITAR monitoring system detected the new version on
25 Mar 2010 05:00:10 UTC (wow fast), and a colleague imported the
trusted keys from the ITAR on 25 Mar 2010 09:19 UTC.

And then almost immediately, inverse lookups started to fail on one of
our recursive nameservers, running BIND 9.7.0 (just like the others).
This is an extract from the log (timestamps in UTC) of that nameserver:

25-Mar-2010 09:19:00.934 general: info: received control channel command 'reconfig'
25-Mar-2010 09:19:00.934 general: info: loading configuration from '/etc/bind/named.conf'
25-Mar-2010 09:19:00.937 general: info: reading built-in trusted keys from file '/etc/bind/bind.keys'
25-Mar-2010 09:19:00.939 general: info: using default UDP/IPv4 port range: [1024, 65535]
25-Mar-2010 09:19:00.940 general: info: using default UDP/IPv6 port range: [1024, 65535]
25-Mar-2010 09:19:00.962 general: info: reloading configuration succeeded
25-Mar-2010 09:19:00.962 general: info: any newly configured zones are now loaded
25-Mar-2010 09:19:28.299 dnssec: info:   validating @0x7f3734f74430: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:19:49.050 dnssec: info:   validating @0x7f373455a1f0: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:20:32.155 dnssec: info:   validating @0x2b0d870: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:21:07.598 dnssec: info:   validating @0x7f3734feeaf0: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:21:24.171 dnssec: info:   validating @0x7f373cacd090: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:21:25.057 dnssec: info:   validating @0x7f373455a1f0: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:21:35.780 dnssec: info:   validating @0x7f373c03dc70: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:22:07.005 dnssec: info:   validating @0x7f3734feeaf0: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:23:29.555 dnssec: info:   validating @0x7f37340ad010: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:24:26.295 dnssec: info:   validating @0x7f373d4e1060: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:25:23.595 dnssec: info:   validating @0x7f373455f2e0: arpa SOA: got insecure response; parent indicates it should be secure
25-Mar-2010 09:27:13.272 dnssec: info:   validating @0x2fb6e90: arpa SOA: got insecure response; parent indicates it should be secure
[...]

Presumably this is the same problem that Chris Thompson mentioned in
this thread (<Prayer.1.3.2.1003251608280.12180 at hermes-2.csi.cam.ac.uk>).
As in Chris' case, only one of our recursive nameservers was affected.

This caused all kind of breakage in our IT infrastructure: NFS servers
refusing to honor mount requests, mail servers falsely rejecting mail
from local machines, various sysadmins breathing down my neck, etc.

Whatever it was, I hope this doesn't happen again.

Like Chris, I didn't take a cache dump in time to find the culprit.
Actually I was only a little too slow, because a colleague had fixed the
problem by flushing the cache two minutes before I dumped it... oh well.

So does anybody have an explanation on how old information in the cache
(or another inconsistency) can have caused this?

And if so, could this have been prevented by the phase-in procedure of
DNSSEC for .ARPA? (And if so, how?)
-- 
Simon.