[dns-operations] Unplanned DLV zone outage on 2009-Apr-06

[Michael Graff]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Early on 6th April, the ZSK for the dlv.isc.org zone underwent one of
its periodic scheduled rollovers. Due to a failure in a key roll script,
the zone was improperly signed. This caused an outage of approximately
four hours from 00:00 UTC to 04:00 UTC on April 6th, 2009. At no time
were the DNS servers serving dlv.isc.org offline.  ISC apologizes for
any inconvenience caused.

Users of DLV may have noticed an increase in network traffic due to
failed resolution retries during the outage.  Our SNS servers, which
serve dlv.isc.org, saw approximately 15,000 queries/sec, though this did
not impact SNS service. However, resolvers using DLV were not able to
validate DNSSEC information via dlv.isc.org during this time.

The problem was caused by the KSK for the dlv.isc.org zone being
inaccessible to the signing process, hence the zone was missing
signatures.  This error was caused in part by a bug in our rolling
script, and was not properly detected by dnssec-signzone, which this
script uses.

The roll script failure has been corrected, and we are implementing
additional procedures and pre-publish tests to prevent similar outages
in the future.  Additionally, dnssec-signzone will have additional
safety checks added in future BIND releases.

DLV is provided by ISC to the community on a Public Benefit basis, and
is funded and resourced internally by ISC.  DLV is monitored closely,
and is currently in Early Production status.  We plan to continue to
test and improve our internal procedures, monitoring and hardware
platform over the next two weeks, at which point the service will be
announced as being in Full Production.

- --Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknaXycACgkQLdqv0r6eD6ZtCQCggxwGqzgObQyGLTKXVKXzAoGm
KNQAn1Nvh7suUT8Lfnvq6fIGm/qDevvh
=fs+S
-----END PGP SIGNATURE-----