[dns-operations] DNSSEC outage in ripe.net and 0.a.2.ip6.arpa

[Wolfgang Nagele]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear colleagues,

As some of you have noticed we had another DNSSEC outage last week. The zones
affected were:

ripe.net:       11:29 - 16:00 UTC on 14 April
0.a.2.ip6.arpa: 02:31 - 10:00 UTC on 15 April

After analysis with our vendor, we determined that the cause of this outage was
the same bug that caused the outage in e164.arpa on 15 February 2011.

Our vendor concluded that the bug on 15 February was caused by an unusually high
load on the signer system, but this time the system was in normal day-to-day
operation, so that can't explain the failure.

We've collected a sufficient amount of data from this incident to allow us to
reproduce the circumstances and have found the bug in the system together with
our vendor. We will receive an updated version of the software within the coming
weeks. We have agreed to this timeline because this bug is only triggered in
specific circumstances during a Key Signing Key rollover.

We apologise for this outage. I would like to use the opportunity to point out
that our long-term mitigation plan is to have a DNSSEC verification
proxy in place. I am happy to say that our efforts for this have been
well-received and a group of other interested parties has formed to work on it.

If you would like to join the mailing list, please see:
http://nlnetlabs.nl/mailman/listinfo/dnssexy

Regards,

Wolfgang Nagele
DNS Group Manager
RIPE NCC
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2vytoACgkQjO7G63Byy8eKfACgs7HMEleAz0pEHIe03npMqUG6
xB4AoLBYtGOYyrk3X2VPOVjcsmpHIIIG
=NFDn
-----END PGP SIGNATURE-----