[dns-operations] ARIN DNSSEC outage on 65.in-addr.arpa?

[Sadiq Saif]

On 3/8/2016 16:27, Doug Barton wrote:
> On another list I heard that there was a DNSSEC outage on
> 65.in-addr.arpa related to expired signatures, which was since remedied.
> Does anyone have information on that?
> 
> Doug

Nate Davis from ARIN posted this in ARIN-PPML:
http://lists.arin.net/pipermail/arin-ppml/2016-March/030726.html

-------- Forwarded Message --------
Subject: Re: [arin-ppml] Just so it is recorded here (DNSSEC.. ) outages
today..
Date: Tue, 8 Mar 2016 17:59:10 +0000
From: Nate Davis <ndavis at arin.net>
To: Chris Woodfield <chris at semihuman.com>, Christopher Morrow
<christopher.morrow at gmail.com>
CC: arin-ppml at arin.net <arin-ppml at arin.net>


ARIN's DNS process moves DNS data from the internal database to a Secure64
DNSSEC appliance to a hidden distribution master. From the hidden
distribution
master, zones are fetched to name server constellations from ARIN,
VeriSign, and PCH.

About two weeks ago a script was run that reset the serial on a zone in
the database. This script was run to accommodate an inter-RIR network
transfer, and is not executed during the normal course of operations. It
reset the serial in our database in an unexpected way, and consequently
zone transfers from the Secure64 to our distribution master did not occur.

This script was cumbersome and error prone, and had already been
identified to be replaced in the upcoming, planned deployment this weekend.

This incident exposed a gap in our monitoring that we are fixing. Our
current, legacy monitoring system does not adequately identify the serial
number inconsistencies between the DNS nodes, nor does it adequately
handle issues with DNSSEC signature validation. We have work underway to
replace our old monitoring system with a new system that solves these
problems.

This update is being posted to both arin-ppml and arin-tech-discuss. To
avoid non-policy related discussion on PPML, we encourage follow up
discussion
on arin-tech-discuss, a public mailing list that ARIN¹s engineering team
monitors. For those not
familiar with arin-tech-discuss, please subscribe here:
http://lists.arin.net/mailman/listinfo/arin-tech-discuss

Regards,

Nate Davis
-- 
Sadiq Saif (AS393949)
https://staticsafe.ca