<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 11, 2019 at 4:12 PM Mark Kosters <<a href="mailto:markk@arin.net">markk@arin.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Everyone<br>
<br>
John Curran posted the message below on NANOG.<br>
<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:verdana,sans-serif">While I'm often critical of ARIN, I would like to publicly acknowledge and thank ARIN for providing a good postmortem. Publishing a clear (and technical!) postmortem goes a long way towards restoring trust in a system / organization -- 'tis easy to screw something up, but takes courage to explain what it was.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">So, thank you again, and well done.</div><div class="gmail_default" style="font-family:verdana,sans-serif">W</div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Thanks,<br>
Mark<br>
<br>
Folks - <br>
<br>
The <a href="http://ARIN.NET" rel="noreferrer" target="_blank">ARIN.NET</a> zone on our public signed DNS servers are populated via an internal DNS server and associated workflow. As part of system maintenance near the end of 2018, the zone file used by the master internal DNS server was updated incorrectly, resulting in an invalid zone file. Since the zone file was invalid, the zone did not reload on our internal master, and the associated workflow to DNSSEC sign and push this zone to the public servers did not execute. Our monitoring systems reported being green until the signatures expired as they presently check that the SOA's match on the internal and external nameservers. <br>
<br>
At approximately 8:30AM eastern time today (11 January 2019), ARIN operations started seeing issues within its monitoring. Initial review suggested the problem was DNSSEC-related due to expired signatures. We pulled the DS record from the zone so that DNSSEC validation would not be performed by those validating resolvers that had not already cached our DS records. Upon further investigation we determined that it was the result of human error in editing a zone file that went undetected and resulted in interruption of our routine zone publication process. The issue was fixed and signed zones where then pushed out at 10:25 AM ET. The DS record was reinstated in the parent at 10:30AM ET.<br>
<br>
As a result of this incident, we will add additional alerting to the zone loading process for any errors and perform monitoring of zone signature lifetimes, with appropriate alerting for any potential expiration of DNSSEC signatures. <br>
<br>
My apologies for this incident – while ARIN does have some fragility in our older systems (which we have been working aggressively to phase out via system refresh and replacements), it is not acceptable to have this situation with key infrastructure such as our DNS zones. We will prioritize the necessary alert and monitor changes and I will report back to the community once that has been completed.<br>
<br>
Thank you for your patience in this regard.<br>
/John<br>
<br>
John Curran<br>
President and CEO<br>
American Registry for Internet Numbers<br>
<br>
<br>
<br>
<br>
On 1/11/19, 10:30 AM, "dns-operations on behalf of Stephane Bortzmeyer" <<a href="mailto:dns-operations-bounces@dns-oarc.net" target="_blank">dns-operations-bounces@dns-oarc.net</a> on behalf of <a href="mailto:bortzmeyer@nic.fr" target="_blank">bortzmeyer@nic.fr</a>> wrote:<br>
<br>
<a href="https://zonemaster.fr/result/b9a10c4558449ffe" rel="noreferrer" target="_blank">https://zonemaster.fr/result/b9a10c4558449ffe</a><br>
<br>
<a href="http://dnsviz.net/d/arin.net/XDiYGA/dnssec/" rel="noreferrer" target="_blank">http://dnsviz.net/d/arin.net/XDiYGA/dnssec/</a><br>
<br>
<br>
% dig NS <a href="http://arin.net" rel="noreferrer" target="_blank">arin.net</a> <br>
<br>
; <<>> DiG 9.10.3-P4-Debian <<>> NS <a href="http://arin.net" rel="noreferrer" target="_blank">arin.net</a><br>
;; global options: +cmd<br>
;; Got answer:<br>
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64645<br>
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1<br>
<br>
;; OPT PSEUDOSECTION:<br>
; EDNS: version: 0, flags: do; udp: 4096<br>
;; QUESTION SECTION:<br>
;<a href="http://arin.net" rel="noreferrer" target="_blank">arin.net</a>. IN NS<br>
<br>
;; ANSWER SECTION:<br>
<a href="http://arin.net" rel="noreferrer" target="_blank">arin.net</a>. 43196 IN NS <a href="http://u.arin.net" rel="noreferrer" target="_blank">u.arin.net</a>.<br>
<a href="http://arin.net" rel="noreferrer" target="_blank">arin.net</a>. 43196 IN NS <a href="http://ns1.arin.net" rel="noreferrer" target="_blank">ns1.arin.net</a>.<br>
<a href="http://arin.net" rel="noreferrer" target="_blank">arin.net</a>. 43196 IN NS <a href="http://ns3.arin.net" rel="noreferrer" target="_blank">ns3.arin.net</a>.<br>
<a href="http://arin.net" rel="noreferrer" target="_blank">arin.net</a>. 43196 IN NS <a href="http://ns2.arin.net" rel="noreferrer" target="_blank">ns2.arin.net</a>.<br>
<a href="http://arin.net" rel="noreferrer" target="_blank">arin.net</a>. 43196 IN RRSIG NS 5 2 43200 (<br>
20190111131206 20181228121206 11986 <a href="http://arin.net" rel="noreferrer" target="_blank">arin.net</a>.<br>
gAdeDWU4F1fHkST9qHC7JcfVFEb5Nmnsewq9DGQM/w8K<br>
KrXdDoG0QIOn2Acdi9m1bW+j4UgU2yDPnIMZoHHV6BoK<br>
3C6+IEooWRBbEYDEMhrr42zAjlkH/P0jnYKvoBSH99aW<br>
RA76gSiXRIBe+a3C2SuwEu/r4LoBoDA7KuIayGQ= )<br>
<br>
;; Query time: 0 msec<br>
;; SERVER: ::1#53(::1)<br>
;; WHEN: Fri Jan 11 15:36:48 CET 2019<br>
;; MSG SIZE rcvd: 275<br>
_______________________________________________<br>
dns-operations mailing list<br>
<a href="mailto:dns-operations@lists.dns-oarc.net" target="_blank">dns-operations@lists.dns-oarc.net</a><br>
<a href="https://lists.dns-oarc.net/mailman/listinfo/dns-operations" rel="noreferrer" target="_blank">https://lists.dns-oarc.net/mailman/listinfo/dns-operations</a><br>
dns-operations mailing list<br>
<a href="https://lists.dns-oarc.net/mailman/listinfo/dns-operations" rel="noreferrer" target="_blank">https://lists.dns-oarc.net/mailman/listinfo/dns-operations</a><br>
<br>
<br>
<br>
_______________________________________________<br>
dns-operations mailing list<br>
<a href="mailto:dns-operations@lists.dns-oarc.net" target="_blank">dns-operations@lists.dns-oarc.net</a><br>
<a href="https://lists.dns-oarc.net/mailman/listinfo/dns-operationsdns-operations" rel="noreferrer" target="_blank">https://lists.dns-oarc.net/mailman/listinfo/dns-operations<br>
dns-operations</a> mailing list<br>
<a href="https://lists.dns-oarc.net/mailman/listinfo/dns-operations" rel="noreferrer" target="_blank">https://lists.dns-oarc.net/mailman/listinfo/dns-operations</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">I don't think the execution is relevant when it was obviously a bad idea in the first place.<br>This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants.<br> ---maf</div></div>