<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Hi all, </div>
<div><br>
</div>
<div>
<div>Further to my previous email, we have now completed our root cause analysis. Please see the following service announcement for the details. </div>
<div><br>
</div>
<div><a href="https://www.apnic.net/publications/news/2016/service-announcement-15-march-2016">https://www.apnic.net/publications/news/2016/service-announcement-15-march-2016</a></div>
</div>
<div><br>
</div>
<div>Thanks again for your patience and apologies for the inconvenience this has caused.</div>
<div><br>
</div>
<div>Anton</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family: Calibri; font-size: 11pt; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);">
<span style="font-weight: bold;">From: </span>Anton Strydom <<a href="mailto:anton@apnic.net">anton@apnic.net</a>><br>
<span style="font-weight: bold;">Date: </span>Wednesday, 16 March 2016 11:41 am<br>
<span style="font-weight: bold;">To: </span>"<a href="mailto:apnic-talk@lists.apnic.net">apnic-talk@lists.apnic.net</a>" <<a href="mailto:apnic-talk@lists.apnic.net">apnic-talk@lists.apnic.net</a>><br>
<span style="font-weight: bold;">Subject: </span>Update on APNIC IPv4 reverse DNS zones validation<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>Hi all,</div>
<div><br>
</div>
<div>
<div>I wanted to provide an update on the problem which affected all APNIC IPv4 reverse DNS zones.</div>
<div><br>
</div>
<div>APNIC was notified via the public DNS operations mailing lists on Wednesday, 16/03 at 00:30 UTC+10 of a DNSSEC problem which affected all APNIC IPv4 reverse DNS zones. </div>
</div>
</div>
</span>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
During the reverse zone updates, which happened on Tuesday, 15/03 at 09:15 UTC+10, incorrect DS records were used. Once we became aware of the problem, the APNIC network operations team began working on resolving the issue immediately and all zones were corrected
by Wednesday, 16/03 at 02:40 UTC+10. Due to DNS cache effects, the old cached data may take time to expire in the public DNS and be replaced with the new information.</div>
</span><span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<div>APNIC do have testing scripts to check for this kind of operational problem; however, in this instance the checks did not detect the issue. The team is currently investigating the root cause of this monitoring failure.</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div></div>
</div>
</span>
<div><br>
</div>
</div>
</span><span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>
<div>We would like to thank the operations community for reporting the issue to us. We recognise this is a significant service outage.</div>
<div><br>
</div>
<div>A more detailed service announcement is being compiled and I'll email the list again with that information once it is completed. Thanks again for your patience and apologies for the inconvenience this outage caused.</div>
<div><br>
</div>
<div>Regards</div>
<div><br>
</div>
<div>Anton</div>
<div><br>
</div>
</div>
<div>Technical Services Director</div>
<div>APNIC</div>
<div><br>
</div>
</div>
</span></div>
</div>
</span></div>
</body>
</html>