[dns-operations] Update on APNIC reverse DNS zones validation

[Anton Strydom]

Hi all,

Further to my previous email, we have now completed our root cause analysis.  Please see the following service announcement for the details.

https://www.apnic.net/publications/news/2016/service-announcement-15-march-2016

Thanks again for your patience and apologies for the inconvenience this has caused.

Anton

From: Anton Strydom <anton at apnic.net<mailto:anton at apnic.net>>
Date: Wednesday, 16 March 2016 11:41 am
To: "apnic-talk at lists.apnic.net<mailto:apnic-talk at lists.apnic.net>" <apnic-talk at lists.apnic.net<mailto:apnic-talk at lists.apnic.net>>
Subject: Update on APNIC IPv4 reverse DNS zones validation

Hi all,

I wanted to provide an update on the problem which affected all APNIC IPv4 reverse DNS zones.

APNIC was notified via the public DNS operations mailing lists on Wednesday, 16/03 at 00:30 UTC+10 of a DNSSEC problem which affected all APNIC IPv4 reverse DNS zones.

During the reverse zone updates, which happened on Tuesday, 15/03 at 09:15 UTC+10, incorrect DS records were used.  Once we became aware of the problem, the APNIC network operations team began working on resolving the issue immediately and all zones were corrected by Wednesday, 16/03 at 02:40 UTC+10.  Due to DNS cache effects, the old cached data may take time to expire in the public DNS and be replaced with the new information.

APNIC do have testing scripts to check for this kind of operational problem; however, in this instance the checks did not detect the issue.  The team is currently investigating the root cause of this monitoring failure.

We would like to thank the operations community for reporting the issue to us.  We recognise this is a significant service outage.

A more detailed service announcement is being compiled and I'll email the list again with that information once it is completed.  Thanks again for your patience and apologies for the inconvenience this outage caused.

Regards

Anton

Technical Services Director
APNIC

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20160317/111d4f93/attachment.html>