[dns-operations] APNIC's in-addr.arpa zones were bogus

[Arth Paulite]

Hi all,

Thank you for reporting this issue.

We had a DNSSEC KSK rollover event yesterday for all APNIC IPv4 and IPv6 blocks.  It's part of our annual DNSSEC rollover where we change DS record of IPv4 and IPv6 blocks in the parent zone to a pre-published DNSKEY.

We apologise if this caused an outage for some of you.  We will perform another KSK rollover on a test zone to see if we can reproduce this issue and prevent this from happening in the future.  We will also announce our rollover event here in the future.

Below are some historical analysis result from dnsviz.net within that period but didn’t show DNSSEC failures.

2022-08-25 01:38:31 UTC https://dnsviz.net/d/1.in-addr.arpa/YwbSlw/dnssec/
2022-08-25 07:54:06 UTC  https://dnsviz.net/d/153.in-addr.arpa/Ywcqng/dnssec/


Regards,

Arth Paulite
APNIC – Infrastructure Services Manager


-----Original Message-----
From: Damick, Jeffrey <jdamick at amazon.com>
Date: Friday, 26 August 2022 at 4:42 am
To: Mitsuru SHIMAMURA <simamura at iij.ad.jp>, dns-operations at lists.dns-oarc.net <dns-operations at lists.dns-oarc.net>
Subject: Re: APNIC's in-addr.arpa zones were bogus
We also noticed this change, was this a rollover mistake?  It looks like RRSIG on the SOA expired at around 2022-08-25 03:12 (UTC) which correlates to approximately when we saw the event begin.


On 8/25/22, 11:26 AM, "Mitsuru SHIMAMURA" <simamura at iij.ad.jp<mailto:simamura at iij.ad.jp>> wrote:

    Hi,

    I found our DNSSEC validating full service resolver(unbound) prints bellow validation failer logs.

    2022-08-25T12:37:04.808871+09:00 resolver unbound - - - [27541:3] info: validation failure <136.197.63.119.in-addr.arpa. PTR IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 2001:13c7:7002:3000::14 for key 119.in-addr.arpa. while building chain of trust
    2022-08-25T13:50:39.964228+09:00 resolver unbound - - - [27541:5] info: validation failure <148.99.253.202.in-addr.arpa. PTR IN>: no keys have a DS with algorithm ECDSAP256SHA256 from 2001:67c:e0::9 for key 202.in-addr.arpa. while building chain of trust

    Not only 119 and 202.in-addr.arpa zones were bogus, below is list.

    1.in-addr.arpa.
    14.in-addr.arpa.
    27.in-addr.arpa.
    36.in-addr.arpa.
    39.in-addr.arpa.
    42.in-addr.arpa.
    43.in-addr.arpa.
    49.in-addr.arpa.
    58.in-addr.arpa.
    59.in-addr.arpa.
    60.in-addr.arpa.
    61.in-addr.arpa.
    101.in-addr.arpa.
    103.in-addr.arpa.
    106.in-addr.arpa.
    110.in-addr.arpa.
    111.in-addr.arpa.
    112.in-addr.arpa.
    113.in-addr.arpa.
    114.in-addr.arpa.
    115.in-addr.arpa.
    116.in-addr.arpa.
    117.in-addr.arpa.
    118.in-addr.arpa.
    119.in-addr.arpa.
    120.in-addr.arpa.
    121.in-addr.arpa.
    122.in-addr.arpa.
    123.in-addr.arpa.
    124.in-addr.arpa.
    125.in-addr.arpa.
    126.in-addr.arpa.
    150.in-addr.arpa.
    153.in-addr.arpa.
    163.in-addr.arpa.
    171.in-addr.arpa.
    175.in-addr.arpa.
    180.in-addr.arpa.
    182.in-addr.arpa.
    183.in-addr.arpa.
    202.in-addr.arpa.
    210.in-addr.arpa.
    211.in-addr.arpa.
    218.in-addr.arpa.
    219.in-addr.arpa.
    220.in-addr.arpa.
    221.in-addr.arpa.
    222.in-addr.arpa.
    223.in-addr.arpa.

    The last bogus log is logged at 18:45(UTC+9).
    So, we were affected over 6 hours.

    I found the problem after fix.
    And I cannot found dnsviz's analyze at the time.

    Does this outage only affect our network?

    --
    Mitsuru SHIMAMURA <simamura at iij.ad.jp<mailto:simamura at iij.ad.jp>>
    Internet Initiative Japan, Inc.





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20220826/4e65f479/attachment.html>