[dns-operations] DNS .com/.net resolution problems in the Asia/Pacific region

[Christian Elmerot]

On 2023-07-12 05:50, Viktor Dukhovni wrote:
> On Tue, Jul 11, 2023 at 10:51:47PM -0400, Viktor Dukhovni wrote:
> 
>> In .COM CZDS zone file snapshot of .COM from ~midnight UTC 2023-07-11
>> the range of non-apex RRSIG inception times was:
>>
>>      20230707025004 – 20230710225021
>>
>> With corresponding expiration times:
>>
>>      20230714040004 – 20230718000021
>>
>> With expiration of the oldest RRSIGS 3 days and 4 hours away, and the
>> newest a full 7 days.
> 
> Apart from some records that are signed intra-day, the expiration times
> of records in .COM are strongly clustered around once a day signing
> events that cover roughly 25% of the zone.  For example, the CZDS
> snapshot for the 11th has expiration times clustered near:
> 
>      2023-07-14T04:00 ~3.4M RRsets
>      2023-07-15T04:00 ~3.4M RRsets
>      2023-07-16T04:00 ~3.4M RRsets
>      2023-07-17T04:00 ~3.4M RRsets
> 
> So the affected delegations would have been ~0%, ~25%, ~50%, ~75% or
> ~100% of the zone, depending on how many days the issue went unnoticed.

This is very much in line with what we would have expected given a 
gradual increase in errors related to expired RRSIGs from the start of 
the incident. Initially the errors we recorded for CNAME resolutions 
were low but gradually increased to the point where it was affecting 
more and more of our customers. It also explains why not all .com/.net 
zones failed to resolve.

/Christian