From borjam at sarenet.es Thu Oct 1 08:25:24 2026 From: borjam at sarenet.es (Borja Marcos) Date: Thu, 1 Oct 2026 10:25:24 +0200 Subject: Anything going on with Hewlett-Packard? Message-ID: <9EAEC493-694F-4BBF-B647-02289836DDDA@sarenet.es> Since yesterday around 10:00 UTC I am seeing lots of dropped UDP queries to their authoritative servers for hpe.com . I have forced my recursive servers to use only TCP when querying their authoritatives. And I see some errors querying Google?s DNS for hpe.com or too many responses with a TTL of zero. Anyone from HPE here? Borja. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 873 bytes Desc: Message signed with OpenPGP URL: From walists at mailbox.org Mon Oct 5 13:02:22 2026 From: walists at mailbox.org (Winfried) Date: Mon, 5 Oct 2026 15:02:22 +0200 Subject: [dns-operations] broken 'apex' NSEC3 for ma. denies big number of delegated names In-Reply-To: <6c6900f5d10e5a3cb882d97eacf315b4adf9cacb.camel@powerdns.com> References: <6c6900f5d10e5a3cb882d97eacf315b4adf9cacb.camel@powerdns.com> Message-ID: <7ddd3ce7-0946-4d01-92e1-e4366058ce5d@mailbox.org> Hello, This problem has recurred (or still persists). We are seeing repeated NXDOMAIN responses for names that appear to exist in the ma. zone. This does not seem to affect just one name. The signed NSEC3 data in NXDOMAIN responses can cover the hash of an existing name. Resolvers using aggressive NSEC caching may then use that data to generate further NXDOMAIN responses. One example is bankalmaghrib.ma.: - For a query for horizonalmaz.ma. A, authoritative nameservers return a signed NSEC3 record covering the range MFB8... ? 4TEST... This range includes the hash of bankalmaghrib.ma. (QH1M...). - A direct query for bankalmaghrib.ma. NS returns the delegation to four Azure DNS nameservers. It also returns an NSEC3 record for the exact hash QH1M..., listing the NS record type. - A resolver cached the first NSEC3 proof while processing horizonalmaz.ma. A, then used it to generate an NXDOMAIN response for bankalmaghrib.ma. SOA. During our tests, the SOA serial numbers differed between authoritative 'ma.' nameservers. This may indicate that they are serving different versions of the ma. zone and could be related to the inconsistent NSEC3 responses. As a separate observation, some ma. nameservers (a.tld.ma, b.tld.ma, c.tld.ma, d.tld.ma) are not reachable over IPv6, although their names have AAAA records. But that's not the main issue. We have already sent a report to ma at anrt.ma. No response so far. Does anyone happen to know a better contact? Kind regards, Winfried > Hello, > > "dig +dnssec txt @b.tld.ma ma." (as do other queries) returns a non- > opt-out NSEC3 covering > EPNGG6KIP1BA66LLJKNBLULI9PUL8OJ9...LB0AJ7AEMMSB3B556MI0DC1GFDUGO17E. > This NSEC3 covers a large number of existing delegations in the zone, > denying their existence. > > Resolvers using NSEC/NSEC3 aggressively (as described in RFC9077) then > end up replying NXDOMAIN for names covered by that range. > > One example of such a name is afmagroup.ma, which hashes to > EUKB329VQPC6CRF4VLLEB9BALQU169UO, falling inside the EPN..LB0 range. > > I suspect your signer is broken. The offending NSEC3 is returned by all > .ma name servers I can find. > > Can you please investigate? Thank you! > > Kind regards, > -- > Peter van Dijk > PowerDNS.COM BV -https://www.powerdns.com/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pawal at amplitut.de Mon Oct 5 13:55:09 2026 From: pawal at amplitut.de (Patrik Wallstrom) Date: Mon, 5 Oct 2026 15:55:09 +0200 Subject: [dns-operations] broken 'apex' NSEC3 for ma. denies big number of delegated names In-Reply-To: References: <6c6900f5d10e5a3cb882d97eacf315b4adf9cacb.camel@powerdns.com> Message-ID: .ma has changed the NSEC3 salt at least once between the reports, so something is happening. But the NSEC3 records still points to the wrong records, and this bad gap is for like half of the delegations. Which is of course bad. So no, it is not a problem with the SOA serials, nor the opt-out. I also have no problems with IPv6. https://gonemaster.evilbit.de/result/NnMCgAgOIkxs / Patrik On Mon, 05 Oct 2026, Winfried via dns-operations wrote: >From: Winfried >Date: Mon, 5 Oct 2026 15:02:22 +0200 >To: dns-operations at lists.dns-oarc.net >Subject: Re: [dns-operations] broken 'apex' NSEC3 for ma. denies big number > of delegated names >User-Agent: Mozilla Thunderbird > >Hello, > >This problem has recurred (or still persists). > >We are seeing repeated NXDOMAIN responses for names that appear to >exist in the ma. zone. This does not seem to affect just one name. The >signed NSEC3 data in NXDOMAIN responses can cover the hash of an >existing name. Resolvers using aggressive NSEC caching may then use >that data to generate further NXDOMAIN responses. > >One example is bankalmaghrib.ma.: > >- For a query for horizonalmaz.ma. A, authoritative nameservers return >a signed NSEC3 record covering the range MFB8... ? 4TEST... This range >includes the hash of bankalmaghrib.ma. (QH1M...). >- A direct query for bankalmaghrib.ma. NS returns the delegation to >four Azure DNS nameservers. It also returns an NSEC3 record for the >exact hash QH1M..., listing the NS record type. >- A resolver cached the first NSEC3 proof while processing >horizonalmaz.ma. A, then used it to generate an NXDOMAIN response for >bankalmaghrib.ma. SOA. > >During our tests, the SOA serial numbers differed between >authoritative 'ma.' nameservers. This may indicate that they are >serving different versions of the ma. zone and could be related to the >inconsistent NSEC3 responses. > >As a separate observation, some ma. nameservers (a.tld.ma, b.tld.ma, >c.tld.ma, d.tld.ma) are not reachable over IPv6, although their names >have AAAA records. But that's not the main issue. > >We have already sent a report to ma at anrt.ma. No response so far. Does >anyone happen to know a better contact? > >Kind regards, >Winfried > >>Hello, >> >>"dig +dnssec txt @b.tld.ma ma." (as do other queries) returns a non- >>opt-out NSEC3 covering >>EPNGG6KIP1BA66LLJKNBLULI9PUL8OJ9...LB0AJ7AEMMSB3B556MI0DC1GFDUGO17E. >>This NSEC3 covers a large number of existing delegations in the zone, >>denying their existence. >> >>Resolvers using NSEC/NSEC3 aggressively (as described in RFC9077) then >>end up replying NXDOMAIN for names covered by that range. >> >>One example of such a name is afmagroup.ma, which hashes to >>EUKB329VQPC6CRF4VLLEB9BALQU169UO, falling inside the EPN..LB0 range. >> >>I suspect your signer is broken. The offending NSEC3 is returned by all >>.ma name servers I can find. >> >>Can you please investigate? Thank you! >> >>Kind regards, >>-- >>Peter van Dijk >>PowerDNS.COM BV -https://www.powerdns.com/ >> >_______________________________________________ >dns-operations mailing list >dns-operations at lists.dns-oarc.net >https://lists.dns-oarc.net/mailman/listinfo/dns-operations From ietf-dane at dukhovni.org Mon Oct 5 14:07:18 2026 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Tue, 6 Oct 2026 01:07:18 +1100 Subject: [dns-operations] broken 'apex' NSEC3 for ma. denies big number of delegated names In-Reply-To: References: <6c6900f5d10e5a3cb882d97eacf315b4adf9cacb.camel@powerdns.com> Message-ID: On Mon, Oct 05, 2026 at 03:02:22PM +0200, Winfried via dns-operations wrote: > This problem has recurred (or still persists). Confirmed, the NSEC3 RR matching the zone apex is much too broad, implausibly covering a large fraction of the hash space. The "TEST" in the "4TEST..." next hash is an interesting coincidence if it is one... ; NXDOMAIN qr rd ra ad do horizonalmaz.ma. IN A ? ma. IN SOA c.tld.ma. ma.anrt.ma. 2039671745 900 90 604800000 1800 MFB8BPVI7H18DJ77DCIKCMDJGHII3D5B.ma. IN NSEC3 1 0 10 4f6e444c70554fbafbb2 4TESTIB4CP7V1199LBC5JAUG3FG5L7BM NS SOA RRSIG DNSKEY NSEC3PARAM RVVP1VSAOPNC2VEO2BARL92RAPR7UN0C.ma. IN NSEC3 1 0 10 4f6e444c70554fbafbb2 S007HCEBT4P1OEQG7Q49FT2TJRKK5TVQ NS 4UKA75NLNGJC3GHJBLLGE7QJB6GD551F.ma. IN NSEC3 1 0 10 4f6e444c70554fbafbb2 4UKI7IER42K636QMF1TJ6NT0A75FQTDU NS $ ldns-nsec3-hash -t 10 -s 4f6e444c70554fbafbb2 ma. mfb8bpvi7h18dj77dcikcmdjghii3d5b. $ ldns-nsec3-hash -t 10 -s 4f6e444c70554fbafbb2 '*.ma' rvvsikok02v87r5s7e960ds4iomven8c. $ ldns-nsec3-hash -t 10 -s 4f6e444c70554fbafbb2 horizonalmaz.ma 4ukculhja1t6h5c3eqfdfkarm63ntl84. > One example is bankalmaghrib.ma.: And indeed that lies in the excluded range: $ ldns-nsec3-hash -t 10 -s 4f6e444c70554fbafbb2 bankalmaghrib.ma qh1m00s34qt97cbgrdmln64s86j39crq. > We have already sent a report to ma at anrt.ma. No response so far. Does anyone > happen to know a better contact? FWIW, the SOA rname is and the DMARC rua/ruf contact email addresses are both . -- Viktor. ?? ????? ???????! From bortzmeyer at nic.fr Mon Oct 5 15:31:07 2026 From: bortzmeyer at nic.fr (Stephane Bortzmeyer) Date: Mon, 5 Oct 2026 17:31:07 +0200 Subject: Anything going on with Hewlett-Packard? In-Reply-To: References: Message-ID: On Thu, Oct 01, 2026 at 10:25:24AM +0200, Borja Marcos via dns-operations wrote a message of 122 lines which said: > Since yesterday around 10:00 UTC I am seeing lots of dropped UDP queries to their authoritative servers for hpe.com . I confirm: % check-soa hpe.com gva01hpeibdns0a.ext.hpecore.net. 16.2.141.20: ERROR: read udp 16.2.141.20:53: i/o timeout ns1.hpe.com. 192.170.12.194: OK: 443244293 ns2.hpe.com. 16.2.148.110: OK: 443244293 ns3.hpe.com. 192.170.11.194: OK: 443244293 ns4.hpe.com. 16.2.132.115: OK: 443244293 ns5.hpe.com. 16.2.141.20: ERROR: read udp 16.2.141.20:53: i/o timeout ns6.hpe.com. 16.2.138.125: ERROR: read udp 16.2.138.125:53: i/o timeout ns7.hpe.com. 192.170.11.194: ERROR: read udp 192.170.11.194:53: i/o timeout ns8.hpe.com. 192.170.12.194: OK: 443244293 ?? hpe.com - DNS grade C (61/100) https://gonemaster.evilbit.de/result/4bE8rWbXtD2p From bortzmeyer at nic.fr Mon Oct 5 15:40:17 2026 From: bortzmeyer at nic.fr (Stephane Bortzmeyer) Date: Mon, 5 Oct 2026 17:40:17 +0200 Subject: broken 'apex' NSEC3 for ma. denies big number of delegated names In-Reply-To: References: <6c6900f5d10e5a3cb882d97eacf315b4adf9cacb.camel@powerdns.com> Message-ID: On Tue, Oct 06, 2026 at 01:07:18AM +1100, Viktor Dukhovni wrote a message of 32 lines which said: > > One example is bankalmaghrib.ma.: > > And indeed that lies in the excluded range: > > $ ldns-nsec3-hash -t 10 -s 4f6e444c70554fbafbb2 bankalmaghrib.ma > qh1m00s34qt97cbgrdmln64s86j39crq. And the problem, as seen by the DNS resolvers of the RIPE Atlas probes: % blaeu-resolve -r 100 --type CNAME www.bankalmaghrib.ma [www.bkam.ma.] : 92 occurrences [ERROR: NXDOMAIN] : 7 occurrences Test #219078014 done at 2026-10-05T15:37:09Z Results on atlas.ripe.net : https://atlas.ripe.net/measurements/219078014 From ietf-dane at dukhovni.org Tue Oct 6 03:11:37 2026 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Tue, 6 Oct 2026 14:11:37 +1100 Subject: [dns-operations] broken 'apex' NSEC3 for ma. denies big number of delegated names In-Reply-To: References: <6c6900f5d10e5a3cb882d97eacf315b4adf9cacb.camel@powerdns.com> Message-ID: On Tue, Oct 06, 2026 at 01:07:19AM +1100, Viktor Dukhovni wrote: > > This problem has recurred (or still persists). > > Confirmed, the NSEC3 RR matching the zone apex is much too broad, > implausibly covering a large fraction of the hash space. The "TEST" in > the "4TEST..." next hash is an interesting coincidence if it is one... Still present, non-recursive direct queries[1] to the "c.tld.ma" auth server return: $ hsdig -R -n c.tld.ma --nocrypto -Dt SRV ma. ; NOERROR qr aa do ma. IN SRV ? ma. IN SOA c.tld.ma. ma.anrt.ma. 2039673474 900 90 604800000 1800 ma. IN RRSIG SOA 8 1 108000 20261105030441 20261006020441 28354 ma. [omitted] MFB8BPVI7H18DJ77DCIKCMDJGHII3D5B.ma. IN NSEC3 1 0 10 4f6e444c70554fbafbb2 4TESTIB4CP7V1199LBC5JAUG3FG5L7BM NS SOA RRSIG DNSKEY NSEC3PARAM MFB8BPVI7H18DJ77DCIKCMDJGHII3D5B.ma. IN RRSIG NSEC3 8 2 1800 20261031124548 20261001114548 28354 ma. [omitted] -- Viktor. ?? ????? ???????! [1] "hsdig" is an unpublished CLI based on the Haskell "dnsbase" stub resolver library. From bortzmeyer at nic.fr Thu Oct 8 14:29:58 2026 From: bortzmeyer at nic.fr (Stephane Bortzmeyer) Date: Thu, 8 Oct 2026 16:29:58 +0200 Subject: Hackers obtain counterfeit TLS certificates for Google and other large services Message-ID: https://arstechnica.com/security/2026/10/hackers-obtain-counterfeit-tls-certificates-for-google-and-other-large-services/ DNSDB show that the servers during the hijack were Cloudflare's. You can see some of the fraudulent certificates at , , . [Don't read the comments of the article if you don't want to risk a heart attack.] From whammyster at gmail.com Sat Oct 10 02:11:30 2026 From: whammyster at gmail.com (Chad Dailey) Date: Fri, 9 Oct 2026 21:11:30 -0500 Subject: [dns-operations] Digicert validation help Message-ID: Long time lurker, first time (?) poster... Looking for a contact at Digicert DNS ops. I'm the DNS admin for a US state govt, and we're having issues with DNS validation for cert renewal. This is causing issues with our websites, and it's not a good time to have that happen (elections, ya know). This also has the attention of our highest executive offices. Need to get this reliably sorted beyond the square dancing we did a few months ago. Drop me a line if you're this person, or have a contact that can actually take action. Thanks a bunch, Chad Dailey official email (because I can't join mailing lists with that account) chad.dailey at nebraska.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at dns-oarc.net Sat Oct 10 19:34:30 2026 From: matt at dns-oarc.net (Matthew Pounsett) Date: Sat, 10 Oct 2026 15:34:30 -0400 Subject: [dns-operations] Upcoming DNSViz Maintenance In-Reply-To: References: Message-ID: DNSViz has been back in full DB-backed operation since around 15:30 UTC yesterday (Friday). Although we didn't have any specific concerns after completing the work, I decided to wait a bit before making a wide announcement, just in case. The maintenance took far longer than expected due to a few unrelated but reinforcing factors: 1. The initial migration took about 18 hours longer than I originally bargained for. Although I did a couple of test dumps prior, a failure of imagination led me to way under-estimate how long it would take to write the database from scratch. I have some ideas for at least getting a better upper-bound estimate the next time we have to do work like this, so that I can set expectations better. 2. After migrating the DB to the updated system we discovered a bad assumption in the code that worked well enough under the older DB but did not hold true after the migration. Casey and I needed time to investigate the issue and come to a consensus on how to fix or work around the problem, which was difficult because... 3. The maintenance was scheduled for just prior to the annual OARC staff/board all-hands week, and Casey has professorial duties. Had the work been completed within 72 hours of when it started this wouldn't have been an issue, but once the work extended beyond that it became very difficult to make headway. Normally I can simply schedule around Casey's availability, but that wasn't possible during the all-hands week. 4. Once we established a plan, testing and validation to make sure we weren't going to corrupt or lose any data also took several days. There were several steps I wanted to take to ensure nothing was left to chance, and on a database with a little over 4 billion rows, each step could take up to four hours to complete. While we don't have the budget for completely downtime-free maintenance, I have several lessons-learned from this that I'll be using in future to minimize interruptions to services like this. Thanks for you patience everyone! -- Matt Pounsett DNS-OARC Systems Engineering -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 870 bytes Desc: not available URL: From list at mens.de Sun Oct 11 09:53:17 2026 From: list at mens.de (Jan-Piet Mens) Date: Sun, 11 Oct 2026 11:53:17 +0200 Subject: [dns-operations] Upcoming DNSViz Maintenance In-Reply-To: References: Message-ID: <20261011-imposing-capuchin-of-improvement-48578f@rabbit> Matthew, >Thanks for you patience everyone! thank *you* for maintaining such an incredibly useful service! DNSViz is an incredibly valuable resource, even in maintenace mode! :-) Best, -JP From marco.davids at sidn.nl Sun Oct 11 10:59:38 2026 From: marco.davids at sidn.nl (Marco Davids (SIDN)) Date: Sun, 11 Oct 2026 12:59:38 +0200 Subject: [dns-operations] Upcoming DNSViz Maintenance In-Reply-To: <20261011-imposing-capuchin-of-improvement-48578f@rabbit> References: <20261011-imposing-capuchin-of-improvement-48578f@rabbit> Message-ID: <76d826a8-bcd6-4de4-949a-fd6f1a6dcb04@sidn.nl> On Sun, 11 Oct 2026 11:53:17 +0200 Jan-Piet Mens wrote: >> Thanks for you patience everyone! > > thank *you* for maintaining such an incredibly useful service! DNSViz is an > incredibly valuable resource, even in maintenace mode! :-) +100...000 ! -- Marco -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4678 bytes Desc: S/MIME-cryptografische ondertekening URL: