[dns-operations] Edge-case, zero-length DNSKEYs

[Brian Somers]

On Oct 6, 2020, at 4:20 PM, Viktor Dukhovni <ietf-dane at dukhovni.org> wrote:
> -- | Compute an RFC 4034 Appendix B key tag over the DNSKEY RData: 16 bit flags,
> -- 8 bit proto, 8 bit alg and key octets.
> --
> -- With the obsolete algorithm 1 we assign key tag 0 to truncated keys, but
> -- RSAMD5 keys are no longer seen in the wild.  We check that the modulus
> -- actually has at least 3 octets.

Interesting - I had to visit this section of RFC 4034 a couple of weeks ago.
Courtesy of https://dnsthought.nlnetlabs.nl/res_AS36692/#rsamd5 it turns
out that it’s necessary to be able to calculate the keytag for RSAMD5 keys
to avoid failing validation due to an RRSIG/DNSKEY matchup failure.

The calculation was hindered by the RFC saying:

   The key tag for algorithm 1 (RSA/MD5) is defined differently from the
   key tag for all other algorithms, for historical reasons.  For a
   DNSKEY RR with algorithm 1, the key tag is defined to be the most
   significant 16 bits of the least significant 24 bits in the public
   key modulus (in other words, the 4th to last and 3rd to last octets
   of the public key modulus).

The piece before the parenthesis is correct.  The piece in parenthesis
is blatantly wrong :(

I ended up with the same code - the modulus must be at least 3 bytes,
otherwise the DNSKEY RR is junk…. Ignored during iteration but used
during DNSKEY RRset validation.

—
Brian