[dns-operations] Unplanned DLV zone outage on 2009-Apr-06

[Paul Vixie]

> From: David Conrad <drc at virtualized.org>
> Date: Mon, 6 Apr 2009 22:04:48 -1000
> 
> FWIW, I now expect the root to be signed before the end of the year but
> given the ITAR, I'm not sure how much that actually matters.  TLDs and
> their children are, of course, another issue.

that's extraordinarily great news.  even though not a formal announcement,
the fact that you as a credible insider think that the root may actually
be signed this calendar year is newsworthy in a good way.

> ...
> 
> Note that this isn't a slam against ISC, rather it is a natural outcome
> of the stage of DNSSEC deployment and the limited benefit DNSSEC provides
> to the end user given the current deployment model.

and indeed, some of the non-recent problems that early adopters have
encountered by using DLV are actually DNSSEC problems that would not have
been reached so early except that DLV exposed certain keys to certain
validators.  (noting also that yesterday's problem was due to an ISC
signing error, and the problem two weeks ago was due to an ISC coding bug.)

> > DLV has no such strings.
> 
> Heh. I suspect it safe to say that political strings will attach
> themselves if DLV gets any real traction.

if that happens, we'll do what we always do: cope.

> I personally don't have any specific gripes about ISC's key management
> policies because I don't know what they are.  ...

if ISC can verify a key's correctness, and the zone's ownership, and the
owner's intent to have that key used by validators, then we include that
key in DLV.  (as a counterexample, we do not scrape.)

> ... I obviously know what IANA's are and know where the holes and
> weaknesses are and the steps IANA staff takes to try to address them.
> Equally obviously, I can't say the same about DLV.

i think michael graff's statement here yesterday showed what ISC knows about
holes and weaknesses and steps.  if it's incomplete then please ask specific
questions that will fill in the gaps you know about in your knowledge.  as
to gaps you don't know about and therefore may not know what question to ask,
what would be ISC's best way to fill *those* gaps?  a tech report showing the
whole ISC DLV key management and zone management process, perhaps?

> I know what's in the IANA ITAR and can (if I cared) verify the entire
> contents by hand.  No idea what is in the DLV or who put it there.

several people now walk DLV.ISC.ORG.  (we use NSEC, not NSEC3.)  we can
also open it to AXFR if desired.  there are not supposed to be any secrets
in this stuff.  as to who "put it there", that varies.  the ITAR for
example is "put there" by ISC.  other keys are put there by zone owners.
if you are ever specifically unsure what's in the DLV or who put it there,
ask specific questions.  if you are generally unsure then please help us
understand how we can generally illuminate this general topic.

> With IANA's ITAR, I know the parties that I have to trust (IANA staff,
> GoDaddy for the EV Cert on the web page, and the contacts for the TLDs as
> specified in the IANA whois database). I don't know who I'd have to trust
> for DLV.

ISC staff.

> But that's not why I think DLV is a stunningly bad idea.
> 
> DLV is a stunningly bad idea because it implies that I, as a caching
> server operator, would need to share fate with ISC's DLV infrastructure
> from polic[i]es to processes to software to hardware over all of which
> I'd have no control and I'd have to share that fate _in real-time_.  You
> guys screw up, I lose instantly.  As empirically demonstrated recently,
> this is a bit risky.

in addition to the root server problem which you mentioned below and which
i'll get to in a minute, i think you also lose instantly if google screws
up, or if verisign screws up, or your ISP screws up.  now, in the case of
the root servers (which we'll discuss below) you have no real choice, you
share fate with the root server system whether you want to or not.  but in
the case of google, or verisign, or DLV, you can avoid fate sharing by not
using that search engine or webmail interface, by not using any domain
names ending in .COM, and by not subscribing your validator to DLV.

to me, this is the nub of this whole repeating nightmare.  i understand why
someone might prefer not to subscribe their validator to DLV.  what i don't
understand is why anyone would argue against other people using it.
instead of just saying "what a bad idea" and moving on, we seem to be
involved in a long winding debate.  instead of just deploying DNSSEC as it
was intended to be deployed (sign the root, sign some TLDs, and then sign
some end user zones and turn on validation in some end user recursive
nameservers and stubs), we're arguing about whether or not an early
deployment aid like DLV is a threat or a menace.  (this is the tin foil hat
point where folks start telling me that early deployment will prevent real
deployment, and i tell them that without early deployment there will never
be real deployment.)

> One could argue that I'm already screwed since I share fate with the root
> servers in a similar way, but as you yourself so frequently point out,
> the root servers are independently run and there are a bunch of them with
> their own policies and processes whereas DLV is run by ISC only.
> 
> But those concerns are probably just me being paranoid...
> 
> Regards,
> -drc

paranoid is a medical term and we're neither of us qualified to assign it.
but what i will say is: ISC has signed an MoU with ICANN concerning f-root,
and for some reason we're the only root server operator who has done so.
and if you as an ICANN executive think that this MoU should not have been
signed or that a different agreement should now be signed, then you know
how to reach us, and you'll find us quite open to new proposals.

you're right that i'm proud of what i call independence, which is to say,
f-root is sponsored independently.  i don't like the idea of a government's
purse strings being able to affect policy (see the .XXX debacle for
example.)  but what i do NOT mean by independence includes secrecy, zone
content, or zone data flow.  when we have outages or installs we announce
them.  we do not edit the root zone data we publish.  we pull the zone from
wherever IANA (ICANN as a USG contractor) tells us to pull it from, using a
TSIG key that verisign (as a USG contractor) tells us they're using.  these
are all terms of the MoU signed by ICANN and ISC, as you must be aware.  if
you think that ISC is a bad example for other rootops, please tell us how
we can improve.

now as to DLV being run by ISC only.  as i've told you in person and as
i've also told steve crocker, ISC is running DLV alone because we have no
help, and not because we feel that we must control it.  we would much
rather be the secretariat to a body larger than ISC for something as
important as DLV.  if ICANN is willing to step forward and help, then you
will find ISC extremely non-possessive about DLV.  the only things you'll
find us unwilling to change our minds about are: the need for DNSSEC, and
the need for early deployment aids like DLV (and ITAR).

paul vixie
president
isc