[dns-operations] Unplanned DLV zone outage on 2009-Apr-06

[Keith Mitchell]

David,

> On Apr 6, 2009, at 9:59 AM, Michael Graff wrote:
>> We plan to continue to test and improve our internal procedures,
>> monitoring and hardware platform over the next two weeks, at which
>> point the service will be announced as being in Full Production.
> 
> I'm curious: what does "being in Full Production" mean?

Same as I said here on 22nd March:

> Keith Mitchell wrote:
> 
>> To be clear, much of our distinction between "limited" and "full" 
>> production relates to the hardware platform we are running DLV on 
>> operationally. To date it has run on a single server located at
>> ISC's Redwood City facility. As part of the UI development, we
>> identified that we would need to run this on higher capacity
>> platforms to cope with likely increased uptake, and split the UI
>> and back-end functions onto separate servers to enhance security.
>> 
>> While we're happy about the robustness of the service from a
>> code-base point of view, we don't want to say the service is in
>> "full" production until we have brought up a more powerful web
>> front-end server, and located this at a data center with more
>> diverse connectivity. We are working on this at present, and the
>> current plan is to have this up and running by 31st March, at which
>> point we'll announce DLV as being in "full" production.

Except:
(a) Obviously the 31st March date has slipped, by about 10-14 days
(b) After this incident we'll be doing more testing than we'd originally
    planned first.

Keith