[dns-operations] Upcoming DNSViz Maintenance

[Matthew Pounsett]

DNSViz has been back in full DB-backed operation since around 15:30 UTC
yesterday (Friday).  Although we didn't have any specific concerns after
completing the work, I decided to wait a bit before making a wide
announcement, just in case.

The maintenance took far longer than expected due to a few unrelated but
reinforcing factors:

1. The initial migration took about 18 hours longer than I originally
   bargained for.  Although I did a couple of test dumps prior, a failure of
   imagination led me to way under-estimate how long it would take to write
   the database from scratch.  I have some ideas for at least getting a better
   upper-bound estimate the next time we have to do work like this, so that I
   can set expectations better.

2. After migrating the DB to the updated system we discovered a bad
   assumption in the code that worked well enough under the older DB but did
   not hold true after the migration.  Casey and I needed time to investigate
   the issue and come to a consensus on how to fix or work around the problem,
   which was difficult because...

3. The maintenance was scheduled for just prior to the annual OARC staff/board
   all-hands week, and Casey has professorial duties.  Had the work been
   completed within 72 hours of when it started this wouldn't have been an
   issue, but once the work extended beyond that it became very difficult to
   make headway.  Normally I can simply schedule around Casey's availability,
   but that wasn't possible during the all-hands week.

4. Once we established a plan, testing and validation to make sure we weren't
   going to corrupt or lose any data also took several days.  There were
   several steps I wanted to take to ensure nothing was left to chance, and on
   a database with a little over 4 billion rows, each step could take up to
   four hours to complete.

While we don't have the budget for completely downtime-free maintenance, I
have several lessons-learned from this that I'll be using in future to
minimize interruptions to services like this.

Thanks for you patience everyone!

-- 
Matt Pounsett
DNS-OARC Systems Engineering
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 870 bytes
Desc: not available
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20261010/0cb98688/attachment.sig>