[dns-operations] smart failover: Lua record experiments

[bert hubert]

On Fri, Nov 04, 2016 at 10:02:31AM -0400, Eli Lindsey wrote:
> > Especially if you think you could be rolling this out to your users in
> > some way let us know your ideas.
> 
> I'd go about this in the opposite direction - define the minimal set of
> failover functionality you need, see how far that gets you, then go from
> there (similar to how Denominator has abstracted out core, common
> control plane configuration).

Hi Eli,

Thank you for these notes - I've let them simmer for the past 10 days or so,
and I think you are very right.

While the 'LUA' record type is undeniably very cool, even from our examples
you can see that most configurations are not actually Lua programs but
data structures passed to single functions.

> My experience from building a lot of those features for R53 is that a
> status code check against a single IP will get you ~99% of the way there
> (string match checks are a bit silly in my opinion, but are popular).

So the string match has saved me in one demo where some of the IP addresses
in the set turned out not to actually host the desired site!

> Most other features can be added purely through config by modeling a
> decision tree in the DNS (we called this a RubberTree in some of the
> docs).

Right - so in the latest code, you can indeed build trees that go from
'location' to a set of DC's in order.

> Letting implementation details (like Lua) leak all the way to the spec
> seems particularly bad. Defining an overly rich syntax (like arbitrary
> Lua) is going to make the mapping to providers much more difficult and
> isn't necessary for the vast majority of use cases.

Right. And scary.

> Also, metapoint to keep in mind: large providers won't be running checks
> directly from the query dataplane. Don't do anything that precludes more
> scalable designs like a separate healthchecking fleet.

Noted - the "uptime bits" could conceivably come from elsewhere.

Eli, thanks for your very useful feedback - expect to eventually see the
'LUA' record exist separately from a 'DYNAMIC' record type which is based on
non-Turing complete data structures.

We might keep the LUA one around because, you know, it is cool even when not
that portable.

	Bert