[dns-operations] smart failover: Lua record experiments

[Eli Lindsey]

> For some more background, please see
> https://gist.github.com/ahupowerdns/1e8bfbba95a277a4fac09cb3654eb2ac 
> 
> At this stage, your thoughts are very welcome on how we should develop
> this.
> Especially if you think you could be rolling this out to your users in
> some way let us know your ideas.

I'd go about this in the opposite direction - define the minimal set of
failover functionality you need, see how far that gets you, then go from
there (similar to how Denominator has abstracted out core, common
control plane configuration).

My experience from building a lot of those features for R53 is that a
status code check against a single IP will get you ~99% of the way there
(string match checks are a bit silly in my opinion, but are popular).
Most other features can be added purely through config by modeling a
decision tree in the DNS (we called this a RubberTree in some of the
docs).

Letting implementation details (like Lua) leak all the way to the spec
seems particularly bad. Defining an overly rich syntax (like arbitrary
Lua) is going to make the mapping to providers much more difficult and
isn't necessary for the vast majority of use cases.

Also, metapoint to keep in mind: large providers won't be running checks
directly from the query dataplane. Don't do anything that precludes more
scalable designs like a separate healthchecking fleet.

-eli