rdobbins at arbor.net
Mon Dec 15 04:21:21 UTC 2014
On 15 Dec 2014, at 9:45, David Conrad wrote:
> Two words: Microsoft Windows.
Here're two words for you:
Linux router botnet.
Or two more:
And two more, for good measure:
> Presumably you too can google "packet of death".
I don't need to search for anything - I know at least as much about them
as you do.
Since you so conveniently elided my comments to that effect, I'll
reproduce them here:
> Having worked for a major vendor of telecommunications gear which is
> quite dominant in its space, and having dealt with packet-of-death
> issues from said vendor's perspective, I'm here to tell you that all
> this preaching about avoiding monoculture is a sideshow compared to
> the real issues faced every day in the trenches.
I've dealt with packet-of-death issues - router input queue wedges, for
one - which would've literally brought the entire Internet to its knees.
And yet, it didn't happen.
Maybe you ought to think about that before you tell me to search for
> The point is that it is a risk that is easily mitigated by having
> diversity in your infrastructure.
Maybe so, for small networks which are operated by a handful of
But it doesn't scale, and my contention, based upon direct operational
experience, is that it often ends up with a negative effect on security
posture, as it's easier to be incompetent at securing multiple platforms
than it is to be incompetent at administering a single platform.
> Does the term "closing the barn door after the horses have fled" mean
> anything to you?
Again, see above.
> Sorry, where is gross incompetence being demonstrated in this
> particular case?
Gross incompetence like, say, ~27M open resolvers? Gross incompetence,
like, say, storing passwords in a plaintext file on a network share in a
folder called 'Passwords'?
Those are the *real* types of problems we face. Beside those, software
monoculture is noise.
Actually, the problems with software are even more fundamental. Since
we've known about buffer overflows for about forty years, and
experienced them in the wild for at least the last 26 years (Morris
worm, anyone?), why do software developers persist in using non-typesafe
Until these very basic issues are resolved (pardon the pun), software
diversity is a red herring.
> Are you really arguing that we should not have diversity in the
> Internet infrastructure because there are a bunch of problems
> diversity in the infrastructure won't fix?
I'm saying that, given the current levels of utter incompetence which
are the norm in all aspects of the IT and networking industries,
expecting people who are utterly unqualified to securely design, deploy,
operate, and defend a single platform to magically be able to do so for
multiple platforms, without further reducing their organizational
security postures, is wishful thinking.
I'm saying that until such time as average people can design, deploy,
operate, and defend a single platform effectively, it's utterly
unrealistic to expect them to do so for multiple platforms.
> Too bad no one has come up with something like Puppet, Chef, Ansible,
> etc., to help manage infrastructure configuration at scale.
No - what's too bad is that due to aforementioned incompetence (as well
as the fact that they aren't as easy to utilize as they ought to be),
they're only used on a fraction of networks/deployments worldwide.
> Software diversity is a tool that network administrators use to
> improve resiliency in their infrastructure.
For above-average - emphasis on *average*, it actually means something -
personnel in focused organizations, sure.
Not at scale, and not with average personnel. Quite the opposite, in my
experience and observation.
> I agree it is not a silver bullet but if I was building out critical
> infrastructure like (oh say) a root server or a resolver cloud that my
> customers depend on, I would want to minimize the risk that my
> infrastructure was vulnerable to a single bug.
You'd be a lot better off worrying about all the BCPs and other
mechanisms required to keep those services up and running, and ensuring
that you've done them *all*, before you start worrying about software
If you make so much progress that software diversity is your biggest
worry, you've pretty much won.
I've yet to see any organization get there.
> I am honestly surprised you're arguing against this.
I'm unsurprised you're arguing for it. Things can look easy/desirable
when you yourself know how to do things right, have decades of
experience and acquired expertise to draw upon, and are part of a
focused organization in which you can be choosy about whom you hire.
But you aren't the norm.
It's quite a different kettle of fish when you're talking about large
organizations and lowest-common-denominators, let me assure you.
Likewise, smaller organizations without someone of your level of skill
and experience - which, unfortunately, are the norm.
By and large, the people on this list, like you, tend to be experts in
their chosen profession. Not so for the hoi polloi.
And I'll leave you with another thought - it's perfectly possible to
achieve codebase diversity for any given piece of software. The bad
guys do it all the time with metamorphic and polymorphic code for
botnets and malware. Why don't developers of legitimate software -
like, say, ISC or Nominum - do something similar, resulting in multiple
codebases which look and feel the same and which perform the same tasks,
but are implemented differently?
Roland Dobbins <rdobbins at arbor.net>
More information about the dns-operations