<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<div id="mail-editor-reference-message-container">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">Hello Peter,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">We operate a blocking DNS resolver. Our false positives are either customer escalated to our support team via portal/email/phone, or our analyst team discover false positives using
monitoring metrics such as source IP diversity, query volume, blocked domain age, recent query trends, recent past performance of a feed/source, etc.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">Very broadly, if we were to block a popular established domain that is requested by a high number of different source IPs it will probably correlate with high customer impact…
but that’s not a hard and fast rule. We’ve experienced very popular false positive domains that seem to cause little to no impact to customer business operations. Perhaps because they’re being queried by unimportant machines rather than humans in front of
browsers. Equally, sometimes a single blocked website interrupting a senior employee’s workflow can elicit a fast escalation from their IT department who have been put under pressure to fix it quickly. In short, it is sensible to collect blocked domain traffic
metrics, but I wouldn’t rely on them exclusively in identifying customer impact.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">We maintain a spreadsheet of all false positives and we join each one back with the originating threat intelligence that caused it to enter the block list. Using this we track
the performance of each source feed over time. If a particular source feed, or a sub-section of a source feed, begins to misbehave we can escalate with the author or remove it from the block list altogether.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">So, regarding your question, we find that evaluating groups of false positives based on their source metadata is significantly easier than playing whack-a-mole with individual
domains as they pop up. As such, we put a lot of effort into maintaining as much metadata as possible as the data flows through the system from source to block list to dashboard.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">I don’t know how others are planning to do their heuristics work, but perhaps the AI/ML model could be encouraged to expose some of its decision-making parameters rather than just
giving a black box yes/no answer? Maybe the different ML stages offer scores, confidence intervals, keyword matches, or other data flags that describe how a blocking outcome was reached. Such data alongside a list of false positives might help an engineer
determine that ‘ML feature X’ seems to be a common problem so maybe dial it down a bit in the model.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">James<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>