[dns-operations] Filtering policy: false positive rate

[Peter Thomassen]

Hi,

Resolver policies typically describe operational rules, such as which data is collected and retained for how long etc. When a resolver offers filtering for ads, abuse, ... their policy ought to say something about this, such as how to unblock a benign domain that was flagged in error.

Now, block-list-based filtering is one thing. For resolvers like DNS4EU which (will) employ heuristic, prediction-based filtering, a new type of error source appears, namely false categorization from prediction.

I think that the resolver policy should say what's an acceptable false positive rate for such filtering. The problem is, how do you measure that?

At a given time, one might not know which names would be blocked by the classifier (until someone asks). So you can't go and check the list for false positives, because there's no list.


Then, how to define a false positive rate?

Look at all blocked queries, and do a post-hoc investigation?

How about popularity -- should one factor in that blocking *.ddns.net is more severe than blocking *.blank.page? I.e., is it a ratio of blocked/total queries, or blocked/total names?

Or, wait for complaints, and somehow relate the complaints to the number of queries, i.e. take "complaints per 1M (blocked?) queries" or something? (That would not exactly be a false positive rate, but it *might* somewhat correlate.)

One may also not compute a ratio at all, and just count complaints (and define an acceptable threshold per day). -- Such a count would have to scale with the user base.


Questions over questions. Is there best practice on this? What do other resolver operators do?

In any case, I want to collect input and feed this back to the DNS4EU consortium, to make sure that *some* level of quality is committed to.

Thanks,
Peter

-- 
https://desec.io/