Increase in DNS over TCP from Chrome Browser on Windows 11

Wed Mar 15 16:29:35 UTC 2023

Yashuiro,

Thanks for the reply.  This makes sense it would be overwhelming DNS on a CPE.    My other concern would be around stateful firewalls, which would need to handle multiple short-lived TCP sessions both in session per second creation and the overall amount, depending on if the TCP sessions close gracefully etc.    It seems that Chrome is leveraging 1 TCP session per DNS query to prevent tracking of the DNS traffic, which unfortunately does not take advantage of TCP pipelining/multiplexing or out-of-order TCP DNS responses over a single TCP stream.

Thanks,

Adam Casella | Solutions Architect
Infoblox | infoblox.com
914.953.8571

From: Yasuhiro Orange Morishita / 森下泰宏 <yasuhiro at jprs.co.jp>
Date: Wednesday, March 15, 2023 at 5:19 AM
To: Adam Casella <acasella at infoblox.com>
Cc: dns-operations at lists.dns-oarc.net <dns-operations at lists.dns-oarc.net>
Subject: Re: Increase in DNS over TCP from Chrome Browser on Windows 11
!-------------------------------------------------------------------|
  This Message Is From an External Sender
  This message came from outside your organization.
|-------------------------------------------------------------------!

Hi,

> Has anyone else seen an increase in DNS over TCP traffic in their
> environment?  We have been seeing a steady increase since late last
> year and have believe we have narrowed down a major cause.  After
> reaching out to the Chromium folks and Cricket Liu reaching out to
> the Microsoft folks it seems that there has been a recent behavior
> change that is incompatible with each other, which is causing DNS
> over TCP to be preferred over UDP.

A few days ago, I saw an issue report from a router vendor that may be
caused by it.  It appears that some CPEs are unable to handle large
amounts of TCP DNS traffic.

Original page, in Japanese:
https://urldefense.com/v3/__https://www.aterm.jp/support/tech/2023/0224.html__;!!JYsgTRAg6ZQ!K8-4av385RV-2l85eIsNwjr5yvUWVuaAg8xs9Nmw071C2Ch7xjljl4lH-QNEyDf64CNYPnaD-BsGcAjaw2gO$<https://urldefense.com/v3/__https:/www.aterm.jp/support/tech/2023/0224.html__;!!JYsgTRAg6ZQ!K8-4av385RV-2l85eIsNwjr5yvUWVuaAg8xs9Nmw071C2Ch7xjljl4lH-QNEyDf64CNYPnaD-BsGcAjaw2gO$>
Google Translation:
https://urldefense.com/v3/__https://www-aterm-jp.translate.goog/support/tech/2023/0224.html?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=ja&_x_tr_pto=wapp__;!!JYsgTRAg6ZQ!K8-4av385RV-2l85eIsNwjr5yvUWVuaAg8xs9Nmw071C2Ch7xjljl4lH-QNEyDf64CNYPnaD-BsGcOV_YwC3$<https://urldefense.com/v3/__https:/www-aterm-jp.translate.goog/support/tech/2023/0224.html?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=ja&_x_tr_pto=wapp__;!!JYsgTRAg6ZQ!K8-4av385RV-2l85eIsNwjr5yvUWVuaAg8xs9Nmw071C2Ch7xjljl4lH-QNEyDf64CNYPnaD-BsGcOV_YwC3$>

> Cricket Liu reach out to the Microsoft folks and found that starting
> with Windows 11, the OS began to use socket caching due to
> exhaustion occurring with UDP ports.  Meaning that DNS UDP port is
> cached when communicating with the same server and any DNS client
> will continue to get the same UDP src-port when connecting to that
> DNS server.

IMHO, this Windows 11 behavior seems to contain security risks...

-- Yasuhiro 'Orange' Morishita <yasuhiro at jprs.co.jp>

From: Adam Casella <acasella at infoblox.com>
Subject: Increase in DNS over TCP from Chrome Browser on Windows 11
Date: Tue, 14 Mar 2023 03:57:03 +0000

> Hey Folks,
>
> Has anyone else seen an increase in DNS over TCP traffic in their environment?   We have been seeing a steady increase since late last year and have believe we have narrowed down a major cause.   After reaching out to the Chromium folks and Cricket Liu reaching out to the Microsoft folks it seems that there has been a recent behavior change that is incompatible with each other, which is causing DNS over TCP to be preferred over UDP.
>
> Based on my discussion with the Chromium team, It appears that for about 3 years Chrome has a bit of internal logic around falling back to TCP when there is a detection of reduced UDP port entropy being handed out by the OS.   When the Chrome stack falls back to TCP, according to the Chromium folks, it will continue to use TCP until Chrome is restarted or there is a network change (port flap, IP address change, etc).  The code that tracks the low entropy can be found here net::DnsUdpTracker.
>
> The Chromium folks confirmed that they are seeing an increase of TCP traffic from Windows client only.   Crickey Liu reach out to the Microsoft folks and dfound that starting with Windows 11, the OS began to use socket caching due to exhaustion occurring with UDP ports.  Meaning that DNS UDP port is cached when communicating with the same server and any DNS client will continue to get the same UDP src-port when connecting to that DNS server.   Now starting in Chrome 105, there was a change made by the Chromium folks to leverage the internat Chrome DNS stack to to run more Windows DNS queries through the Chrome stack instead of delegating the resolutions to the OS.   Due to the low UDP port entropy logic discussed above in combination to the socket caching introduced Windows 11,  we are seeing DNS clients preferring TCP over UDP for what seemed like to discernable reason until these discussions with the Chromium and Microsoft folks.
>
>>From our perspective this is and will cause a lot of issues for DNS providers as more and more Chrome + Windows clients begin to prefer TCP over UDP for DNS. And believe this has the potential to quickly become a rather large issue for DNS providers, especially at scale.
>
> Is anyone here seeing a seemingly unexplained increase in DNS over TCP traffic and if it is causing any issues within their network?
>
> For reference, Google Chrome version 105 was released on August 30th, 2022 and Windows 11 was released on October 5th, 2021.  Only with the combination of the two (post August 30th, 2022) would the issue be seen.
>
> Thanks,
>
> Adam Casella | Solutions Architect
> Infoblox | infoblox.com
> 914.953.8571
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dns-oarc.net/pipermail/dns-operations/attachments/20230315/6307c802/attachment.html>