<div dir="ltr"><span id="gmail-docs-internal-guid-700affa0-7fff-d457-4fdd-619c2343e9b5"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Google Public DNS is a large open resolver, available to the public.  DNS Flag Day 2020 ( </span><a href="https://dnsflagday.net/2020/" style="text-decoration-line:none"><span style="font-size:12pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://dnsflagday.net/2020/</span></a><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> ) was a coordinated internet-wide effort, whereby DNS operators agreed to set the EDNS buffer size parameter in outgoing queries with the goal of limiting IP fragmentation and thereby improving the overall reliability and performance of the global DNS.  This is a summary of how Google Public DNS participated and what we learned through the experience.</span></p><h1 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt"><span style="font-size:20pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:500;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Participation</span></h1><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Being a large open resolver, we generate outgoing queries to many DNS servers across the world, and across many heterogeneous networks.  Large response packets can sometimes be fragmented, and delivery of fragments is generally less reliable.  When a UDP response packet is not delivered, a client can only wait and give up.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Waiting for an outgoing UDP query to time out consumes a significant portion of the overall deadline that we set for the iterative resolution process, contributing towards an overall failure for the query.  In other words, for us with large responses the small penalty of receiving a truncation response and retrying over TCP is better than the large penalty of completely timing out (and then retrying anyway).</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">While difficult to measure accurately (the only signal is the lack of a response, but we can never know which missing responses were due to dropped fragments), we had past anecdotal evidence of a higher than average failure rate for queries with large responses.  These tended to involve queries for the DNSKEY and TXT record types.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Having observed such failures, we were interested in participating in the flag day as a path towards better service.  As previously discussed ( </span><a href="https://youtu.be/CHprGFJv_WE" style="text-decoration-line:none"><span style="font-size:12pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://youtu.be/CHprGFJv_WE</span></a><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> ) we evaluated the suggested ( </span><a href="https://dnsflagday.net/2020/#message-size-considerations" style="text-decoration-line:none"><span style="font-size:12pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://dnsflagday.net/2020/#message-size-considerations</span></a><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> ) 1232 byte and an alternative 1400 byte limit, and we selected the latter.  (We may reconsider these specific values over time, including configuring IPv4 and IPv6 separately.)</span></p><h1 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt"><span style="font-size:20pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:500;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Release</span></h1><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">In order to monitor impact as we released this change, and to provide a path to roll back, we used an experiment approach.  We defined an experiment rate (percentage) which applied randomly to each outgoing query.  We set this rate very small and increased it gradually over time.  At each step we measured a variety of signals including UDP timeouts, TCP retries, and truncations.  We compared the baseline queries to those in the experiment, and generally found results that aligned with our expectations: rates of truncations rose and timeouts fell.</span></p><h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt"><span style="font-size:16pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:500;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Timeline</span></h2><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Date</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">     </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">        </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Experiment %</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2020-09-10</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">   </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">1%</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2020-10-08</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">     </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2%</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2020-10-22</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">     </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">5%</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2020-11-05</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">     </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">15%</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2020-11-19</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">    </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">50%</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2021-01-21</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">    </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">100%</span></p><h1 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt"><span style="font-size:20pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:500;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Problem</span></h1><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">We first set the experiment rate to 100% in January of 2021.  Until that point all signals looked good, and had for months to that point before reaching 100%.  At a full 100% bufsize-limited query rate we exacerbated a latent issue in the system. Specifically: in the case where UDP truncation forces TCP retry and the TCP query also fails, we might mishandle this response, caching the (empty, truncated) UDP response for too long.  (A primary indicator: some Google systems began to have persistent issues looking up some large email related TXT records when these queries always followed the TCP retry path.)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">At this point we reverted back to a low experiment rate, and started the process of identifying and fixing the underlying problem in our service.  Once that was done we first did a geographically targeted experiment, then proceeded through another gradual increase in experiment rate across the globe.</span></p><h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt"><span style="font-size:16pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:500;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Timeline 2</span></h2><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Date</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">        </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">        </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Experiment %</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2021-02-18</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">   </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">50%</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2021-04-15</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">    </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">50% + 100% in one small geography</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2021-04-29</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">      </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">50% + 100% in two small geographies</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2021-05-13</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">    </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">75%</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2021-05-27</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">    </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">90%</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2021-06-10</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="gmail-Apple-tab-span" style="white-space:pre">    </span></span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">100%</span></p><h1 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt"><span style="font-size:20pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:500;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Results</span></h1><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">We’ve been specifying a buffer size of 1400 for all outgoing queries (when they include EDNS) since June 2021.  Though we took a while to get here thanks to the need to roll back and fix an internal bug, we’ve been stable this way for several months.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">We’ve observed no other problems caused by the change, while also anecdotally seeing issue reports involving queries with large response sizes disappear.  Since only a small amount of traffic (under half a percent) involves large enough responses to be affected, the absolute numbers are small but as predicted we see an increase in UDP truncated responses, and a decrease in UDP timeouts.</span></p><br><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">We’re happy with the change and expect to keep it for the long term.  However we are tracking the IETF DNS operations working group draft about fragmentation ( </span><a href="https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-avoid-fragmentation" style="text-decoration-line:none"><span style="font-size:12pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-avoid-fragmentation</span></a><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> ).</span></span><br></div>