[dns-operations] specifics of UDP response with truncate bit; odd google fail on AAAA responses w/ truncation

Fri May 25 11:50:24 UTC 2012

Ryan,

We believe that Amazon has fixed the issue specifically with this ELB
(looks like by paring back the number of responses, which stops the flip
to TCP, which is where things break).  It's a known issue that's supposed
to be fixed globally within a week.

Thanks,
-Dave

On 5/24/12 9:38 PM, "Ryan Rawdon" <ryan at u13.net> wrote:

>Since Netflix added AAAAs to movies.netflix.com (or more specifically,
>enabled IPv6 on the Amazon ELB instance that movies.netflix.com CNAMEs to
>in the eastern US), I have seen inconsistent answers from caching
>resolvers for 
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com.
>
>Below are three different responses for
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com -
>from Google DNS, Amazon's authoritative NS, and my local caching
>resolver, respectively.
>
>You can view pcaps for these 3 at:
>http://cloudshark.org/captures/4d24c193533b Google
>http://cloudshark.org/captures/530a0fda5234 Amazon
>http://cloudshark.org/captures/582e87dfda67 Local resolver
>
>
>The UDP answer from Amazon has the Truncate bit set to 1, as expected. It
>also says that there are 24 answer RRs but the UDP response contains zero
>answers.
>
>This combination of behaviors seems to throw a curveball to resolvers and
>clients alike.  You can see that the host output below says that a
>malformed message was encountered, as does the wireshark cloudshark link
>above for the Amazon UDP response.
>
>Google fails to report any AAAA answers for this name, more information
>on that after the wall of output below.   I have looked through the
>various RFCs pertaining to DNS a bit, but haven't found any authoritative
>statements on the correct behavior for a properly-formed UDP response
>packet with the truncate bit set.  So here are the questions I am left
>with right now:
>- Should a packet with the truncate bit set contain answers, or is this
>optional?  I'm guessing optional, but could see arguments for the UDP
>response with the truncate bit containing at least the first few RRs
>- Should a packet with the truncate bit set have the field for the number
>of Answers reflect how many answers are in that packet, or how many are
>in the actual forthcoming response?  I believe that it should contain the
>number of RRs contained in the UDP response itself, not the full answer
>to the query - and this is where I believe the Amazon response is
>malformed.  In the UDP response it says there are 24 answer RRs when
>there are zero
>
>Output of host usage against these 3 servers below, with a bit more
>information on the Google issue below
>
>
>nova-dhcp-host111:~ ryan$ host -t AAAA
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com
>8.8.8.8
>;; Truncated, retrying in TCP mode.
>;; communications error to 8.8.8.8#53: end of file
>nova-dhcp-host111:~ ryan$
>
>nova-dhcp-host111:~ ryan$ host -t AAAA
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com
>ns-927.amazonaws.com
>;; Warning: Message parser reports malformed message packet.
>;; Truncated, retrying in TCP mode.
>Using domain server:
>Name: ns-927.amazonaws.com
>Address: 72.21.204.209#53
>Aliases: 
>
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:6cc8
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3211:b4fa
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3211:c04e
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:7430
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:5488
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:7262
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:6d95
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:6d73
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::6b14:e26c
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3211:c354
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:5149
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3210:fa0f
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3210:c1b2
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::ae81:f9ac
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:e771
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:f545
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:7747
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:545b
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::6b14:d04f
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:765d
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::6b14:fa4b
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:7702
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:722d
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:d9dc
>nova-dhcp-host111:~ ryan$
>
>
>nova-dhcp-host111:~ ryan$ host -t AAAA
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com
>172.25.254.253
>;; Truncated, retrying in TCP mode.
>Using domain server:
>Name: 172.25.254.253
>Address: 172.25.254.253#53
>Aliases: 
>
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:6cc8
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:6d73
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:6d95
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:722d
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:7262
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:7430
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:765d
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:7702
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:7747
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:d9dc
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:e771
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:f545
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::6b14:d04f
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::6b14:e26c
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::6b14:fa4b
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::ae81:f9ac
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3210:c1b2
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3210:fa0f
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3211:b4fa
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3211:c04e
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3211:c354
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:5149
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:545b
>dualstack.merchweb-frontend-us-999408195.us-east-1.elb.amazonaws.com has
>IPv6 address 2406:da00:ff00::3213:5488
>nova-dhcp-host111:~ ryan$
>
>
>
>Will Dean wanted to test the failed Google response independently of the
>malformed Amazon response, as I was finishing up typing the above
>message.  It looks like the EOF failure from Google is reproducible with
>other queries that result in the truncate bit being set.
>dnstest.managemydedi.com is set up with the intention of creating a large
>response that results in the truncate bit being sent in the UDP response.
>
>nova-dhcp-host111:~ ryan$ host -t AAAA dnstest.managemydedi.com 8.8.4.4
>;; Truncated, retrying in TCP mode.
>;; communications error to 8.8.4.4#53: end of file
>nova-dhcp-host111:~ ryan$
>
>
>It looks like this is only broken with AAAA queries.  dns2test is packed
>with A records, and does not cause the same problem with Google
>
>nova-dhcp-host111:~ ryan$ host -t A dns2test.managemydedi.com 8.8.8.8
>;; Truncated, retrying in TCP mode.
>Using domain server:
>Name: 8.8.8.8
>Address: 8.8.8.8#53
>Aliases: 
>
>dns2test.managemydedi.com has address 203.0.113.0
><bunch more answers>
>dns2test.managemydedi.com has address 203.0.113.35
>
>