...
- I don't think this is because of broadcast noise on the 10Gb/s network (10.2.120.0/24) as I don't see see large dropped packet counts on all naasc-vs hosts.
- 2022-09-26 krowe: Interestingly, if I watch the number of packets dropped per minute (I worte a script) and run the scripts at the same time on all four naasc-vs hosts, I see patterns. The number of dropped packets each minute is identical between naasc-vs-2 and naasc-vs-4 and hovers around 100. The number of dropped packets each minute is identical between naasc-vs-3 and naasc-vs-5 and hovers around 2. This tells me that naasc-vs-2 and naasc-vs-4 are getting the same traffic and dropping it the same way. What is this traffic?
- 2022-09-26 krowe: I set na-arc-6, the only guest on naasc-vs-2, to drain in docker swarm to see if that reduced the number of dropped packets seen on naasc-vs-2. Thinking it was docker swarm creating this traffic. There was no change in dropped packet rate. It continued to match naasc-vs-4.
- 2022-09-26 krowe: On naasc-vs-3 and naasc-vs-5 I see the dropped packet count per minute at about 2 but every 5 or 6 minutes the count inreases to 10 or 11.
- 2022-09-26 krowe: I tried looking at other nodes on the 10Gb 10.2.120.0/24 network but I couldn't login to most of them. One I could login to is cv-vs-4 and it is also seeing dropped Rx packets on its 10Gb interface at about the same rate as naasc-vs-3 and naasc-vs-5. This makes me think that these dropped packets have nothing to do with docker swarm. Perhaps there is just something on that network (some misconfigured Windows box or something) that is throwing bad packets around. That doesn't explain the difference in the dropped packet rates though.
- 2022-09-27 krowe: try clearing the ARP cache on the switch? Perhaps the switch is sending packets to the node to an IP address that is no longer there like because the container moved.
- use tcpdump and sort by destination looking for the number of dropped packets per minute.
- 2022-10-03 krowe: dhart and thalstead inserted a second 10Gb/s card in naasc-vs-2. This one is supposedly a Solarflare SNF8522 even though Linux detects it as the same model as the original card (Solarflare Communications SFC9220 10/40G Ethernet Controller [1924:0a03]). Tracy configured this card to be the 10Gb/s NIC of naasc-vs-2 (ens2f0np0). I don't know why they didn't just remove the original card an insert the new card thus requiring no changes to the configuration but whatever. I am still seeing about 60 dropped packets per minute, and it still matches the dropped packets on naasc-vs-4. So the idea that the original card had some hardware flaw (like bad memory or something) is disproven.
- 2022-10-14 krowe: I may have a line on what is causing the dropped packets. Several times now when I do tcpdumps on either naasc-vs-2 or naasc-vs-4, tcpdump at the end tells me how many packets were dropped by the interface. This number matches what my droprate.sh script shows (it gets its information from ifconfig which is probably what tcpdump does also). Looking at these tcpdumps, I see a number of LLMNR (that's the protocol) packets which is about twice this number. Half of these packets are IPV4 and half are IPV6. They are both going to multicast addresses (224.0.0.252 and ff02::1:2). This is leading me to think naasc-vs-2 and naasc-vs-4 are dropping packets either because the are from IPV6 addresses or they are destened for IPV6 multicast addresses. Not sure which or even if that is correct. But so far these sorts of packets fit the number of dropped packets. I see these same LLMNR packets, both IPV4 and IPV6 on naasc-vs-3 but that host reports almost no dropped packets. Configuration difference?
- This page https://access.redhat.com/articles/22304 has a python script that will join a multicast group.
- I see with netstat -ng naasc-vs-3 and naasc-vs-5 are in the special multicast group 224.0.0.251 while naasc-vs-2 and naasc-vs-4 are not. This configuration difference lines up with seeing dropped packets on naasc-vs-2 and naasc-vs-4 but not on naasc-vs-3 and naasc-vs-5 (or at least a lot fewer dropped packets). But joining this multicast group on naasc-vs-2 didn't change the dropped packet rate.
- I will ask if the LLMNR protocol on cvsccm can be disabled. Looking around the internet it seems that this is a pretty old way for Windows to share hostnames and is largely deprecated.
- 2022-10-17 krowe: Actually, I see a strong correlation between IPV6 packets in tcpdump and dropped packets reported by both my script and tcpdump itself on naasc-vs-2 and naasc-vs-4. But I don't understand why these two hosts would be dropping IPV6 packets while naasc-vs-3 would not. If I tcpdump all three hosts, I see the same IPV6 packets on all three (usually LLMNR multicast or DHCP6 broadcast). And all three hosts have ipv6.disable=1 in /proc/cmdline.
- 2022-10-17 krowe: I see an even stronger correlation between dropped packets and IPV6 packets on VLAN ID 96. This could explain the dropped packets as neither naasc-vs-2 nor naasc-vs-4 have VLAN 96 configured while both naasc-vs-3 and naasc-vs-5 do have VLAN 96 configured. This should be an easy test. Just configure VLAN 96 on naasc-vs-2 and see if the dropped packet rate goes from about 1 per second to less than 1 per minute.
Comparisons
naasc-vs-2, 3, 4, 5
...