...
At some point the Maximum Transmission Unit (MTU) for ethernet frames on the production servers was changed from 1500 to 9000. This is a common technique to improve performance in certain situations. But in order to benefit from a 9000 MTU, all ethernet devices in the data path must be set 9000 MTU. Simply changing the interfaces on the naasc-vs and na-arc nodes is not enough. All the NGAS nodes, docker containers, and namespaces in the data path must also be changed. This means recreating the entire ingress overlay network among other changes. Also, since it is unlikely the end user is going to have an MTU of 9000, there is little advantage in setting an MTU of 9000 if your goal is to move data to the user faster. Finally, because of the overhead of vxlan, an MTU of 8900 would be better than 9000. I suggest leaving the MTU at the default 1500 until there is good evidence that a larger MTU is an improvement.
- https://en.wikipedia.org/wiki/Maximum_transmission_unit
- https://docs.docker.com/engine/swarm/networking/
- https://vswitchzero.com/2018/08/02/jumbo-frames-and-vxlan-performance/
- https://vswitchzero.com/2017/09/26/vmxnet3-rx-ring-buffer-exhaustion-and-packet-loss/
- https://engineering.telefonica.com/maximizing-performance-in-vxlan-overlay-networks-ec35ebe29440
Dropped packets
Some of the NAASC VM hosts show lots of dropped Rx packets. The rate ranges from 2 to over 100 per minute. This is really unacceptable on a modern, well-designed network. While I can't say these dropped packets are indicative of a problem, they could become a problem with increased load and they certainly will make debugging more difficult when there is a problem. I suggest the reason for these dropped packets be found and resolved.
...