network Tuning

For a host with a 10G NIC optimized for network paths up to 200ms RTT, and for friendlyness to single and parallel stream tools, or a 40G NIC up on paths up to 50ms RTT:

# allow testing with buffers up to 128MB
net.core.rmem_max = 134217728 
net.core.wmem_max = 134217728 
# increase Linux autotuning TCP buffer limit to 64MB
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
# increase the length of the processor input queue
net.core.netdev_max_backlog = 250000
# recommended default congestion control is htcp 
net.ipv4.tcp_congestion_control=htcp
# recommended for hosts with jumbo frames enabled
net.ipv4.tcp_mtu_probing=1

Also add this to /etc/rc.local (where N is the number for your 10G NIC): 

    /sbin/ifconfig ethN txqueuelen 10000

    ifconfig ethX txqueuelen 300000

   ethtool -K eth3 gso on

   ethtool -k eth3

ip link show eth3

tuning NIC

You can use ethtool to check on the number of descriptors your NIC has, and whether the driver is configured to use them.  Some sample Linux output from an Intel 10GE NIC that is using the default config is below:

[user@perfsonar ~]# ethtool -g eth2
Ring parameters for eth2:
Pre-set maximums:
RX:          4096
RX Mini:     0
RX Jumbo:    0
TX:          4096
Current hardware settings:
RX:          256
RX Mini:     0
RX Jumbo:    0
TX:          256

Under Linux, to check which driver you are using, do this:

  ethtool -i eth0

If you're using e1000 chips (Intel 1GE, often integrated into motherboards; note that this does not apply to newer variants) the driver defaults to 256 Rx and 256 Tx descriptors. This is because early versions of the chipset only supported this. All recent versions have 4096, but the driver doesn't autodetect this. Increasing the number of descriptors can improve performance dramatically on some hosts.

You can also add this to /etc/rc.local to get the same result.

   ethtool -G ethN rx 4096 tx 4096

[1]https://fasterdata.es.net/host-tuning/linux/