​IP Forwarding

• ​​Update the kernel.

  Some recent in-distro kernels have degraded routing performance due to kernel changes in the routing code starting with the removal of the routing cache due to security. Recent out-of-distro kernels should have patches that alleviate the performance impact of these changes and might provide improved performance.

• ​​Disable hyper-threading (logical cores).
• ​​Edit the kernel boot parameters.
  • Force iommu off (intel_​iommu=off or iommu=off) from the kernel boot line unless required for virtualization
  • Turn off power management: processor.max_cstates=1 idle=poll pcie_aspm=off
• ​​Limit the number of queues to be equal to the number of cores on the local socket (12 in this example). ethtool -L ethX combined 12
• ​​Pin interrupts to local socket only. set_irq_affinity -X local ethX OR set_irq_affinity -X local ethX Note:-X or -x can be used depending on the workload.
• ​​Change the Tx and Rx ring sizes as needed. A larger value takes more resources but can provide better forwarding rates. ethtool -G ethX rx 4096 tx 4096
• ​​Disable GRO when routing.

  Due to a known kernel issue, GRO must be turned off when routing/forwarding.

  ethtool -K ethX gro off

  where ethX is the Ethernet interface to be modified.

• ​​Disable adaptive interrupt moderation and set a static value. ethtool -C ethX adaptive-rx off adaptive-tx off ethtool -C ethX rx-usecs 64 tx-usecs 64 Note:Depending on the type of processor and workload, the coalescing parameters for RX and TX can be adjusted for improved performance (or less frame loss).
• ​​Disable the firewall. sudo systemctl disable firewalld sudo systemctl stop firewalld
• ​​Enable IP forwarding. sysctl -w net.ipv4.ip_forward=1
• Configure maximum values for the receive and send socket buffer sizes. sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216 Note:Depending on the workload or requirement, these values can be changed from the default.