IRQ Affinity

Intel® Ethernet 700 Series

Linux Performance Tuning Guide

ID	Date	Version	Classification
334019	12/09/2024	1.1	Public

Configuring IRQ affinity so that interrupts for different network queues are affinitized to different CPU cores can have a huge impact on performance, particularly multi-thread throughput tests.

To configure IRQ affinity, stop irqbalance and then either use the set_irq_affinity script from the i40e source package or pin queues manually.

Disable user-space IRQ balancer to enable queue pinning:

systemctl disable irqbalance

systemctl stop irqbalance

Using the set_irq_affinity script from the i40e source package (recommended):

To use all cores: [path-to-i40epackage]/scripts/set_irq_affinity -X all ethX

To use only cores on the local NUMA socket: [path-to-i40epackage]/scripts/set_irq_affinity -X local ethX

You can also select a range of cores. Avoid using cpu0 because it runs timer tasks. [path-to-i40epackage]/scripts/set_irq_affinity 1-2 ethX

Note:The affinity script enables Transmit Packet Steering (XPS) as part of the pinning process when the -x option is specified. When XPS is enabled, Intel recommends that you disable irqbalance, as the kernel balancer with XPS can cause unpredictable performance. The affinity script disables XPS when the -X option is specified. Disabling XPS and enabling symmetric queues is beneficial for workloads where best performance is achieved when Tx and Rx traffic get serviced on the same queue pair(s).

Configuring symmetric queues in Linux involves tuning the network interface driver parameters to enable symmetric receive queues (Rx) and symmetric transmit queues (Tx) for supported network adapters.

Note:

Symmetric queues are an advanced networking feature, and not all 700 series network adapters or drivers support them.

Ensure you have the necessary driver and hardware support before attempting to configure symmetric queues.

To configure symmetric queues, follow these general steps:

Edit Network Interface Configuration File: Use a text editor (for example, vi, nano, or gedit) to edit the network interface configuration file. The file is typically located under the /etc/sysconfig/network-scripts/ directory and has a name like ifcfg-ethX, where ethX is the name of your network interface.

Add Symmetric Queue Parameters.
Add the following lines to the network interface configuration file:
ETHTOOL_OPTS="rx-queues 8 tx-queues 8"

Restart Network Service.
After making the changes, restart the network service to apply the new configuration.
sudo systemctl restart network

Manually:

Find the processors attached to each node using: numactl --hardware lscpu

Find the bit masks for each of the processors:

Assuming cores 0-11 for node 0: [1,2,4,8,10,20,40,80,100,200,400,800]

Find the IRQs assigned to the port being assigned: grep ethX /proc/interrupts and note the IRQ values For example, 181-192 for the 12 vectors loaded.

Echo the SMP affinity value into the corresponding IRQ entry. Note that this needs to be done for each IRQ entry: echo 1 > /proc/irq/181/smp_affinity echo 2 > /proc/irq/182/smp_affinity echo 4 > /proc/irq/183/smp_affinity

Show IRQ affinity:

To show the IRQ affinity for all cores: <path-to-i40epackage>/scripts/set_irq_affinity -s ethX

To show only cores on the local NUMA socket: <path-to-i40epackage>/scripts/set_irq_affinity -s local ethX

You can also select a range of cores: <path-to-i40epackage>/scripts/set_irq_affinity -s 0-8,16-24 ethX

Note:The set_irq_affinity script supports the -s flag in i40e driver version 2.16.11 and later.

Intel® Ethernet 700 Series

Linux Performance Tuning Guide

​IRQ Affinity

IRQ Affinity