Intel® Ethernet 800 Series
Linux Performance Tuning Guide
IRQ Affinity
Configuring IRQ affinity to ensure interrupts for the hardware Tx/Rx queues are affinitized to proper CPU cores can have a huge impact on performance, particularly in multi-thread workloads. Proper queue affinitization is even more important when configurations include multiple high-speed ports. Intel has a utility to configure IRQ affinity, the set_irq_affinity script, which is included with ice source package.
The efficiency benefit if pinning to local CPUs versus all cores varies between different workloads.
Using the set_irq_affinity script from the ice source package (recommended):
- To use all cores:
<path-to-ice-package>/scripts/set_irq_affinity -X all ethX - To use only cores on the local NUMA socket:
<path-to-ice-package>/scripts/set_irq_affinity -X local ethX - You can also select a range of cores. Avoid using cpu0 because it runs timer tasks.
<path-to-ice-package>/scripts/set_irq_affinity -X 1-8,16-24 ethX
The affinity script disables XPS when the -X option is specified. Disabling XPS and enabling symmetric queues is beneficial for workloads where best performance is achieved when Tx and Rx traffic get serviced on the same queue pair(s). For more details, see Transmit and Receive Queue Alignment.
The effect of the irqbalance in your configuration depends on the workload and platform configuration. However, when irqbalance is enabled it can cause unpredictable performance measurements, as the kernel can move work to different cores.
Disabling user-space IRQ balancer to enable queue pinning:
Manually configuring IRQ affinity:
- Find the bit masks for each of the processors:
Assuming cores 0-11 for node 0: [1,2,4,8,10,20,40,80,100,200,400,800]
- Find the IRQs assigned to the port being assigned:
grep ethX /proc/interrupts and note the IRQ values. For example, 181-192 for the 12 vectors loaded.
- Echo the SMP affinity value into the corresponding IRQ entry. Note that this needs to be done for each IRQ entry:
echo 1 > /proc/irq/181/smp_affinity echo 2 > /proc/irq/182/smp_affinity echo 4 > /proc/irq/183/smp_affinity
Showing IRQ affinity:
- To show the IRQ affinity for all cores:
<path-to-ice-package>/scripts/set_irq_affinity -s ethX - To show only cores on the local NUMA socket:
<path-to-ice-package>/scripts/set_irq_affinity -s local ethX - You can also select a range of cores:
<path-to-ice-package>/scripts/set_irq_affinity -s 0-8,16-24 ethX