Intel® Ethernet Controller E810 Application Device Queues (ADQ)
Configuration Guide
Configure Independent Pollers
The following variables are used in the examples in this section:
| $iface | The interface in use (PF). |
| $iface_bdf | The network interface BDF notation (Bus:Device.Function) used by devlink (PF). |
| $queues | The number of queues in the TC. |
| $qps_per_poller | The number of queue pairs per independent poller for a given TC (max value is #queues in the TC). |
| $poller_timeout | The timeout value for the independent pollers for a given TC (nonzero integer value in jiffies, default value 10000). |
Setting global sysctl.net.core.busy_poll and sysctl.net.core.busy_read settings is NOT needed with ADQ independent poller configuration.
After creating TCs (in Create TCs):
- Set the interface BDF, needed for devlink param commands.
iface_bdf=$(ethtool -i ${iface} | grep bus-info | awk '{print $2}') - Set the number of queue pairs per poller for the TC.
devlink dev param set pci/${iface_bdf} name tc1_qps_per_poller value \ $qps_per_poller cmode runtime Note:The example above is for TC1. Valid devlink param flags include tc1_qps_per_poller throughtc15_qps_per_poller , to configure independent pollers on up to 16 TCs (max #TCs). If the TC does not exist, the devlink command will return error:devlink answers: Invalid argument Note:The maximum allowed value for $qps_per_poller is the number of $queues in the TC. If the $qps_per_poller value specified is greater than $queues, the following error will be returned by the ice driver: Error: ice: Value cannot be greater than number of queues in TC. devlink answers: Invalid argument - Set the poller timeout value for the TC.
devlink dev param set pci/${iface_bdf} name tc1_poller_timeout value \ $poller_timeout cmode runtime Note:The example above is for TC1. Valid devlink param flags include tc1_poller_timeout throughtc15_poller_timeout , to configure independent poller timeout values on up to 16 TCs (max #TCs). If not specified, the default $poller_timout is set to 10000 jiffies. - Enable threaded napi.
echo 1 > /sys/class/net/$iface/threaded Note:Independent pollers work with both threaded napi as well as default napi that runs in softirq context and triggers ksoftirqd threads. - Pin Independent Poller kthreads to specific CPU cores.
Note:Threaded napi must be enabled in order to pin application threads/pollers to specific CPU cores. - Get the napi threads PIDs (order from highest to lowest).
ps -x | grep "$iface" | head -n -1 | tac | awk '{print $1}' - Affinitize all Poller threads to a specific CPU core (skipping the PIDs that are for non-polled queues).
taskset -c 2 -p <pid of thread 2> taskset -c 2 -p <pid of thread 3> taskset -c 2 -p <pid of thread 4> taskset -c 2 -p <pid of thread 5> etc Note:Depending on the application workload, it is preferable to pin application threads and pollers to different CPU cores. If the application is not overly busy, running both types of threads on the same cores may yield better performance results.
- Get the napi threads PIDs (order from highest to lowest).
- Verify independent pollers configuration.
To query all devlink parameters:
devlink dev param show To query a specific devlink parameter for the pci device:
devlink dev param show pci/${iface_bdf} name tc1_qps_per_poller
Example Configuration Using devlink Commands
To configure 4 queues of TC1 to be polled by each independent poller:
If TC1 has a total of 8 $queues, the above command will configure 2 independent pollers, where each poller is used to poll 4 queues. If TC1 has a total of 16 $queues, the above command will configure 4 independent pollers, each polling 4 queues.
To set the timeout value in jiffies for TC1 when no traffic is flowing (setting to 5000 jiffies in this example, default is 10000 jiffies):
Example Configuration Using ADQ Setup Script
- To configure TC1 with 8 queues using 2 independent pollers (4 queue pairs per poller) with default poller timeout:
# adqsetup --dev=ens8f0 create myapp queues 8 pollers 2 ports 7000 - To configure TC1 with 16 queues using 4 independent pollers (4 queue pairs per poller) with default poller timeout:
# adqsetup --dev=ens8f0 create myapp queues 16 pollers 4 ports 7000 - To configure TC1 with 16 queues using 8 independent pollers (2 queue pairs per poller) with modified poller timeout of 5000 jiffies:
# adqsetup --dev=ens8f0 create myapp queues 16 pollers 8 \ poller_timeout 5000 ports 7000 Note:The last example uses modified poller_timeout argument (using underscore rather than hyphen). This is a known issue with adqsetup not recognizingpoller-timeout , fixed in adqsetup v1.2.3.Note:The adqsetup examples above specify only the device and port (required) and parameters to configure independent pollers. Additional recommended parameters, depending on the workload: --priority=skbedit to configure clsact qdisc and create egress TC filters for Tx traffic (instead of using Linux cgroups to align Tx packets to queues)--rxadapt=off --rxusecs=0 --txadapt=off --txusecs=500 to set the interrupt moderation rate to a static value for Tx and turn off interrupt moderation for Rx. These values are recommended for most ADQ workloads. Setting rx-usecs to a non-zero value, such as 50 or 100, might improve interrupt handling and ADQ performance for certain applications and workloads. Experiment withrxusecs=0, 50, 100 to determine best ADQ performance.Note:Global system settings for --busypoll ,--busyread ,--optimize ,--bpstop , and--bpstop-cfg are not used whenpollers is specified. Independent pollers use the $poller_timeout for the TC, rather than global sysctl settings.