Intel® Ethernet Controller E810 Application Device Queues (ADQ)

Configuration Guide

ID Date Version Classification
609008 07/04/2024 Public
Document Table of Contents
Document Table of Contents
Notices and Disclaimers Revision History Fixed and Known Issues - Read First Introduction ADQ Requirements ADQ System Under Test (SUT) Installation General System Tuning ADQ Setup Using ADQ Setup Script ADQ Configuration on SR-IOV Virtual Functions Troubleshooting Testing ADQ with Netperf Testing ADQ with Redis Testing ADQ with Memcached/rpc-perf Testing ADQ with NGINX Testing ADQ with AF_XDP Testing ADQ with Apache Traffic Server (ATS) Testing ADQ with VirtIO Appendix
Notices and Disclaimers
Revision History
Fixed and Known Issues - Read First
Fixed and Known Issues - Read First Fixed Issues Known Issues
Fixed Issues
ADQ PF Issues ADQ Setup Script Issues ADQ VF Issues
ADQ PF Issues
ADQ Setup Script Issues
ADQ VF Issues
Known Issues
ADQ PF Known Issues ADQ Setup Script Known Issues ADQ VF Known Issues
ADQ PF Known Issues
ADQ Setup Script Known Issues
ADQ VF Known Issues
Introduction
Introduction ADQ Overview ADQ Reference
ADQ Overview
ADQ Reference
ADQ Requirements
Hardware Requirements Software Requirements
Hardware Requirements
Software Requirements
ADQ System Under Test (SUT) Installation
Install OS Update Kernel (If Needed) Install ice Driver with ADQ Flags Install iproute2 (Update If Needed) Install cgroup Packages (If Needed)
Install OS
Update Kernel (If Needed)
Install ice Driver with ADQ Flags
Install iproute2 (Update If Needed)
Install cgroup Packages (If Needed)
General System Tuning
ADQ Setup Using ADQ Setup Script
ADQ Setup Using ADQ Setup Script ADQ Setup Script Prerequisites ADQ Setup Script Installation ADQ Setup Script Usage
ADQ Setup Script Prerequisites
ADQ Setup Script Installation
ADQ Setup Script Usage
ADQ Configuration on SR-IOV Virtual Functions
ADQ Configuration on SR-IOV Virtual Functions General SUT Configuration for VFs SUT Configuration Bare Metal VF SUT Configuration VF Inside a VM ADQ VF Configuration
General SUT Configuration for VFs
SUT Configuration Bare Metal VF
SUT Configuration VF Inside a VM
ADQ VF Configuration
Troubleshooting
Testing ADQ with Netperf
Testing ADQ with Netperf Installation and Configuration – Both Systems netperf Server netperf Client
Installation and Configuration – Both Systems
netperf Server
netperf Client
Testing ADQ with Redis
Testing ADQ with Redis Redis Server Setup Redis Clients (non-ADQ) Configuration
Redis Server Setup
Redis Clients (non-ADQ) Configuration
Testing ADQ with Memcached/rpc-perf
Testing ADQ with Memcached/rpc-perf Memcached Server Setup Memcached/rpc-perf Clients (non-ADQ) Configuration
Memcached Server Setup
Memcached/rpc-perf Clients (non-ADQ) Configuration
Testing ADQ with NGINX
Testing ADQ with NGINX NGINX Server Setup wrk Clients Configuration
NGINX Server Setup
wrk Clients Configuration
Testing ADQ with AF_XDP
Testing ADQ with AF_XDP AF_XDP Server Configuration Traffic Generator Example Configuration (Non-ADQ)
AF_XDP Server Configuration
Traffic Generator Example Configuration (Non-ADQ)
Traffic Generator Example Configuration (Non-ADQ) Traffic Pattern to Be Generated Example: Running DPDK-pktgen Traffic
Traffic Pattern to Be Generated
Example: Running DPDK-pktgen Traffic
Testing ADQ with Apache Traffic Server (ATS)
Testing ADQ with Apache Traffic Server (ATS) ATS Server Setup Vegeta Clients Configuration
ATS Server Setup
Vegeta Clients Configuration
Testing ADQ with VirtIO
Testing ADQ with VirtIO Installation and Configuration - Both Systems Run the Traffic
Installation and Configuration - Both Systems
Run the Traffic
Appendix
ADQ Configuration Reference (Manual)
ADQ Configuration Reference (Manual)
ADQ Configuration on SUT
ADQ Configuration on SUT
ADQ Configuration on SUT Adapter Preparation Configure ADQ Traffic Class (TC) on SUT Independent Pollers (Optional) Apply Adapter Tuning Configure Symmetric Queues Configure Intel® Ethernet Flow Director Settings Set cgroup Priority for Application Network Traffic (If Needed) Verify ADQ Application Traffic and Independent Pollers (If Applicable) Clear the ADQ Configuration
Adapter Preparation
Configure ADQ Traffic Class (TC) on SUT
Configure ADQ Traffic Class (TC) on SUT Create TCs TC Filtering Confirm TC Configuration
Create TCs
Create TCs TC Minimum Rate Creation TC Maximum Rate Creation TC Minimum and Maximum Rate Creation
TC Minimum Rate Creation
TC Maximum Rate Creation
TC Minimum and Maximum Rate Creation
TC Filtering
TC Filtering TC Filtering Requirements
TC Filtering Requirements
Confirm TC Configuration
Independent Pollers (Optional)
Independent Pollers (Optional) Independent Pollers Introduction Configure Independent Pollers
Independent Pollers Introduction
Configure Independent Pollers
Apply Adapter Tuning
Configure Symmetric Queues
Configure Intel® Ethernet Flow Director Settings
Set cgroup Priority for Application Network Traffic (If Needed)
Verify ADQ Application Traffic and Independent Pollers (If Applicable)
Clear the ADQ Configuration

Configure Intel® Ethernet Flow Director Settings

Linux traffic control filtering allows an application to be mapped to a set of specified queues on an interface, using dedicated queue sets (also called traffic classes) for application traffic with ADQ. Intel® Ethernet Flow Director allows for incoming connections to be distributed and load balanced across these application queues, within the queue set, in a more reliably defined way than the default RSS hash. This is important to the inner workings of ADQ because it is recommended that no two application threads are busy polling on the same queue.

There are two options for Intel® Ethernet Flow Director configuration in the ice driver:

  • Option 1: channel inline or per-TC inline
  • Option 2: ntuple

The following variables are used in the examples in this section:

$iface The interface in use (PF).
$iface_​bdf The network interface BDF notation (Bus:Device.Function) used by devlink (PF).
$port The starting port of the application.
$queues The number of queues in the TC.
$tc1offset The offset for the application TC (for example, if TC0 has 2 queues, TC1 offset would be 2).
Note:Commands in this section must be applied to the PF interface (not available on VF interfaces).

Option 1: channel inline or per-TC inline

Intel® Ethernet Flow Director distributes connections to queues in a round-robin fashion. This is recommended for ADQ-enabled, single-instance, multi-threaded applications that manage their own worker thread load distribution (such as Memcached).

ice version 1.7.x (and earlier):

channel inline: Intel® Ethernet Flow Director is configured using a global private flag and gets applied to all TCs on the interface:

ethtool --set-priv-flags $iface channel-inline-flow-director on

ice version 1.8.x (and later):

per-TC inline: Intel® Ethernet Flow Director is configured using a per-TC configuration and gets applied to only the queues within the TC specified for the interface. A different devlink command is required for each TC. Example for tc1:

iface_bdf=$(ethtool -i ${iface} | grep bus-info | awk '{print $2}') devlink dev param set pci/${iface_bdf} name tc1_inline_fd value true cmode runtime Note:Kernel with devlink param support is required for ice-1.8.x and later. See Install OS and Update Kernel (If Needed) to determine OS and kernel requirements. Note:Valid devlink param flags include tc1_​inline_​fd through tc15_​inline_​fd, to configure inline flow director on up to 16 TCs (max #TCs). Note:XPS using receive queues map must be configured for per-TC inline flow director to work correctly. See Configure Symmetric Queues to configure XPS.

Option 2: ntuple

Intel® Ethernet Flow Director assigns perfect filters to direct each application TCP port to a specific queue within the application TC range. This is useful for applications (such as Redis) where each instance of the application runs a single worker thread, and load balancing is done either manually or through a separate load balancer application.

ethtool --features $iface ntuple on for ((i=0; i < queues; i++)) do ethtool --config-ntuple $iface flow-type tcp4 dst-port $((port + i)) action \ $(((i %queues) + tc1offset)) sleep 0.5 done #verify set correctly ethtool --show-ntuple $iface Note:If rmgr: Cannot insert Rx class rule: Invalid argument error message appears, verify that no previous assignment is in place: ethtool --show-ntuple $iface

Assignments as a whole can be deleted:

ethtool --features $iface ntuple off Note: Linux forward to queue using class ID is an alternative to ntuple filter rules using the action forward to queue to forward packets to a specific device queue. Ntuple can set the specific packet flow types (i.e., tcp4, udp4) and depends on the ethtool version, RHEL 8.x, or upstream kernel to configure Ntuple flags, whereas forward to queues can be set per-filter using the class ID parameter. See TC Filtering, Example of Forwarding to a Queue Using classid as an alternative to ntuple filter rules in this section.
Set cgroup Priority for Application Network Traffic (If Needed)