Intel® Ethernet Controller E810 Application Device Queues (ADQ)

Configuration Guide

ID Date Version Classification
609008 07/04/2024 Public
Document Table of Contents
Document Table of Contents
Notices and Disclaimers Revision History Fixed and Known Issues - Read First Introduction ADQ Requirements ADQ System Under Test (SUT) Installation General System Tuning ADQ Setup Using ADQ Setup Script ADQ Configuration on SR-IOV Virtual Functions Troubleshooting Testing ADQ with Netperf Testing ADQ with Redis Testing ADQ with Memcached/rpc-perf Testing ADQ with NGINX Testing ADQ with AF_XDP Testing ADQ with Apache Traffic Server (ATS) Testing ADQ with VirtIO Appendix
Notices and Disclaimers
Revision History
Fixed and Known Issues - Read First
Fixed and Known Issues - Read First Fixed Issues Known Issues
Fixed Issues
ADQ PF Issues ADQ Setup Script Issues ADQ VF Issues
ADQ PF Issues
ADQ Setup Script Issues
ADQ VF Issues
Known Issues
ADQ PF Known Issues ADQ Setup Script Known Issues ADQ VF Known Issues
ADQ PF Known Issues
ADQ Setup Script Known Issues
ADQ VF Known Issues
Introduction
Introduction ADQ Overview ADQ Reference
ADQ Overview
ADQ Reference
ADQ Requirements
Hardware Requirements Software Requirements
Hardware Requirements
Software Requirements
ADQ System Under Test (SUT) Installation
Install OS Update Kernel (If Needed) Install ice Driver with ADQ Flags Install iproute2 (Update If Needed) Install cgroup Packages (If Needed)
Install OS
Update Kernel (If Needed)
Install ice Driver with ADQ Flags
Install iproute2 (Update If Needed)
Install cgroup Packages (If Needed)
General System Tuning
ADQ Setup Using ADQ Setup Script
ADQ Setup Using ADQ Setup Script ADQ Setup Script Prerequisites ADQ Setup Script Installation ADQ Setup Script Usage
ADQ Setup Script Prerequisites
ADQ Setup Script Installation
ADQ Setup Script Usage
ADQ Configuration on SR-IOV Virtual Functions
ADQ Configuration on SR-IOV Virtual Functions General SUT Configuration for VFs SUT Configuration Bare Metal VF SUT Configuration VF Inside a VM ADQ VF Configuration
General SUT Configuration for VFs
SUT Configuration Bare Metal VF
SUT Configuration VF Inside a VM
ADQ VF Configuration
Troubleshooting
Testing ADQ with Netperf
Testing ADQ with Netperf Installation and Configuration – Both Systems netperf Server netperf Client
Installation and Configuration – Both Systems
netperf Server
netperf Client
Testing ADQ with Redis
Testing ADQ with Redis Redis Server Setup Redis Clients (non-ADQ) Configuration
Redis Server Setup
Redis Clients (non-ADQ) Configuration
Testing ADQ with Memcached/rpc-perf
Testing ADQ with Memcached/rpc-perf Memcached Server Setup Memcached/rpc-perf Clients (non-ADQ) Configuration
Memcached Server Setup
Memcached/rpc-perf Clients (non-ADQ) Configuration
Testing ADQ with NGINX
Testing ADQ with NGINX NGINX Server Setup wrk Clients Configuration
NGINX Server Setup
wrk Clients Configuration
Testing ADQ with AF_XDP
Testing ADQ with AF_XDP AF_XDP Server Configuration Traffic Generator Example Configuration (Non-ADQ)
AF_XDP Server Configuration
Traffic Generator Example Configuration (Non-ADQ)
Traffic Generator Example Configuration (Non-ADQ) Traffic Pattern to Be Generated Example: Running DPDK-pktgen Traffic
Traffic Pattern to Be Generated
Example: Running DPDK-pktgen Traffic
Testing ADQ with Apache Traffic Server (ATS)
Testing ADQ with Apache Traffic Server (ATS) ATS Server Setup Vegeta Clients Configuration
ATS Server Setup
Vegeta Clients Configuration
Testing ADQ with VirtIO
Testing ADQ with VirtIO Installation and Configuration - Both Systems Run the Traffic
Installation and Configuration - Both Systems
Run the Traffic
Appendix
ADQ Configuration Reference (Manual)
ADQ Configuration Reference (Manual)
ADQ Configuration on SUT
ADQ Configuration on SUT
ADQ Configuration on SUT Adapter Preparation Configure ADQ Traffic Class (TC) on SUT Independent Pollers (Optional) Apply Adapter Tuning Configure Symmetric Queues Configure Intel® Ethernet Flow Director Settings Set cgroup Priority for Application Network Traffic (If Needed) Verify ADQ Application Traffic and Independent Pollers (If Applicable) Clear the ADQ Configuration
Adapter Preparation
Configure ADQ Traffic Class (TC) on SUT
Configure ADQ Traffic Class (TC) on SUT Create TCs TC Filtering Confirm TC Configuration
Create TCs
Create TCs TC Minimum Rate Creation TC Maximum Rate Creation TC Minimum and Maximum Rate Creation
TC Minimum Rate Creation
TC Maximum Rate Creation
TC Minimum and Maximum Rate Creation
TC Filtering
TC Filtering TC Filtering Requirements
TC Filtering Requirements
Confirm TC Configuration
Independent Pollers (Optional)
Independent Pollers (Optional) Independent Pollers Introduction Configure Independent Pollers
Independent Pollers Introduction
Configure Independent Pollers
Apply Adapter Tuning
Configure Symmetric Queues
Configure Intel® Ethernet Flow Director Settings
Set cgroup Priority for Application Network Traffic (If Needed)
Verify ADQ Application Traffic and Independent Pollers (If Applicable)
Clear the ADQ Configuration

Create TCs

The following example creates three TCs with a small number of queues for TC0 (default TC) and the remaining number of logical cores divided between TC1 and TC2 (application TC) queues. In general, two queues for TC0 and the remaining number of cores for the application TCs are sufficient.

For best performance, the number of application worker threads/processes should match with the number of queues configured for a given application-specific TC.

Traffic originating from an application with priority set to 1 is mapped to TC1, with priority 2 mapped to TC2, etc. All default traffic is mapped to TC0.

Note:The allocation of bandwidth between TCs is not guaranteed to be proportional to the number of queues assigned. By default, the bandwidth available is roughly evenly split among all TCs. If a certain minimum bandwidth is required for a particular TC, it is recommended to use rate limiting to set either maximum or minimum bandwidth limits with the TC shaper bw_​rlimit parameter.

Due to resource allocation rules, the following rules apply to the number of TC queues:

  • For TC0: The number of queues for the default traffic class ($num_​queues_​tc0) must always be a power of two.
  • For TC1 or higher:
    • If all TCs are powers of two, they can be any number.
    • If the number of queues for any TC (for TC1 or above) is not a power of two, any subsequent TC must be either exactly that number or a power of two that is less than that number.
  • ice-1.8.x and earlier: The total number of queues from all TCs cannot be greater than the max number of logical cores in the system.
  • ice-1.9.x and later: The total number of queues from all TCs is increased to a max number of 256 (the number of MSI-X interrupts per PF).
  • The total number of TCs allowed 16.

Example:

If num_​queues_​TC0=2, num_​queues_​TC1=16, and num_​queues_​TC2=7, then num_​queues_​TC3+ must be either exactly 7 or some power of 2 less than 7 (4,2,1).

Example:

${pathtotc}/tc qdisc add dev $iface root mqprio num_tc 3 map 0 1 2 queues \ $num_queues_tc0@0 $num_queues_tc1@$num_queues_tc0 $num_queues_tc2@$((num_queues_tc0 \ + num_queues_tc1)) hw 1 mode channel

For example, if there are 96 logical cores in a system, one possible configuration is TC0 with 2 queues, TC1 with 32 queues and TC2 with 62 queues.

${pathtotc}/tc qdisc add dev $iface root mqprio num_tc 3 map 0 1 2 queues \ 2@0 32@2 62@34 hw 1 mode channel

With ice driver version 1.9.7 and later, large TCs with up to 256 queues can be used.

Example:

To create 8 TCs with 256 queues spread across all of the TCs:

${pathtotc}/tc qdisc add dev $iface root mqprio num_tc 8 map 0 1 2 3 4 5 6 7 \ 2@0 4@2 8@6 16@14 32@30 64@62 128@126 2@254 hw 1 mode channelNote:E810 network adapters with 1 and 2 ports support up to 256 queue pairs (256 MSI-X interrupts) per-PF. The maximum number of queue pairs supported by 4 and 8 port E810 adapters is limited by the number of system cores.

Parameter Definitions

  • Map — Priority mapping for up to 16 priorities to TCs. For example, map 0 0 0 0 1 1 1 1 sets priorities 0-3 to use tc0 and 4-7 to use tc1).
  • Queues — For each TC, <num queues>@<offset>. For example, queues 2@0 16@2 assigns 2 queues to tc0 at offset 0 and 16 queues to tc1 at offset 2. A maximum total number of queues for all TCs is the number of cores.
  • hw 1 mode channel — Channel with hardware set to 1 is a new hardware offload mode in mqprio that makes full use of the mqprio options, the TCs, the queue configurations, and the QoS parameters.
TC Minimum Rate Creation