Intel® Ethernet Controller E810 Application Device Queues (ADQ)

Configuration Guide

ID Date Version Classification
609008 07/04/2024 Public
Document Table of Contents
Document Table of Contents
Notices and Disclaimers Revision History Fixed and Known Issues - Read First Introduction ADQ Requirements ADQ System Under Test (SUT) Installation General System Tuning ADQ Setup Using ADQ Setup Script ADQ Configuration on SR-IOV Virtual Functions Troubleshooting Testing ADQ with Netperf Testing ADQ with Redis Testing ADQ with Memcached/rpc-perf Testing ADQ with NGINX Testing ADQ with AF_XDP Testing ADQ with Apache Traffic Server (ATS) Testing ADQ with VirtIO Appendix
Notices and Disclaimers
Revision History
Fixed and Known Issues - Read First
Fixed and Known Issues - Read First Fixed Issues Known Issues
Fixed Issues
ADQ PF Issues ADQ Setup Script Issues ADQ VF Issues
ADQ PF Issues
ADQ Setup Script Issues
ADQ VF Issues
Known Issues
ADQ PF Known Issues ADQ Setup Script Known Issues ADQ VF Known Issues
ADQ PF Known Issues
ADQ Setup Script Known Issues
ADQ VF Known Issues
Introduction
Introduction ADQ Overview ADQ Reference
ADQ Overview
ADQ Reference
ADQ Requirements
Hardware Requirements Software Requirements
Hardware Requirements
Software Requirements
ADQ System Under Test (SUT) Installation
Install OS Update Kernel (If Needed) Install ice Driver with ADQ Flags Install iproute2 (Update If Needed) Install cgroup Packages (If Needed)
Install OS
Update Kernel (If Needed)
Install ice Driver with ADQ Flags
Install iproute2 (Update If Needed)
Install cgroup Packages (If Needed)
General System Tuning
ADQ Setup Using ADQ Setup Script
ADQ Setup Using ADQ Setup Script ADQ Setup Script Prerequisites ADQ Setup Script Installation ADQ Setup Script Usage
ADQ Setup Script Prerequisites
ADQ Setup Script Installation
ADQ Setup Script Usage
ADQ Configuration on SR-IOV Virtual Functions
ADQ Configuration on SR-IOV Virtual Functions General SUT Configuration for VFs SUT Configuration Bare Metal VF SUT Configuration VF Inside a VM ADQ VF Configuration
General SUT Configuration for VFs
SUT Configuration Bare Metal VF
SUT Configuration VF Inside a VM
ADQ VF Configuration
Troubleshooting
Testing ADQ with Netperf
Testing ADQ with Netperf Installation and Configuration – Both Systems netperf Server netperf Client
Installation and Configuration – Both Systems
netperf Server
netperf Client
Testing ADQ with Redis
Testing ADQ with Redis Redis Server Setup Redis Clients (non-ADQ) Configuration
Redis Server Setup
Redis Clients (non-ADQ) Configuration
Testing ADQ with Memcached/rpc-perf
Testing ADQ with Memcached/rpc-perf Memcached Server Setup Memcached/rpc-perf Clients (non-ADQ) Configuration
Memcached Server Setup
Memcached/rpc-perf Clients (non-ADQ) Configuration
Testing ADQ with NGINX
Testing ADQ with NGINX NGINX Server Setup wrk Clients Configuration
NGINX Server Setup
wrk Clients Configuration
Testing ADQ with AF_XDP
Testing ADQ with AF_XDP AF_XDP Server Configuration Traffic Generator Example Configuration (Non-ADQ)
AF_XDP Server Configuration
Traffic Generator Example Configuration (Non-ADQ)
Traffic Generator Example Configuration (Non-ADQ) Traffic Pattern to Be Generated Example: Running DPDK-pktgen Traffic
Traffic Pattern to Be Generated
Example: Running DPDK-pktgen Traffic
Testing ADQ with Apache Traffic Server (ATS)
Testing ADQ with Apache Traffic Server (ATS) ATS Server Setup Vegeta Clients Configuration
ATS Server Setup
Vegeta Clients Configuration
Testing ADQ with VirtIO
Testing ADQ with VirtIO Installation and Configuration - Both Systems Run the Traffic
Installation and Configuration - Both Systems
Run the Traffic
Appendix
ADQ Configuration Reference (Manual)
ADQ Configuration Reference (Manual)
ADQ Configuration on SUT
ADQ Configuration on SUT
ADQ Configuration on SUT Adapter Preparation Configure ADQ Traffic Class (TC) on SUT Independent Pollers (Optional) Apply Adapter Tuning Configure Symmetric Queues Configure Intel® Ethernet Flow Director Settings Set cgroup Priority for Application Network Traffic (If Needed) Verify ADQ Application Traffic and Independent Pollers (If Applicable) Clear the ADQ Configuration
Adapter Preparation
Configure ADQ Traffic Class (TC) on SUT
Configure ADQ Traffic Class (TC) on SUT Create TCs TC Filtering Confirm TC Configuration
Create TCs
Create TCs TC Minimum Rate Creation TC Maximum Rate Creation TC Minimum and Maximum Rate Creation
TC Minimum Rate Creation
TC Maximum Rate Creation
TC Minimum and Maximum Rate Creation
TC Filtering
TC Filtering TC Filtering Requirements
TC Filtering Requirements
Confirm TC Configuration
Independent Pollers (Optional)
Independent Pollers (Optional) Independent Pollers Introduction Configure Independent Pollers
Independent Pollers Introduction
Configure Independent Pollers
Apply Adapter Tuning
Configure Symmetric Queues
Configure Intel® Ethernet Flow Director Settings
Set cgroup Priority for Application Network Traffic (If Needed)
Verify ADQ Application Traffic and Independent Pollers (If Applicable)
Clear the ADQ Configuration

TC Filtering

Note:Due to timing issues, applying TC filters immediately after the tc qdisc add command might result in the filters not being offloaded in hardware. An error in dmesg is logged if the filter fails to add properly. It is recommended to wait five seconds after tc qdisc add before adding TC filters. sleep 5

Rx and Tx filtering for ADQ application traffic can be configured with the iproute2 utility. ADQ supports a wide variety of filter options and protocols, summarized here with examples listed below in this section:

IPv4 Protocol

  • src/dst IP + port (Rx and Tx)
  • DMAC (destination MAC)
  • Forward to a queue using classid
  • GTP Protocol
  • VLAN Protocol
  • VXLAN Protocol
  • 256 queue pairs (256 MSI-X interrupts) per-PF

IPv6 Protocol

  • src/dst IP + port

Queues from TC1 are used as a destination for the ADQ filter, which is a combination of the destination IP Address plus the destination port. This aligns both transmit and receive traffic on queues belonging to TC1.

Note:The /32 in dst_​ip $ipaddr/32 is not the subnet of the network being used, but the subnet of the filter you are creating. In other words, /32 indicates a single IP Address being filtered. It is recommended to use /32 when creating filters to limit the addresses being filtered.

Example Using IP Protocol

Method 1: ingress filter for RX (requires a cgroup to align TX):

${pathtotc}/tc qdisc add dev $iface clsact ${pathtotc}/tc filter add dev $iface protocol ip ingress prio 1 flower dst_ip \ $ipaddr/32 ip_proto tcp dst_port $app_port skip_sw hw_tc 1 ${pathtotc}/tc filter show dev $iface ingress

Method 2: ingress/egress filters for RX/TX (cgroup not required):

${pathtotc}/tc qdisc add dev $iface clsact ${pathtotc}/tc filter add dev $iface protocol ip ingress prio 1 flower dst_ip \ $ipaddr/32 ip_proto tcp dst_port $app_port skip_sw hw_tc 1 ${pathtotc}/tc filter add dev $iface protocol ip egress prio 1 flower src_ip \ $ipaddr/32 ip_proto tcp src_port $app_port action skbedit priority 1 ${pathtotc}/tc filter show dev $iface ingress ${pathtotc}/tc filter show dev $iface egress

Example Using DMAC (Destination MAC)

Notes:
  • This feature is supported by ice driver versions 1.9.7 and later.
  • In bridged networking setups, the DMAC of the incoming packets will be the virtual interface's MAC, where PF will be added to OVS or bridge for external connectivity to VMs/Containers.
${pathtotc}/tc qdisc add dev $iface clsact ${pathtotc}/tc filter add dev $iface ingress protocol ip prio 1 flower skip_sw \ dst_mac $macaddr dst_ip $ipaddr/32 hw_tc 1 ${pathtotc}/tc filter show dev $iface ingress

Example of Forwarding to a Queue Using classid

Note:With ice driver version 1.9.7 and later, TC filtering can be used to directly forward ingress traffic to a specific queue rather than just a qdisc queue set. Forwarding packets to a specific device queue can be used as an alternative to ntuple perfect filter rules which direct each application TCP/UDP port to a specific queue within the TC range. See Configure Intel Ethernet Flow Director Settings for more details. ${pathtotc}/tc qdisc add dev $iface clsact ${pathtotc}/tc filter add dev $iface protocol ip ingress prio 1 flower dst_ip \ $ipaddr/32 ip_proto tcp dst_port $app_port skip_sw classid ffff:$queue_id

Where:

  • $queue_​id is a hexadecimal queue id starting with an initial value of 0x1

    For example: 0xb is hexadecimal queue id and it corresponds to queue 10 [0xa + 0x1]

${pathtotc}/tc filter show dev $iface ingress

Example Using GTP Protocol

Notes:
  1. The feature is supported by ice driver versions 1.7.x and later.
  2. GTP filters require the Telecommunications DDP Package to be loaded on the PF port before TC configuration begins. For details on DDP and how to load the package, refer the Intel® Ethernet Controller E810 Dynamic Device Personalization (DDP) Technology Guide.

    The Device Dynamic Personalization (DDP) Telecommunication package can be downloaded here:

    https://downloadcenter.intel.com/download/29889/Intel-Ethernet-800-Series-Telecommunication-Comms-Dynamic-Device-Personalization-DDP-Package

${pathtotc}/tc qdisc add dev $iface clsact ${pathtotc}/tc filter add dev $iface protocol ip parent ffff: prio 1 flower dst_ip $ipaddr/32 ip_proto udp dst_port $app_port enc_key_id $tunnel_id enc_dst_port $gtp_port skip_sw hw_tc 1 ${pathtotc}/tc filter show dev $iface ingress

Example Using VLAN Protocol

${pathtotc}/tc qdisc add dev $iface clsact ${pathtotc}/tc filter add dev $iface ingress protocol 802.1Q prio 1 flower vlan_id $vlan_id vlan_ethtype $vlan_ethtype ip_proto tcp dst_ip $ipaddr/32 dst_port \ $app_port skip_sw hw_tc 1 ${pathtotc}/tc filter show dev $iface ingress

Example Using VXLAN Protocol

Note:The feature is supported by ice driver versions 1.6.x and later. ${pathtotc}/tc qdisc add dev $vxlan clsact ${pathtotc}/tc filter add dev $vxlan protocol ip ingress flower dst_ip \ $ipaddr/32 ip_proto $protocol dst_port $app_port enc_key_id $tunnel_id hw_tc 1 ${pathtotc}/ tc filter show dev $vxlan protocol ip ingress

Example Using IPv6 Protocol

Note:This feature is supported by ice driver versions 1.7.x and later. ${pathtotc}/tc qdisc add dev $iface clsact ${pathtotc}/tc filter add dev $iface protocol ip ingress prio 1 flower dst_ip \ $ipv6addr ip_proto tcp dst_port $app_port skip_sw hw_tc 1 ${pathtotc}/tc filter show dev $iface ingress Note:Unlike IPv4 filters, dst ip $ipv6addr does not require netmask to be defined.

Parameter Definitions

  • dst_​ip – IP V4/V6 Address of DUT (in VXLAN filters, it is VXLAN interface IP).
  • dst_​mac – MAC Address of DUT.
  • dst_​port – Destination port.
  • flower – Flow-based traffic control filter.
  • ip_​proto – IP protocol to use (TCP or UDP).
  • prio – Priority.
  • protocol – Encapsulation protocol (valid options are IP, and 802.1Q).
  • skip_​sw – Flag to add the rule only in hardware.
  • hw_​tc <tc > – Route incoming traffic flow to this hardware TC. The TC count starts at 0. For example, hw_​tc 1 indicates that the filter is on the second TC.
  • vlan_​id – VLAN ID.
  • vlan_​ethtype – Type of Ethernet (IPv4).
  • enc_​key_​ID – GTP Tunnel Key ID (in VXLAN filters, it is network identifier).
  • enc_​dst_​port – GTP port number.
  • classid - Queue id specification
  • ffff - qdisc ID (fixed value - tc qdisc show dev $iface ingress/clsact)
TC Filtering Requirements