System Setup

1. Perform the system OS install and setup.
   1. Complete the OS install and kernel update inADQ System Under Test (SUT) Installation on the server under the test system. Note:For busy polling, kernel version v5.11.0 or later is required. Note:Ensure XDP socket flag is enabled in the kernel .config file in Step 4c in Update Kernel (If Needed)

      Add CONFIG_​XDP_​SOCKETS=y flag in the .config file.

      If the kernel is compiled without CONFIG_​XDP_​SOCKETS=y, the following error message is observed while running the XDPSOCK application: Address family not supported by protocol.

      Note:Disable Hyper threading in BIOS for better performance with AF_​XDP.
   2. Complete the ADQ setup in General System Tuning on the SUT system.

      One change to the system tuning is recommended for improved performance with ADQ. Set the following private flags on the interface under test:

      ethtool --set-priv-flags $iface channel-pkt-inspect-optimize off ethtool --set-priv-flags $iface channel-pkt-clean-bp-stop on ethtool --set-priv-flags $iface channel-pkt-clean-bp-stop-cfg on

      The above settings can also be set using an ADQ Setup script; for more information, see Using ADQ Setup Script below.

2. Install prerequisites: Packages needed for LLVM and BPF samples: cmake, libcap-devel. yum install -y cmake libcap-devel

   If libpcap-devel is not available in the OS distribution, download and install libpcap and libpcap-devel:

   wget http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages/libpcap-1.9.1-5.el8.x86_64.rpm wget http://mirror.centos.org/centos/8/PowerTools/x86_64/os/Packages/libpcap-devel-1.9.1-5.el8.x86_64.rpm rpm -i libpcap-devel* Note:If installing packages using the rpm command, and if libpcap is already installed on the system, use rpm -U instead of rpm -i to upgrade the package. Version numbers might differ depending on OS being used.
3. Install LLVM tool.
   1. Download LLVM. wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.1/llvm-project-10.0.1.tar.xz
   2. Untar the package. tar -xf llvm-project-10.0.1.tar.xz
   3. Install LLVM. cd to the directory "llvm-project-10.0.1" mkdir build; cd build cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS=clang ../llvm make -j"$(nproc)" install Note:$(nproc) = number of logical cores. Note:LLVM requires a lot of disk space, be sure to have at least 65 GB of disk space available before building and installing LLVM.
4. Build BPF Samples.
   1. cd to the Linux kernel source (for example, /kernels/5.11.0/) and build tools. cd $kernelpath make -C tools clean
   2. Build sample bpf: make M=samples/bpf

      or

      cd $kernelpath/samples/bpf make -j all
   3. Verify that the xdpsock executable was created: ls $kernelpath/samples/bpf | grep xdpsock xdpsock xdpsock_ctrl_proc xdpsock_ctrl_proc.c xdpsock_ctrl_proc.o xdpsock.h xdpsock_kern.c xdpsock_kern.o xdpsock_user.c xdpsock_user.o

Using ADQ Setup Script

The ADQ Setup script allows you to quickly and easily configure required ADQ parameters such as traffic classes, priority, filters, and ethtool parameters etc.

1. To configure ADQ, run the following command: adqsetup --dev=$iface –-priority=skbedit --busypoll=$bp --busyread=$br create $file_name mode shared \ queues $num_queues_tc1 ports $portrange addrs $addr protocol udp

   See Notes below for customizing the ADQ configuration. Once ADQ is configured by adqsetup, launch the XDPSOCK application with multiple parallel sockets in different modes l2fwd/TX-only/RX-drop.

   Notes:
   1. ADQ Setup script handles symmetric queues and affinity.
   2. The Setup script sets per-queue coalesce by default.
   3. Set the transmit and receive interrupt coalescing values to --rxadapt=off --rxusecs=0 --txadapt=off --txusecs=500 for improved ADQ performance.
   4. Set the ice driver settings to --optimize off --bpstop on --bpstop-cfg on for better performance with ADQ.
   5. To configure independent pollers, add the -pollers=$pollers parameter in the adqsetup command (and optionally --pollers_​timeout), and remove the flags to set global --busypoll=$bp --busyread=$br.
   6. Use the cpu parameter in the command to bind the independent pollers to specific CPU cores. Refer to ADQ Setup Using ADQ Setup Script for more information on pinning pollers to specific CPU cores.
   7. The --debug parameter is optional, but it is useful for obtaining complete stack trace.
   8. For more information on how to use the script, refer ADQ Setup Using ADQ Setup Script.
2. Create and run a test script like the example below. The script opens multiple parallel sockets in the background with the selected mode: l2fwd/TX-only/RX-drop.

   Example:

   xdpsock.sh mode=$1 if [ "x${mode}" == "x" ] then echo -e "Enter mode in which the script should run: -l for L2FWD or -r for RX-DROP or -t for TX-ONLY:" echo -e "Usage: ./xdpsock.sh -l or ./xdpsock.sh -r or ./xdpsock.sh -t" exit 1 fi inst=${2:-<number_of_queues_to_test>} iface=ens1f1 kernelpath=/kernels/5.11.0/ XDPSOCK="${kernelpath}/samples/bpf/xdpsock" trap 'kill $(jobs -p)' EXIT max=$(( inst + 2 )) for (( i = 2; i < $max; i++ )) do printf -v q "%d" $((i*1)) taskset -c $i $XDPSOCK -i $iface ${mode} -q $i & #mode is -l for l2fwd, -r for RX-DROP, -t for TX-ONLY sleep 0.1 done while true do sleep 1 done
3. Run the traffic from an external packet generator of choice. See Traffic Generator Example Configuration (Non-ADQ) for details.
4. After the test finishes, kill the XDPSOCK process. killall xdpsock