NGINX Server Setup

Intel® Ethernet Controller E810 Application Device Queues (ADQ)

Configuration Guide

Download as ZIP

ID 609008

Date 04/03/2023

Version 2.8

Note:This section uses application dependent polling (ADQ1.0) for application and testing and does not use an independent poller (ADQ2.0).

The following variables are used in the examples in this section:

$iface	The interface in use (PF).
$num_queues_tc0	The number of queues for default traffic (note that this should be a power of 2, for example, 2).
$num_queues_tc1	The number of queues for application traffic class (this is the number of application threads).
$ipaddr	The IP Address of the interface under test.
$app_port	The TCP port of the nginx application being run on the SUT (for example, 80 to listen NGINX HTTP server).
$application	The path to application install (for example, /opt/NGINX).
ADQ Setup Script Variables
$bp	Global busy_poll value to be set by ADQ Setup Script (system-wide setting, not to be used with independent poller).
$br	Global busy_read value to be set by ADQ Setup Script (system-wide setting, not to be used with independent poller).
$file_name	Application name to be set and used by ADQ Setup Script (any descriptive string, i.e., nginx).
$port	TCP port to be configured by ADQ Setup Script.
$addr	Destination IP address of traffic to be configured by ADQ Setup Script

System Setup

Perform general system OS install and setup.
Complete the ADQ install and setup in ADQ System Under Test (SUT) Installation and General System Tuning on just the SUT system.
Note:For best NGINX performance with ADQ, the following default ice driver settings should be used on the NGINX Server: ethtool --set-priv-flags $iface channel-pkt-inspect-optimize on ethtool --set-priv-flags $iface channel-pkt-clean-bp-stop-cfg off ethtool --set-priv-flags $iface channel-pkt-clean-bp-stop off
The above settings can also be set using an ADQ Setup script; for more information, see Using ADQ Setup Script.

One change to the system tuning is required. Instead of tuned-adm profile latency-performance, we recommend using the throughput-performance profile.
tuned-adm profile throughput-performance Note:The tuned-adm daemon is not installed by default in RHEL9.0 systems. Install it with the command yum install tuned.
Check that the settings are applied correctly:
cat /etc/tuned/active_profile
Output: throughput-performance
cat /etc/tuned/profile_mode
Output: manual
Note:Many settings in General System Tuning do not persist between reboots and might need to be reapplied.

Perform NGINX build:
1. Install prerequisite NGINX package(s): yum install -y pcre-devel
2. Download the latest echo-nginx-module. The echo module is a testing utility that allows for multi-worker content access without file contention, improving simulated benchmark performance. Under a real workload (not simulated for benchmarking) this module would not be necessary for full ADQ performance benefit.
  Example:
  git clone https://github.com/openresty/echo-nginx-module.git
3. Download the latest stable NGINX release.
  Example:
  wget http://nginx.org/download/nginx-1.16.1.tar.gz
4. Untar the package. tar -xf nginx-1.16.1.tar.gz
5. [ADQ only] Apply the patch nginx_reuseport_bpf.patch. Note:If you are viewing this document in HTML, click the Download PDF button to download the zip package containing the nginx_reuseport_bpf patch. cd nginx-${VER} patch -p1 < <pathtopatch>/nginx_reuseport_bpf.patch Note:This nginx_reuseport_bpf patch is required in an ADQ environment, but might provide benefit even without other ADQ configuration (such as TCs and cgroups). The nginx_reuseport_bpf patch establishes a 1:1 model of application thread to hardware queue use in NGINX by adding the socket option SO_ATTACH_REUSEPORT_CBPF to the application's listening socket code. This socket option allows the application to attach a small classic Berkley Packet Filter (cBPF) program to perform hardware NIC queue-aware load balancing among the application threads.
6. Configure NGINX with applied patch. make clean ./configure --prefix=$application --add-module=<pathtoecho-nginx>/echonginxmodule
7. Copy NGINX configuration into build so that it is deployed in the install. cp -f nginx.conf conf/ Note:NGINX has its own set of configuration parameters to tune. Each application deployment might be different according to its individual use case. For full ADQ performance, the use epoll statement is required in the events {} section. To enable the nginx_reuseport_pbf patch, listen $app_port reuseport is required in the server {} section. An example nginx.conf file is included below.
  Example nginx.conf (note the $app_port variable needs to match with the application $app_port in the rest of the section):
  daemon off; error_log stderr; events { use epoll; worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; tcp_nopush on; tcp_nodelay on; open_file_cache max=1000 inactive=10m; keepalive_timeout 120; keepalive_requests 1000000; server { listen $app_port reuseport; server_name localhost; access_log off; location / { default_type text/html; echo "<html><body><h2>Test Server</h2></body></html>"; } location /test256.dat { echo_duplicate 256 "*"; } location /test512.dat { echo_duplicate 512 "*"; } location /test1024.dat { echo_duplicate 1024 "*"; } location /test2048.dat { echo_duplicate 2048 "*"; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } }
8. Install NGINX. make install
  After this, the NGINX server is found under $application/sbin.

Using ADQ Setup Script

The ADQ Setup script allows you to quickly and easily configure required ADQ parameters such as traffic classes, priority, filters, and ethtool parameters etc.

To configure ADQ, run the following command: adqsetup --dev=$iface –-priority=skbedit --busypoll=$bp create $file_name queues $num_queues_tc1 ports \ $port addrs $addr
See Notes below for customizing the ADQ configuration. Once ADQ is configured by adqsetup, start the nginx server.
Notes:
1. The example command above creates both ingress (Rx) and egress (Tx) filters, and Linux cgroups are not needed to be created and can be skipped (cgroups are only needed if --priority=skbedit was NOT specified in adqsetup command).
2. ADQ Setup script handles symmetric queues and affinity.
3. The Setup script sets per-queue coalesce by default.
4. The Setup script sets per-TC inline flow director, for ice-1.8.x and later. For earlier versions of the ice driver, the global channel-inline-flow-director flag is used by default.
5. Set the transmit and receive interrupt coalescing values to --rxadapt=off --rxusecs=0 --txadapt=off --txusecs=500 for improved ADQ performance.
6. Set the ice driver settings to --optimize on --bpstop off --bpstop-cfg off for better NGINX performance with ADQ.
7. To configure independent pollers, add the -pollers=$pollers parameter in the adqsetup command (and optionally --pollers_timeout), and remove the flags to set global --busypoll=$bp --busyread=$br.
8. Use the cpu parameter in the command to bind the independent pollers to specific CPU cores. Refer to ADQ Setup Using ADQ Setup Script for more information on pinning pollers to specific CPU cores.
9. The --debug parameter is optional, but it is useful for obtaining complete stack trace.
10. For more information on how to use the script, refer to ADQ Setup Using ADQ Setup Script.

Start NGINX either in the cgroup for ADQ, or without the cgroup if testing baseline performance.
1. [ADQ only] Start NGINX in the cgroup with the same number of threads as were configured for TC1.
  Example:
  server="${application}/sbin/nginx" ${server} -g "worker_processes ${num_queues_tc1};error_log stderr info;"
2. [Baseline] Start NGINX.
  Example:
  server="${application}/sbin/nginx" ${server} -g "worker_processes ${num_queues_tc1};error_log stderr info;"

[ADQ only] Troubleshooting — While test is running, verify that ADQ traffic is on the correct queues.

While ADQ application traffic is running, watch ethtool statistics to check that only the ADQ queues are being used (have significant traffic) with busy poll (pkt_busy_poll) for ADQ traffic. If non busy poll (pkt_not_busy_poll) have significant counts and/or if traffic is not confined to ADQ queues, recheck the configuration steps carefully.
watch -d -n 0.5 "ethtool -S $iface | grep busy | column"
See Verify ADQ Application Traffic and Independent Pollers (If Applicable) for example watch output.
Note:Note:If the benefits of ADQ enabled vs. disabled are not as dramatic as expected even though the queues appear to be well aligned, it is possible that the performance is limited by the processing power of the client systems. If client CPU shows greater than an average of 80% CPU utilization on the CPU cores in use, it is probable that the client is becoming overloaded. To achieve maximum performance benefits, try increasing the number of or processing power of the client systems.

Note:The ADQ Setup script clears the existing configuration before proceeding with the new ADQ configuration. To clear manually, follow the steps in Clear the ADQ Configuration.