General System Tuning

1. Disable firewalls. service firewalld stop; systemctl mask firewalld
2. Disable Security-Enhanced Linux (SELinux) (requires reboot to take affect).

   Change SELINUX=enforcing to SELINUX=disabled in /etc/selinux/config.

3. Enable latency-performance tuned profile, if it makes sense for the application and workload. Refer to https://github.com/redhat-performance/tuned/blob/master/profiles/latency-performance/tuned.conf for a full list of what is expected to be included in the profile. Note:This performance mode (and the performance governor in the next step) limit the CPU power management settings in the OS to provide more consistent results on a highly loaded system with many application threads. If using fewer queues than the total number of CPUs or on lightly loaded systems (for example netperf with only a few processes), skipping this step might increase performance, since under certain circumstances allowing Turbo mode on more cores can give a greater benefit than the avoidance of CPU power management sleep states (C States). Similar tuning choices should be made for CPU power management and Turbo settings in the BIOS. tuned-adm profile latency-performance Note:The tuned-adm daemon is not installed by default in RHEL9.0 systems. Install it with the command yum install tuned.

   Check that settings are applied correctly:

   cat /etc/tuned/active_profile

   Output: latency-performance

   cat /etc/tuned/profile_mode

   Output: manual

   Note:The default performance tuned setting is tuned-adm profile throughput-performance, which works well with many applications and workloads. Test both modes for optimal performance. Refer to https://github.com/redhat-performance/tuned/blob/master/profiles/throughput-performance/tuned.conf for a full list of what is expected to be included in the profile.
4. Set the CPU scaling governor to performance mode. x86_energy_perf_policy performance
5. Increase the hard/soft nofile limits. vi /etc/security/limits.conf * hard nofile 16384 * soft nofile 16384 Note:Some OS distros do not allow a non-root user to set limits >64, even when listed in /etc/security/limits.conf. A workaround is to set DefaultLimitMEMLOCK= in /etc/systemd/system.conf and /etc/systemd/user.conf, where the limit is defined in bytes.

   Example:

   The following output is on a system that has the default limit set to 1GB:

   # grep ^DefaultLimitMEMLOCK /etc/systemd/system.conf DefaultLimitMEMLOCK=1073741824 # grep ^DefaultLimitMEMLOCK /etc/systemd/user.conf DefaultLimitMEMLOCK=1073741824
6. Stop the irqbalance service. (Needed for interface interrupt affinity settings.) systemctl stop irqbalance
7. Stop LLDP agent (if installed and running). systemctl stop lldpad.service systemctl stop lldpad.socket
8. Set system network polling parameters. For the application testing sections listed in this document, follow the recommended parameters.

   In general, whether to use busy_​poll alone or busy_​poll and busy_​read depends on the architecture of the application being used.

   For a majority of applications, which use epoll_​wait/epoll_​pwait followed by read/recv, it is recommended to set only net.core.busy_​poll to the number of microseconds to wait for packets on the device queue for socket polling. Recommend 50K µs (50 ms) for a starting point to stay in poll mode as much as possible. This gives time for up to thousands of connections to be formed before timing out and dropping back to interrupt mode.

   sysctl -w net.core.busy_poll=50000

   Some applications, such as netperf, which uses select/poll followed by read/recv, require net.core.busy_​read to also be set along with net.core.busy_​poll:

   Example:

   sysctl -w net.core.busy_poll=50000 sysctl -w net.core.busy_read=50000 Notes:
   • If setting system-wide busy_​read is not acceptable, busy_​read can be set per socket in the application using socket option SO_​BUSY_​LOOP.
   • The system-wide busy_​poll and busy_​read settings are only required when using application dependent polling mode; they are not required when using independent pollers.
   • Newer Linux kernels (6.9.X or later) extend epoll busy polling support to make it configurable per epoll context rather than the existing system wide /proc/sys/net/core/busy_​poll setting, which affects all applications. This allows users to run low latency epoll-based server applications with epoll-based busy poll while leaving the rest of the system's applications (which may also use epoll) unaffected. EPIOCSPARAMS and EPIOCGPARAMS are new epoll ioctl commands for configuring and querying busy_​poll_​usecs, busy_​poll_​budet, and prefer_​busy_​poll context parameters.
   • When using the ADQ setup script to configure ADQ for application dependent polling (ADQ1.0), global busy_​poll and busy_​read settings are not required to be set manually. The script can handle these as well.
9. Configure OS tuning parameters to handle increased packet rate (note that some of these parameters will increase memory usage on the system):
   1. Increase the maximum number of connection requests queued for any listening socket. sysctl -w net.core.somaxconn=4096
   2. Increase the maximum number of packets allowed to queue when a particular interface receives packets faster than the kernel can process them. sysctl -w net.core.netdev_max_backlog=8192 Note:Set netdev_​max_​backlog to 250000 to improve ADQ performance with Apache Traffic Server (ATS).
   3. Increase the maximum length of the queue for incomplete TCP sockets (SYN RECEIVED state). sysctl -w net.ipv4.tcp_max_syn_backlog=16384 Note:Set tcp_​max_​syn_​backlog to 250000 to improve ADQ performance with ATS. Note:Applications might have their own backlog settings that could need to be adjusted. If backlog is set too low, performance can be impacted due to uneven queue use. Note:If the TCP backlog setting is too low, you might see dmesg logs such as: TCP: request_​sock_​TCP: Possible SYN flooding.
   4. Increase the Rx and Tx socket buffer maximum size (in bytes). sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216
   5. Adjust TCP memory usage auto-tuning for the stack (low, pressure, and max - in pages). sysctl -w net.ipv4.tcp_mem="764688 1019584 16777216"
   6. Adjust TCP buffer auto-tuning for each connection (min, default, and max - in bytes). sysctl -w net.ipv4.tcp_rmem="8192 87380 16777216" sysctl -w net.ipv4.tcp_wmem="8192 65536 16777216"
   7. Apply immediately. sysctl -w net.ipv4.route.flush=1

For convenience, the commands in this section (except for disabling SELinux, which requires a reboot) are condensed below: