​CPU Utilization

If throughput is below expectation, first confirm the link speed, PCIe width and speed, driver and NVM versions, CPU utilization per core, and packet drops. If these checks are clean, continue with workload-level tuning, such as application thread count, queue distribution, and benchmark process placement.

Check CPU utilization per core while the workload is running. Per-core utilization is more relevant to performance than overall CPU utilization because it shows CPU utilization per network queue. If only a few threads are generating network traffic, only a few cores may be used. However, if those cores are at 100%, network throughput is likely limited by CPU utilization, and you should take the following action:

Increase the number of application threads to spread CPU load across more cores. If all cores are running at 100%, the application may be CPU-bound rather than network-bound.

Commonly available tools to monitor CPU core utilization:

• top
  • Press 1 to expand list of CPUs and check which ones are being used.
  • Notice the level of utilization.
  • Notice which processes are listed as most active (top of list).
• mpstat

  The following example command line was tested on Red Hat Enterprise Linux 9.x. It displays CPU utilization per core (by finding the total percent idle and subtracting from 100) and highlights the values above 80% in red.

  mpstat -P ALL 1 1 | grep -v Average | tail -n +5 | head -n -1 | awk '{ print (100-$13)}' | egrep '[^\.][8-9][0-9][\.]?.*|^[8-9][0-9][\.]?.*| 100' | column
• perf top

  Look for where cycles are being spent.