Intel® Ethernet 800 Series
Linux Performance Tuning Guide
Transmit Balancing
The 800 Series devices allow you to enable a Transmit Balancing feature to improve transmit queue fairness under certain conditions. When the feature is enabled, you should experience more consistent transmit performance across queues and/or PFs and VFs.
The default Tx queue assignment is allocated in groups of eight at the highest layer and eight VFs per parent node at the VSI layer. Tx operations are scheduled at each layer across each of the nodes in the layer.
Example tree with Tx Balancing feature disabled (default), using a single E810 port with 80 queues.
- Minimum guaranteed bandwidth is equally allocated based on the grouping at each layer.
- Maximum bandwidth depends on location of adjacent threads.
Default Tx Tree
With the Transmit Balancing feature enabled, the Tx tree architecture is flattened to allow up to 512 queues allocated to the highest level and up to 64 VFs in a parent node. This change does not reduce the overall number of queues the device supports.
Example tree with Tx Balancing feature enabled, using a single E810 port with 80 queues.
- Minimum guaranteed bandwidth is equally allocated across all queues.
- Maximum bandwidth depends on location of adjacent threads.
Tx Balancing Enabled
By default, transmit balancing is disabled in the NVM. Enabling the transmit balancing feature is recommended in environments where highly-scaled workloads are present and uneven transmit performance is seen between threads or VM groups. Not all configurations and workloads see the same benefit from enabling Tx Balancing, as the feature is highly dependent on the number of tx queues, how they get mapped out by the default Tx tree scheduler, workload and TX traffic flows, HT on/off, VFs, and so on. If uneven Tx bandwidth is observed in your environment, it is recommended to enable the transmit balancing feature to see if it helps with more even distribution of Tx traffic flows.
To enable the transmit balancing feature, use one of the following methods to persistently change the setting for the device:
- Use the Ethernet Port Configuration Tool (EPCT) to enable the tx_balancing option. Refer to the EPCT README file for more information.
- Enable the Transmit Balancing device setting system BIOS or in UEFI HII.
- Enable transmit balancing via Linux devlink command (see below).
When the driver loads, it reads the transmit balancing setting from the NVM and configures the device accordingly.
- The user selection for transmit balancing in EPCT, system BIOS, HII, or Linux devlink is a persistent setting in the firmware of the device.
- You must reboot the system for the selected setting to take effect.
- This setting is physical interface (PF) specific.
- The driver, NVM, and DDP package must all support this functionality to enable the feature. If an earlier version of the driver or NVM are installed that do not support the Tx Balancing feature, the firmware loads the default settings (with transmit balancing disabled).
Requirements
Configuration
- To set the transmit balancing feature via devlink:
devlink dev param set <pci/D:b:d.f> name txbalancing value <setting> cmode permanent - Where:
- To show the current transmit balancing setting:
devlink dev param show [ <pci/D:b:d.f> name txbalancing ] - To view the device's transmit queue topology:
- Clear dmesg:
dmesg -C - Get the Tx tree specific device port using the PCI address xx:xx.x. Use ethtool -i to find PCIe bus details for the port.
echo get scheduling tree topology > /sys/kernel/debug/ice/0000:xx:xx.x/command - Pipe dmesg to output.txt:
dmesg > output.txt The specific output depends on the VSI instances that have been configured in your environment. Note the values of the Node, Parent, and num_children. With the default configuration the num_children max allocation is eight per parent. With the feature enabled, the limit is 512.
grep Node output.txt
- Clear dmesg: