Intel® Ethernet 800 Series Linux Flow Control

Configuration Guide for RDMA Use Cases

ID 635330
Date 07/13/2023
Version 1.3
Document Table of Contents
Document Table of Contents
Notices and Disclaimers Revision History Introduction Background Link-Level Flow Control Priority Flow Control - Fundamentals Congestion Management – Tuning Parameters Priority Flow Control - Planning and Guidelines Priority Flow Control - Verification Troubleshooting
Notices and Disclaimers
Revision History
Introduction
Introduction QoS/Flow Control Limitations on the 800 Series
QoS/Flow Control Limitations on the 800 Series
Background
Ethernet Flow Control Flow Control in RDMA Networks Types of Flow Control: LFC vs. PFC
Ethernet Flow Control
Flow Control in RDMA Networks
Types of Flow Control: LFC vs. PFC
Link-Level Flow Control
LFC Setup Instructions Symmetric vs. Asymmetric LFC
LFC Setup Instructions
Symmetric vs. Asymmetric LFC
Priority Flow Control - Fundamentals
Priority Flow Control - Fundamentals DCB Standards Determining PFC Priority Mode: PCP vs. DSCP Assigning an Application to a Traffic Class
DCB Standards
DCB Standards DCB Willing vs. Non-willing Modes
DCB Willing vs. Non-willing Modes
Determining PFC Priority Mode: PCP vs. DSCP
Assigning an Application to a Traffic Class
Assigning an Application to a Traffic Class Mapping Steps Details
Mapping Steps Details
Step 1: Determine Flow Control Design Needed Step 2: Kernel Priority (sk_prio) or DSCP to UP Mapping Step 3: UP to TC Step 4: Set ToS in the Application (or for All RoCEv2 Traffic)
Step 1: Determine Flow Control Design Needed
Step 2: Kernel Priority (sk_prio) or DSCP to UP Mapping
Step 3: UP to TC
Step 4: Set ToS in the Application (or for All RoCEv2 Traffic)
Congestion Management – Tuning Parameters
Overview DCQCN Congestion Control Parameter Settings
Overview
DCQCN
DCQCN DCQCN Specification
DCQCN Specification
Congestion Control Parameter Settings
Priority Flow Control - Planning and Guidelines
Priority Flow Control - Planning and Guidelines Steps Example Configurations
Steps
Steps Network Host and Switch Setup Willing vs. Non-willing DCB Mode Firmware vs. Software DCB Software DCB Willing Mode Separating and Prioritizing Traffic Streams Configuring ETS: Map Priorities to TCs/Allocate Bandwidth Configuring PFC Run Applications with the Right Priority
Network Host and Switch Setup
Willing vs. Non-willing DCB Mode
Firmware vs. Software DCB
Software DCB Willing Mode
Separating and Prioritizing Traffic Streams
Configuring ETS: Map Priorities to TCs/Allocate Bandwidth
Configuring PFC
Configuring PFC Option A: PCP PFC - Set Priorities for Drop or No-drop Option B: DSCP PFC
Option A: PCP PFC - Set Priorities for Drop or No-drop
Option B: DSCP PFC
Run Applications with the Right Priority
Example Configurations
Example 1 - 800 Series-800 Series Back-to-Back – PCP PFC with Single TC Example 2 - Adapters Connected Through a Switch – Willing Mode on Adapters, DCB Configured on Switch Example 3 - DSCP PFC with Non-Default TCs - Non-Willing Mode on Adapters, ECN Configured Example 4 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – No VLANs Example 5 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – with VLANs
Example 1 - 800 Series-800 Series Back-to-Back – PCP PFC with Single TC
Example 2 - Adapters Connected Through a Switch – Willing Mode on Adapters, DCB Configured on Switch
Example 3 - DSCP PFC with Non-Default TCs - Non-Willing Mode on Adapters, ECN Configured
Example 4 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – No VLANs
Example 5 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – with VLANs
Priority Flow Control - Verification
Priority Counters Discard Counters tpcdump
Priority Counters
Discard Counters
Discard Counters LAN Packet Drops RDMA Discards
LAN Packet Drops
RDMA Discards
tpcdump
Troubleshooting
Notices and Disclaimers
Revision History
Introduction
Background
Link-Level Flow Control
Priority Flow Control - Fundamentals
Congestion Management – Tuning Parameters
Priority Flow Control - Planning and Guidelines
Priority Flow Control - Verification
Troubleshooting

LFC Setup Instructions

Configuring LFC on an 800 Series network is relatively straightforward; enable flow control in both directions (Tx and Rx) on both sides of the link.

  • If your hosts are connected through a switch, you must also enable flow control on the switch ports.
  • If your hosts are connected back-to-back, enable LFC on both adapters.

The examples below use eth0 as the 800 Series net device name (use ip a to find the device name on your system).

Switch settings vary by manufacturer. In your switch manual, look for syntax containing words like: flowcontrol, flow-control, tx-pause, or rx-pause.

To enable LFC on your network:

  1. Disable firmware-based DCB on the adapter. # ethtool --set-priv-flags <interface> fw-lldp-agent off
  2. Verify that firmware DCB is disabled. # ethtool --show-priv-flags <interface> | grep fw-lldp-agent fw-lldp-agent : off
  3. Ensure that lldpad is not running. # ps -ef | grep lldpad
  4. Disable PFC on your switch, if applicable (show PFC status per port). switch>show priority-flow-control

    For example, disable PFC on a given port (like port 31/1) on an Arista 7060CX:

    switch>enable switch#configure switch(config)#interface Ethernet 31/1 switch(config-if-Et31/1)#no priority-flow-control Note:Some switches allow for a range of ports to be specified, for example 1/1-32/1.
  5. Enable link-level flow control on the adapter. # ethtool -A eth0 rx on tx on
  6. Check LFC settings on the adapter. # ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: on TX: on RX negotiated: on TX negotiated: on
  7. Enable flow control on the switch ports.

    For example, enable Rx and Tx flow control on switch port 21 on an Arista 7060CX:

    switch>enable switch#configure switch(config)#interface Ethernet 21/1 switch(config-if-Et31/1)#flowcontrol receive on switch(config-if-Et31/1)#flowcontrol send on
  8. Check LFC settings on the switch.

    For example, show flow control settings on ports 21-22 on an Arista 7060CX:

    Switch(config-if-Et31/1)#show interfaces ethernet 21/1-22/1 flowcontrol Port Send FlowControl Receive FlowControl RxPause TxPause admin oper admin oper ---------- -------- -------- -------- -------- ------------- ------------- Et21/1 on on off off 170373384 0 Et22/1 on on off off 289143370 0

Symmetric vs. Asymmetric LFC