×
Search
Intel® Ethernet 800 Series Linux Flow Control
Intel® Ethernet 800 Series Linux Flow Control
Configuration Guide for RDMA Use Cases
ID
635330
Date
07/13/2023
Version
1.3
View More
See Less
Close Filter Modal
Document Table of Contents
Document Table of Contents
Notices and Disclaimers
Revision History
Introduction
Background
Link-Level Flow Control
Priority Flow Control - Fundamentals
Congestion Management – Tuning Parameters
Priority Flow Control - Planning and Guidelines
Priority Flow Control - Verification
Troubleshooting
Notices and Disclaimers
Revision History
Introduction
Introduction
QoS/Flow Control Limitations on the 800 Series
QoS/Flow Control Limitations on the 800 Series
Background
Ethernet Flow Control
Flow Control in RDMA Networks
Types of Flow Control: LFC vs. PFC
Ethernet Flow Control
Flow Control in RDMA Networks
Types of Flow Control: LFC vs. PFC
Link-Level Flow Control
LFC Setup Instructions
Symmetric vs. Asymmetric LFC
LFC Setup Instructions
Symmetric vs. Asymmetric LFC
Priority Flow Control - Fundamentals
Priority Flow Control - Fundamentals
DCB Standards
Determining PFC Priority Mode: PCP vs. DSCP
Assigning an Application to a Traffic Class
DCB Standards
DCB Standards
DCB Willing vs. Non-willing Modes
DCB Willing vs. Non-willing Modes
Determining PFC Priority Mode: PCP vs. DSCP
Assigning an Application to a Traffic Class
Assigning an Application to a Traffic Class
Mapping Steps Details
Mapping Steps Details
Step 1: Determine Flow Control Design Needed
Step 2: Kernel Priority (sk_prio) or DSCP to UP Mapping
Step 3: UP to TC
Step 4: Set ToS in the Application (or for All RoCEv2 Traffic)
Step 1: Determine Flow Control Design Needed
Step 2: Kernel Priority (sk_prio) or DSCP to UP Mapping
Step 3: UP to TC
Step 4: Set ToS in the Application (or for All RoCEv2 Traffic)
Congestion Management – Tuning Parameters
Overview
DCQCN
Congestion Control Parameter Settings
Overview
DCQCN
DCQCN
DCQCN Specification
DCQCN Specification
Congestion Control Parameter Settings
Priority Flow Control - Planning and Guidelines
Priority Flow Control - Planning and Guidelines
Steps
Example Configurations
Steps
Steps
Network Host and Switch Setup
Willing vs. Non-willing DCB Mode
Firmware vs. Software DCB
Software DCB Willing Mode
Separating and Prioritizing Traffic Streams
Configuring ETS: Map Priorities to TCs/Allocate Bandwidth
Configuring PFC
Run Applications with the Right Priority
Network Host and Switch Setup
Willing vs. Non-willing DCB Mode
Firmware vs. Software DCB
Software DCB Willing Mode
Separating and Prioritizing Traffic Streams
Configuring ETS: Map Priorities to TCs/Allocate Bandwidth
Configuring PFC
Configuring PFC
Option A: PCP PFC - Set Priorities for Drop or No-drop
Option B: DSCP PFC
Option A: PCP PFC - Set Priorities for Drop or No-drop
Option B: DSCP PFC
Run Applications with the Right Priority
Example Configurations
Example 1 - 800 Series-800 Series Back-to-Back – PCP PFC with Single TC
Example 2 - Adapters Connected Through a Switch – Willing Mode on Adapters, DCB Configured on Switch
Example 3 - DSCP PFC with Non-Default TCs - Non-Willing Mode on Adapters, ECN Configured
Example 4 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – No VLANs
Example 5 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – with VLANs
Example 1 - 800 Series-800 Series Back-to-Back – PCP PFC with Single TC
Example 2 - Adapters Connected Through a Switch – Willing Mode on Adapters, DCB Configured on Switch
Example 3 - DSCP PFC with Non-Default TCs - Non-Willing Mode on Adapters, ECN Configured
Example 4 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – No VLANs
Example 5 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – with VLANs
Priority Flow Control - Verification
Priority Counters
Discard Counters
tpcdump
Priority Counters
Discard Counters
Discard Counters
LAN Packet Drops
RDMA Discards
LAN Packet Drops
RDMA Discards
tpcdump
Troubleshooting
Close Filter Modal
Notices and Disclaimers
Revision History
Introduction
QoS/Flow Control Limitations on the 800 Series
Background
Ethernet Flow Control
Flow Control in RDMA Networks
Types of Flow Control: LFC vs. PFC
Link-Level Flow Control
LFC Setup Instructions
Symmetric vs. Asymmetric LFC
Priority Flow Control - Fundamentals
DCB Standards
DCB Willing vs. Non-willing Modes
Determining PFC Priority Mode: PCP vs. DSCP
Assigning an Application to a Traffic Class
Mapping Steps Details
Step 1: Determine Flow Control Design Needed
Step 2: Kernel Priority (sk_prio) or DSCP to UP Mapping
Step 3: UP to TC
Step 4: Set ToS in the Application (or for All RoCEv2 Traffic)
Congestion Management – Tuning Parameters
Overview
DCQCN
DCQCN Specification
Congestion Control Parameter Settings
Priority Flow Control - Planning and Guidelines
Steps
Network Host and Switch Setup
Willing vs. Non-willing DCB Mode
Firmware vs. Software DCB
Software DCB Willing Mode
Separating and Prioritizing Traffic Streams
Configuring ETS: Map Priorities to TCs/Allocate Bandwidth
Configuring PFC
Option A: PCP PFC - Set Priorities for Drop or No-drop
Option B: DSCP PFC
Run Applications with the Right Priority
Example Configurations
Example 1 - 800 Series-800 Series Back-to-Back – PCP PFC with Single TC
Example 2 - Adapters Connected Through a Switch – Willing Mode on Adapters, DCB Configured on Switch
Example 3 - DSCP PFC with Non-Default TCs - Non-Willing Mode on Adapters, ECN Configured
Example 4 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – No VLANs
Example 5 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – with VLANs
Priority Flow Control - Verification
Priority Counters
Discard Counters
LAN Packet Drops
RDMA Discards
tpcdump
Troubleshooting
LAN Packet Drops
# ethtool -S enp175s0f0 | grep drop rx_dropped: 0 tx_dropped_link_down.nic: 0 rx_dropped.nic: 0
RDMA Discards
Discard Counters
RDMA Discards