Example 3 - DSCP PFC with Non-Default TCs - Non-Willing Mode on Adapters, ECN Configured

This benchmarking setup using DSCP based PFC to configure two priorities: one for data and the other for ECN packets

Settings in this example:

• Non-willing mode
• Software DCB — Required to use non-willing mode.
• Priority 3 with DSCP set to 24 mapped to TC3 for data
• Priority 6 with DSCP set to 48 mapped to TC6 for ECN

Perform the following steps on both servers:

1. Disable LFC (LFC and PFC cannot co-exist). # ethtool -A <interface> rx off tx off
2. Verify that LFC is disabled. # ethtool -a <interface> Pause parameters for <interface>: Autonegotiate: on RX: off TX: off RX negotiated: off TX negotiated: off
3. Configure the adapter for software DCB mode by disabling firmware DCB mode. # ethtool --set-priv-flags <interface> fw-lldp-agent off
4. Verify that firmware DCB is disabled. # ethtool --show-priv-flags <interface> | grep fw-lldp-agent fw-lldp-agent : off
5. Install OpenLLDP (the software that controls PFC and other DCB settings), if not already installed.
   • RHEL: # yum install lldpad
   • SLES or Ubuntu: zypper or apt-get might work (untested)
   • All operating systems:

     Download and build from source from https://github.com/intel/openlldp.

6. Start the LLDP daemon. # lldpad -d
7. Verify LLDP is active by showing current LLDP settings on the interface.

   These are the default llpad settings (some outputs might be different).

   # lldptool -ti <interface> Chassis ID TLV MAC: 68:05:ca:a3:89:78 Port ID TLV MAC: 68:05:ca:a3:89:78 Time to Live TLV 120 IEEE 8021QAZ ETS Configuration TLV Willing: yes CBS: not supported MAX_TCS: 8 PRIO_MAP: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 TC Bandwidth: 0% 0% 0% 0% 0% 0% 0% 0% TSA_MAP: 0:strict 1:strict 2:strict 3:strict 4:strict 5:strict 6:strict 7:strict IEEE 8021QAZ PFC TLV Willing: yes MACsec Bypass Capable: no PFC capable traffic classes: 8 PFC enabled: none End of LLDPDU TLV
8. Plan your DSCP configuration.

   Traffic Stream	Lossless	DSCP	Prio	ToS	TC=Prio	Bandwidth
   RDMA Application	Yes	24	3	96	3	80%
   ECN Congestion Notification	Yes	48	6	192	6	--
   General Traffic	Yes		Any		0	20%

   Unused priorities can be mapped to any TC (no traffic is being steered to specific priorities). Leaving them mapped to TC 0 is acceptable.

9. Enable DSCP of 24 (Priority 3) on TC3 in non-willing mode. # lldptool -Ti <interface> -V PFC willing=no enabled=3

   Output:

   willing = no prio = 3 # lldptool -Ti <interface> -V APP app=3,5,24

   Output:

   0:(3,5,24) local hw (set) Note:5 is a fixed value indicating that DSCP is set to TC mapping.
10. Add DSCP mapping for ECN 48 on TC6 in non-willing mode. # lldptool -Ti <interface> -V APP app=6,5,48

    Output:

    0:(3,5,48) local hw (set) 1:(6,5,24) local hw (set)
11. Enable ETS to map DSCP priority/TC3 and the ECN to priority/TC6, and to allocate 80% bandwidth to TC3 and 20% bandwidth to TC6. Note:The following example is one command line. # lldptool -Ti <interface> -V ETS-CFG willing=no \ tsa=0:ets,1:ets,2:ets,3:strict,4:ets,5:ets,6:strict,7:ets tcbw=0,0,0,80,0,0,20,0

    Output:

    willing = no up2tc = 0:0,1:0,2:0,3:3,4:0,5:0,6:6,7:0 TSA = 0:ets 1:ets 2:ets 3:strict 4:ets 5:ets 6:strict 7:ets tcbw = 0% 0% 0% 80% 0% 0% 20% 0%
12. Verify the new settings. # lldptool -ti <interface> Chassis ID TLV MAC: 68:05:ca:a3:89:78 Port ID TLV MAC: 68:05:ca:a3:89:78 Time to Live TLV 120 IEEE 8021QAZ ETS Configuration TLV Willing: no CBS: not supported MAX_TCS: 8 PRIO_MAP: 0:0 1:0 2:0 3:3 4:0 5:0 6:6 7:0 TC Bandwidth: 0% 0% 0% 80% 0% 0% 20% 0% TSA_MAP: 0:ets,1:ets,2:strict,3:strict,4:strict,5:strict,6:strict,7:strict IEEE 8021QAZ PFC TLV Willing: no MACsec Bypass Capable: no PFC capable traffic classes: 8 PFC enabled: 3 End of LLDPDU TLV
13. Set the default_​roce_​tos.

    In this example, DSCP is set to 24 (0x18) and priority 3 so default_​roce_​tos is 96 (4 x DSCP value).

    # mkdir /sys/kernel/config/rdma_cm/<interface> # echo 96 > /sys/kernel/config/rdma_cm/<interface>/ports/1/default_roce_tos # echo 0x706050418020100 > up_up_map
14. Configure ECN and add DSCP mapping for ECN.

    In this example, ECN has a DSCP value of 48.

    # echo 1 > roce_dcqcn_enable # echo 48 > cnp_up_override # echo 1 > up_map_enable
15. Repeat on other node.
16. Run the application on the configured ports.