Example 4 - PCP PFC with Multiple TCs (1 for RDMA, 1 for LAN) – No VLANs

This example describes how to run both RDMA and LAN traffic on the same link using the parent interface (no explicit VLANs, although VLAN 0 are used transparently).

These steps can be used in a back-to-back configuration, or if you are using a switch, be sure to configure the neighboring switch ports for the same configuration (consult the appropriate switch manual for more detail).

Settings in this example:

• Non-willing mode — In this example, adapter settings are configured explicitly using lldptool (vs. configuring DCB on a switch and using willing mode on adapters).
• Software DCB — Required to use non-willing mode.
• Two traffic classes:
  • One loss-less TC for RDMA, with 80% bandwidth allocated.
  • One lossy TC for LAN, with 20% bandwidth allocated.
• PFC enabled for only the RDMA traffic class (this makes it loss-less).

Perform the following steps on both servers:

1. Disable LFC (LFC and PFC cannot co-exist). # ethtool -A <interface> rx off tx off
2. Verify that LFC is disabled. # ethtool -a <interface> Pause parameters for <interface>: Autonegotiate: on RX: off TX: off RX negotiated: off TX negotiated: off
3. Configure the adapter for software DCB mode by disabling firmware DCB mode. # ethtool --set-priv-flags <interface> fw-lldp-agent off
4. Verify that firmware DCB is disabled: # ethtool --show-priv-flags <interface> | grep fw-lldp-agent fw-lldp-agent : off
5. Install OpenLLDP (the software that controls PFC and other DCB settings), if not already installed:
   • RHEL: # yum install lldpad
   • SLES or Ubuntu: zypper or apt-get might work (untested)
   • All operating systems:

     Download and build from source from https://github.com/intel/openlldp.

6. Start the LLDP daemon. # lldpad -d
7. Verify LLDP is active by showing current LLDP settings on the interface.

   The following example shows the OpenLLDP default:

   # lldptool -ti <interface> Chassis ID TLV MAC: 68:05:ca:a3:89:78 Port ID TLV MAC: 68:05:ca:a3:89:78 Time to Live TLV 120 IEEE 8021QAZ ETS Configuration TLV Willing: yes CBS: not supported MAX_TCS: 8 PRIO_MAP: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 TC Bandwidth: 0% 0% 0% 0% 0% 0% 0% 0% TSA_MAP: 0:strict 1:strict 2:strict 3:strict 4:strict 5:strict 6:strict 7:strict IEEE 8021QAZ PFC TLV Willing: yes MACsec Bypass Capable: no PFC capable traffic classes: 8 PFC enabled: none End of LLDPDU TLV
8. Plan your DCB configuration.

   Traffic Stream	Loss-less	TC	Priority	ToS	Bandwidth
   RDMA Application	Yes	0	0	0	80%
   LAN Application	No	1	4	24	20%
   Unused	No	Any8.a	All Others	N/A	0%
   Notes: Unused priorities can be mapped to any TC (no traffic is being steered to specific priorities). Leaving them mapped to TC 0 is acceptable.

9. Configure ETS.
   • Map priorities to traffic classes.
   • Allocate bandwidths.
   Note:The following is a single long command line: # lldptool -Ti <interface> -V ETS-CFG willing=no \ up2tc=0:0,1:0,2:0,3:0,4:1,5:0,6:0,7:0 \ tsa=0:ets,1:ets,2:strict,3:strict,4:strict,5:strict,6:strict,7:strict \ tcbw=80,20,0,0,0,0,0,0

   Output:

   willing = no up2tc = 0:0,1:0,2:0,3:0,4:1,5:0,6:0,7:0 TSA = 0:ets 1:ets 2:strict 3:strict 4:strict 5:strict 6:strict 7:strict tcbw = 80% 20% 0% 0% 0% 0% 0% 0%
10. Enable PFC on priority 0 (non-willing). # lldptool -Ti <interface> -V PFC willing=no enabled=0

    Output:

    willing = no prio = 0
11. Verify new settings. # lldptool -ti <interface> Chassis ID TLV MAC: 68:05:ca:a3:89:78 Port ID TLV MAC: 68:05:ca:a3:89:78 Time to Live TLV 120 IEEE 8021QAZ ETS Configuration TLV Willing: no CBS: not supported MAX_TCS: 8 PRIO_MAP: 0:0 1:0 2:0 3:0 4:1 5:0 6:0 7:0 TC Bandwidth: 80% 20% 0% 0% 0% 0% 0% 0% TSA_MAP: 0:ets 1:ets 2:strict 3:strict 4:strict 5:strict 6:strict 7:strict IEEE 8021QAZ PFC TLV Willing: no MACsec Bypass Capable: no PFC capable traffic classes: 8 PFC enabled: 0 ← This means PFC is enabled on prio 0 (not that PFC is disabled) End of LLDPDU TLV
12. Repeat on neighbor node:
    • If using a back-to-back configuration, either repeat the configuration on the other host or enable willing mode on that host.
    • If using a switch, configure the same DCB scheme on the switch port. Consult the appropriate switch manual for details.
13. Run the application.
    • RDMA:

      Run the RDMA application normally. Since RDMA is running on the default TC0 with prio 0 in this example, you do not need any command line options to set ToS. ToS (and therefore priority) is 0 by default.

    • LAN:

      Run the LAN application with a command line option to set ToS=24. In Linux, this maps to prio=4.