Contents

Introduction

Prerequisites

Requirements

Components Used

Understand Priority Transmit B2B Credits in MDS Switch

For an Inter-Switch Link Protocol (ISL) Trunk Interface

MDS 9148S Platform Example:

MDS 9710 switch example:

Introduction

This document describes the receive and transmit buffer-to-buffer (B2B) credits in different interfaces, which are useful when you troubleshoot performance issues in a Fibre Channel fabric.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Cisco Multilayer Data Switch (MDS)

- Fibre Channel protocol

Components Used

The information in this document is based on Cisco MDS 9000 Series Multilayer Switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Understand Priority Transmit B2B Credits in MDS Switch

When you troubleshoot the slow drain issue for Fibre Channel fabric network, an easy way to confirm the available credits for interface is the show interface command in the MDS switches.

The following explains the transmit B2B credits in an MDS interface.

For an Inter-Switch Link Protocol (ISL) Trunk Interface

Low priority credits are used for data traffic while the high priority credits are for inter-switch-link control traffic. This is just a logical division on sender side to limit itself so that it does not consume all the available Receive (Rx) credits on the receiver.

For example, switch MDS Tempest (receiver) tells switch MDS Maestro (sender) that it has 64 Rx credits.

Switch MDS Maestro marks 56 credits as low priority Transmit (Tx) credits and 8 as high priority Tx credits.

In case of congestion, MDS Maestro stops sending data traffic when none of the 56 low priority credits is available.

However, it can still send control traffic to MDS Tempest using the available 8 high priority credits.

There is no need for MDS Tempest to make a differentiation between low or high priority credits.

Note: The number of high priority TX transmit credit depends on different hardware platforms.

This is the general concept: one out of eight TX transmit credit will be a high priority credit and the maximum is 15 - sufficient to handle the ELP or other ISL control messages.

This high/low priority differentiation is only for ISL links between E ports or TE ports and is not used in F ports.

MDS 9148S Platform Example:

For an E port interface:

Tempest# show int fc1/26

fc1/26 is trunking

Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)

Port WWN is 20:1a:00:2a:6a:b9:d2:30

Peer port WWN is 20:1a:00:2a:6a:56:f0:00

Admin port mode is auto, trunk mode is on

snmp link state traps are enabled

Port mode is TE

Port vsan is 2

Admin Speed is auto

Operating Speed is 4 Gbps

Rate mode is dedicated

Port flow-control is R_RDY

Transmit B2B Credit is 32

Receive B2B Credit is 64

B2B State Change Number is 14

Receive data field Size is 2112

Beacon is turned off

Logical type is core

Trunk vsans (admin allowed and active) (2)

Trunk vsans (up)                       (2)

Trunk vsans (isolated)                 ()

Trunk vsans (initializing)             ()

5 minutes input rate 160 bits/sec,20 bytes/sec, 0 frames/sec

5 minutes output rate 96 bits/sec,12 bytes/sec, 0 frames/sec

22464 frames input,1801536 bytes

0 discards,0 errors

0 invalid CRC/FCS,0 unknown class

0 too long,0 too short

22977 frames output,1318564 bytes

0 discards,0 errors

0 input OLS,0  LRR,0 NOS,2 loop inits

1 output OLS,2 LRR, 0 NOS, 1 loop inits

64 receive B2B credit remaining

32 transmit B2B credit remaining

28 low priority transmit B2B credit remaining

Interface last changed at Thu Jul 20 15:40:32 2017

Last clearing of "show interface" counters :  never

Tempest#

The orther end of the link shows total transmit credit is 64 and low priority is 56:

Maestro# show int fc1/26

fc1/26 is trunking

Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)

Port WWN is 20:1a:00:2a:6a:56:f0:00

Peer port WWN is 20:1a:00:2a:6a:b9:d2:30

Admin port mode is auto, trunk mode is on

snmp link state traps are enabled

Port mode is TE

Port vsan is 2

Speed is 4 Gbps

Rate mode is dedicated

Transmit B2B Credit is 64

Receive B2B Credit is 32

B2B State Change Number is 14

Receive data field Size is 2112

Beacon is turned off

admin fec state is down

oper fec state is down

Trunk vsans (admin allowed and active) (2)

Trunk vsans (up)                       (2)

Trunk vsans (isolated)                 ()

Trunk vsans (initializing)             ()

5 minutes input rate 160 bits/sec,20 bytes/sec, 0 frames/sec

5 minutes output rate 216 bits/sec,27 bytes/sec, 0 frames/sec

23189 frames input,1330368 bytes

0 discards,0 errors

0 invalid CRC/FCS,0 unknown class

0 too long,0 too short

22677 frames output,1818076 bytes

0 discards,0 errors

1 input OLS,2  LRR,0 NOS,3 loop inits

1 output OLS,0 LRR, 0 NOS, 1 loop inits

32 receive B2B credit remaining

64 transmit B2B credit remaining

56 low priority transmit B2B credit remaining

Interface last changed at Wed Jul 19 16:39:58 2017

Last clearing of "show interface" counters 1w 5d

Maestro#

For an F Port Interface:

In the case of an F port, this calculation does not apply since there is no control traffic between the MDS switch and the initiator/target device.

However, you can see all the transmit credits in an F port which are marked as low priority for data traffic.

For example, total transmit credit is 3 and low priority is 3 in the following output:

switch# show int fc1/47

fc1/47 is up

Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)

Port WWN is 20:2f:00:2a:6a:56:f0:00

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port mode is F, FCID is 0x570f00

Port vsan is 2

Speed is 4 Gbps

Rate mode is dedicated

Transmit B2B Credit is 3

Receive B2B Credit is 32

Receive data field Size is 2112

Beacon is turned off

admin fec state is down

oper fec state is down

5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec

5 minutes output rate 16 bits/sec,2 bytes/sec, 0 frames/sec

1220 frames input,74256 bytes

0 discards,0 errors

0 invalid CRC/FCS,0 unknown class

0 too long,0 too short

2231 frames output,220056 bytes

0 discards,0 errors

1 input OLS,1  LRR,0 NOS,0 loop inits

1 output OLS,0 LRR, 1 NOS, 0 loop inits

32 receive B2B credit remaining

3 transmit B2B credit remaining

3 low priority transmit B2B credit remaining

Interface last changed at Wed Jul 19 15:56:15 2017

Last clearing of "show interface" counters 1w 5d

switch#

MDS 9710 switch example:

There are no high priority Tx transmit credits in MDS97xx switches. However, this can be changed by the command system default tx-credit double-queue.

The maximum Tx transmit credits can also be seen from the interface of the connected switch.

Topology:  MDS 9710 Leonard fc3/14 is connected to MDS 9148S Tempest fc1/13 back to back via an E port.

Default setting:

By default, MDS 9710 DS-X9448-768K9 line cards do not have high priority credits, all 64 transmit credits are deemed low priority.

Leonard(config-if)# show int fc3/14 counter

fc3/14

5 minutes input rate 320 bits/sec, 40 bytes/sec, 3 frames/sec

5 minutes output rate 2688 bits/sec, 336 bytes/sec, 3 frames/sec

1416819 frames input, 77233324 bytes

0 discards, 0 errors, 0 CRC/FCS

0 unknown class, 0 too long, 0 too short

1416813 frames output, 138598368 bytes

0 discards, 0 errors

0 timeout discards, 0 credit loss

53 input OLS, 85 LRR, 119 NOS, 0 loop inits

77 output OLS, 7 LRR, 93 NOS, 0 loop inits

14 link failures, 37 sync losses, 2 signal losses

140 Transmit B2B credit transitions to zero

92 Receive B2B credit transitions to zero

0 2.5us TxWait due to lack of transmit credits

Percentage Tx credits not available for last 1s/1m/1h/72h: 0%/0%/0%/0%

500 receive B2B credit remaining

64 transmit B2B credit remaining

64 low priority transmit B2B credit remaining

Last clearing of "show interface" counters : never

Leonard(config-if)#

Using tx-credit double-queue:

After issuing the command on switch Leonard system default tx-credit double-queue

(Need to reset the interface by shut/no shut)

Leonard(config)# show system default switchport

System default port state is down

System default trunk mode is on

System default link failure syslog logging level is 5

System default tx credit queue type is 0

Leonard(config)# system default tx-credit double-queue

Leonard(config)# show system default switchport

System default port state is down

System default trunk mode is on

System default link failure syslog logging level is 5

System default tx credit queue type is 1

Leonard(config)# int fc3/14

Leonard(config-if)# shut

Leonard(config-if)# no shut

it can be seen that there are 8 high priority Tx transmit B2B credits and 56 low priority credits:

Leonard(config-if)# show int fc3/14 counter

fc3/14

5 minutes input rate 320 bits/sec, 40 bytes/sec, 2 frames/sec

5 minutes output rate 256 bits/sec, 32 bytes/sec, 2 frames/sec

1420613 frames input, 77459300 bytes

0 discards, 0 errors, 0 CRC/FCS

0 unknown class, 0 too long, 0 too short

1420607 frames output, 138968464 bytes

0 discards, 0 errors

0 timeout discards, 0 credit loss

54 input OLS, 87 LRR, 122 NOS, 0 loop inits

79 output OLS, 7 LRR, 95 NOS, 0 loop inits

14 link failures, 38 sync losses, 2 signal losses

143 Transmit B2B credit transitions to zero

94 Receive B2B credit transitions to zero

0 2.5us TxWait due to lack of transmit credits

Percentage Tx credits not available for last 1s/1m/1h/72h: 0%/0%/0%/0%

500 receive B2B credit remaining

64 transmit B2B credit remaining

56 low priority transmit B2B credit remaining

Last clearing of "show interface" counters : never

Leonard(config-if)#

Maximum RX transmit credit is 15 from switch MDS 9148S Tempest:

From the connected MDS 9148S,Tempest, it can be seen the maximum Tx transmit B2B credits is 15.

MDS9710 switch Leonard has 500 receive credits by default hence MDS9148S switch Tempest will also have 500 Tx transmit B2B credits remaining but 15 credits (the maximum) is allocated as high priority (500 - 485 = 15).

Tempest# show int fc1/13 counter

fc1/13

5 minutes input rate 2624 bits/sec, 328 bytes/sec, 3 frames/sec

5 minutes output rate 1504 bits/sec, 188 bytes/sec, 3 frames/sec

1343331 frames input, 131489464 bytes

0 discards, 0 errors, 0 CRC/FCS

0 unknown class, 0 too long, 0 too short

1343332 frames output, 72878124 bytes

0 discards, 0 errors

0 timeout discards, 0 credit loss

9 input OLS, 2 LRR, 0 NOS, 0 loop inits

12 output OLS, 20 LRR, 9 NOS, 0 loop inits

9 link failures, 0 sync losses, 0 signal losses

66 Transmit B2B credit transitions to zero

24 Receive B2B credit transitions to zero

0 2.5us TxWait due to lack of transmit credits

Percentage Tx credits not available for last 1s/1m/1h/72h: 0%/0%/0%/0%

64 receive B2B credit remaining

500 transmit B2B credit remaining

485 low priority transmit B2B credit remaining

Last clearing of "show interface" counters  :never

Tempest#

When the receive credits from MDS 9710 Leonard is changed to lower than 120, then one out of eight Tx credits will be marked as high priority:

Leonard(config)# int fc3/14

Leonard(config-if)# switchport fcrxbbcredit 112

Tempest# show int fc1/13 count

fc1/13

5 minutes input rate 1024 bits/sec, 128 bytes/sec, 3 frames/sec

5 minutes output rate 1536 bits/sec, 192 bytes/sec, 3 frames/sec

1360026 frames input, 133105756 bytes

0 discards, 0 errors, 0 CRC/FCS

0 unknown class, 0 too long, 0 too short

1360027 frames output, 73885304 bytes

0 discards, 0 errors

0 timeout discards, 0 credit loss

14 input OLS, 2 LRR, 0 NOS, 0 loop inits

17 output OLS, 30 LRR, 14 NOS, 0 loop inits

14 link failures, 0 sync losses, 0 signal losses

91 Transmit B2B credit transitions to zero

34 Receive B2B credit transitions to zero

0 2.5us TxWait due to lack of transmit credits

Percentage Tx credits not available for last 1s/1m/1h/72h: 0%/0%/0%/0%

64 receive B2B credit remaining

112 transmit B2B credit remaining

98 low priority transmit B2B credit remaining

Last clearing of "show interface" counters  :never

Tempest#