Spanning Tree Protocol Troubleshooting on a Nexus 5000 Series Switch

Introduction

This document describes various methods to troubleshoot common issues related to Spanning Tree Protocol (STP).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Nexus Operating System CLI
• STP

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Troubleshoot

This section covers some methods to troubleshoot common issues with STP.

STP Root

In order to troubleshoot an STP issue, it is critical to know which switch is currently the root. The command to show the STP root on a Nexus 5000 Series Switch is:

Nexus-5000# show spanning-tree vlan 1

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    32769
             Address     c84c.75fa.6000
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     c84c.75fa.6000
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Here are some other relevant commands:

Nexus-5000# show spanning-tree vlan 1 detail
Nexus-5000# show spanning-tree vlan 1 summary

Once you have determined who the current root is, you can check the event history to see if it has changed and where the Topology Change Notifications originate from.

Nexus-5000# show spanning-tree internal event-history tree 1 brief
2012:11:06 13h:44m:20s:528204us T_EV_UP
VLAN0001 [0000.0000.0000.0000 C 0 A  0 R none P none]
2012:11:06 13h:44m:21s:510394us T_UT_SBPDU
VLAN0001 [8001.547f.ee18.e441 C 0 A  0 R none P Po1]
2012:11:06 13h:44m:21s:515129us T_EV_M_FLUSH_L
VLAN0001 [1001.001b.54c2.5a42 C 6 A  5 R Po1 P none]
2012:11:06 13h:44m:23s:544632us T_EV_M_FLUSH_R
VLAN0001 [1001.001b.54c2.5a42 C 6 A  5 R Po1 P Po1]
2012:11:06 13h:44m:24s:510352us T_EV_M_FLUSH_R
VLAN0001 [1001.001b.54c2.5a42 C 6 A  5 R Po1 P Po1]

Tip: Here are some definitions for acronyms that appear in the output of the commands. SBPDU: Superior Bridge Protocol Data Unit received; FLUSH_L: Local flush; FLUSH_R: Flush from remote switch.

Note: NX-OS Versions prior to Version 5.1(3)N1(1) do not log more than 149 events, and the log does not roll.

STP Interface

This command is used in order to display the events for an interface.

Nexus-5000# show spanning-tree internal event-history tree 1 interface 
ethernet 1/3 brief
2012:11:05 13h:42m:20s:508027us P_EV_UP   Eth1/3   [S DIS  R Unkw  A  0  Inc no]
2012:11:05 13h:42m:20s:508077us P_STATE   Eth1/3   [S BLK  R Desg  A  0  Inc no]
2012:11:05 13h:42m:20s:508294us P_STATE   Eth1/3   [S LRN  R Desg  A  0  Inc no]
2012:11:05 13h:42m:20s:508326us P_STATE   Eth1/3   [S FWD  R Desg  A  0  Inc no] 

This command is used in order to investigate STP changes on an interface. This output offers many details:

Nexus-5000# show spanning-tree internal info tree 1 interface port-channel 11
------- STP Port Info (vdc 1, tree 1, port Po11) ---------
dot1d info: port_num=4106, ifi=0x1600000a (port-channel11)
ISSU FALSE non-disr, prop 0, ag 0, flush 0 peer_not_disputed_count 0
if_index               0x1600000a
namestring             port-channel11
....................        cut to save space        ............

stats                  
fwd_transition_count   1       bpdus_in      40861   bpdus_out    40861
config_bpdu_in         0       rstp_bpdu_in  40861   tcn_bpdu_in  0     
config_bpdu_out        0       rstp_bpdu_out 40861   tcn_bpdu_out 0     
bpdufilter_drop_in     0     
bpduguard_drop_in      0     
err_dropped_in         0     
sw_flood_in            0     
....................        cut to save space        ............ 

BPDU Investigation with Ethanalyzer

This section describes how to use Ethanalyzer in order to capture BPDUs:

Ethanalyzer local interface inbound-hi display-filter "vlan.id == 1 && stp"

Example:
Nexus-5000# ethanalyzer local interface inbound-hi display-filter "vlan.id
== 1 && stp"
Capturing on eth4
2013-05-11 13:55:39.280951 00:05:73:f5:d6:27 -> 01:00:0c:cc:cc:cd STP RST. 
Root = 33768/00:05:73:ce:a9:7c  Cost = 1  Port = 0x900a
2013-05-11 13:55:40.372434 00:05:73:ce:a9:46 -> 01:00:0c:cc:cc:cd STP RST. 
Root = 33768/00:05:73:ce:a9:7c  Cost = 0  Port = 0x900a
2013-05-11 13:55:41.359803 00:05:73:f5:d6:27 -> 01:00:0c:cc:cc:cd STP RST. 
Root = 33768/00:05:73:ce:a9:7c  Cost = 1  Port = 0x900a
2013-05-11 13:55:42.372405 00:05:73:ce:a9:46 -> 01:00:0c:cc:cc:cd STP RST. 
Root = 33768/00:05:73:ce:a9:7c  Cost = 0  Port = 0x900a

In order to see detailed packets, use the detail command:

Nexus-5000# ethanalyzer local interface inbound-hi detail display-filter
"vlan.id == 1 && stp" 
Capturing on eth4
Frame 7 (68 bytes on wire, 68 bytes captured)
    Arrival Time: May 11, 2013 13:57:02.382227000
    [Time delta from previous captured frame: 0.000084000 seconds]
    [Time delta from previous displayed frame: 1368280622.382227000 seconds]
    [Time since reference or first frame: 1368280622.382227000 seconds]
    Frame Number: 7
    Frame Length: 68 bytes
    Capture Length: 68 bytes
    [Frame is marked: False]
    [Protocols in frame: eth:vlan:llc:stp]
Ethernet II, Src: 00:05:73:ce:a9:46 (00:05:73:ce:a9:46), Dst: 01:00:0c:cc:cc:cd 
(01:00:0c:cc:cc:cd)
    Destination: 01:00:0c:cc:cc:cd (01:00:0c:cc:cc:cd)
        Address: 01:00:0c:cc:cc:cd (01:00:0c:cc:cc:cd)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Source: 00:05:73:ce:a9:46 (00:05:73:ce:a9:46)
        Address: 00:05:73:ce:a9:46 (00:05:73:ce:a9:46)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
    Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN
    111. .... .... .... = Priority: 7
    ...0 .... .... .... = CFI: 0
    .... 0000 0000 0001 = ID: 1
    Length: 50
Logical-Link Control
    DSAP: SNAP (0xaa)
    IG Bit: Individual
    SSAP: SNAP (0xaa)
    CR Bit: Command
    Control field: U, func=UI (0x03)
        000. 00.. = Command: Unnumbered Information (0x00)
        .... ..11 = Frame type: Unnumbered frame (0x03)
    Organization Code: Cisco (0x00000c)
    PID: PVSTP+ (0x010b)
Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x3c (Forwarding, Learning, Port Role: Designated)
        0... .... = Topology Change Acknowledgment: No
        .0.. .... = Agreement: No
        ..1. .... = Forwarding: Yes
        ...1 .... = Learning: Yes
        .... 11.. = Port Role: Designated (3)
        .... ..0. = Proposal: No
        .... ...0 = Topology Change: No
    Root Identifier: 33768 / 00:05:73:ce:a9:7c
    Root Path Cost: 0
    Bridge Identifier: 33768 / 00:05:73:ce:a9:7c
    Port identifier: 0x900a
    Message Age: 0
    Max Age: 20
    Hello Time: 2
    Forward Delay: 15
    Version 1 Length: 0

In order to write this information to a PCAP file, use this command:

Nexus-5000# ethanalyzer local interface inbound-hi display-filter 
"vlan.id == 1 && stp" write bootflash:bpdu.pcap
Capturing on eth4
3  << Lists how many packets were captured.

On BPDU captures, the source MAC address is the interface MAC address of the far end device.

On the Ethanalyzer capture, the port appears in a hexadecimal format. In order to identify the port number, you need to first convert the number into hexadecimal:

0x900a (from the previous trace) = 36874

This is the command that decodes that number to a port:

Nexus-5000# show spanning-tree internal info all  |
grep -b 50 "port_id                36874" | grep "Port Info"
------- STP Port Info (vdc 1, tree 1, port Po11) ----------
------- STP Port Info (vdc 1, tree 300, port Po11) ---------
------- STP Port Info (vdc 1, tree 800, port Po11) ---------
------- STP Port Info (vdc 1, tree 801, port Po11) ---------
------- STP Port Info (vdc 1, tree 802, port Po11) ---------
------- STP Port Info (vdc 1, tree 803, port Po11) ---------
------- STP Port Info (vdc 1, tree 999, port Po11) ---------

In this case, it is port-channel 11.

STP Convergence

If you need to investigate STP convergence, use the show spanning-tree internal interactions command. This command provides insight into what events triggered the STP changes. It is important to gather this information as soon as the issue occurs, because the logs are large, and they wrap over time.

Nexus-5000#show spanning-tree internal interactions
- Event:(null), length:123, at 81332 usecs after Sat May 11 12:01:47 2013
Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1
VDC 1, state FWD, rr_token 0x21b9c3 msg_size 584
- Event:(null), length:140, at 81209 usecs after Sat May 11 12:01:47 2013
vb_vlan_shim_set_vlans_multi_port_state(2733): Req (type=12) to PIXM 
vdc 1, inst 0, num ports 1, state FWD 
[Po17 v 800-803,999-1000]
- Event:(null), length:123, at 779644 usecs after Sat May 11 12:01:46 2013
Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 
VDC 1, state FWD, rr_token 0x21b99a msg_size 544<
- Event:(null), length:127, at 779511 usecs after Sat May 11 12:01:46 2013
vb_vlan_shim_set_vlans_multi_port_state(2733): Req (type=12) to PIXM 
vdc 1, inst 0, num ports 1, state FWD 
[Po17 v 300]
- Event:(null), length:123, at 159142 usecs after Sat May 11 12:01:32 2013
Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 
VDC 1, state LRN, rr_token 0x21b832 msg_size 584
- Event:(null), length:140, at 159023 usecs after Sat May 11 12:01:32 2013
vb_vlan_shim_set_vlans_multi_port_state(2733): Req (type=12) to PIXM 
vdc 1, inst 0, num ports 1, state LRN 
[Po17 v 800-803,999-1000]
- Event:(null), length:123, at 858895 usecs after Sat May 11 12:01:31 2013
Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 
VDC 1, state LRN, rr_token 0x21b80b msg_size 544
- Event:(null), length:127, at 858772 usecs after Sat May 11 12:01:31 2013
vb_vlan_shim_set_vlans_multi_port_state(2733): Req (type=12) to PIXM 
vdc 1, inst 0, num ports 1, state LRN 
[Po17 v 300]
.................... cut to save space ............

External VLAN Mapping

Nexus 5000 Series switches use internal VLANs in order to map to external VLAN numbers for forwarding. Sometimes the VLAN ID is the internal VLAN ID. In order to get the mapping to an external VLAN, enter:

Nexus-5000# show platform afm info global
 Gatos Hardware version 0
 Hardware instance mapping
-------------------------
Hardware instance:  0  asic id: 0  slot num: 0
-------------- cut to save space  --------------------------
Hardware instance: 12  asic id: 1  slot num: 3
AFM Internal Status
-------------------
[unknown label                                     ]: 324
[no free statistics counter                        ]: 2
[number of verify                                  ]: 70
[number of commit                                  ]: 70
[number of request                                 ]: 785
[tcam stats full                                   ]: 2

Vlan mapping table
------------------
Ext-vlan: 1     -  Int-vlan: 65 

STP Debugs

Another way to troubleshoot STP issues is to use debugs. However, the use of STP debugs might cause the CPU usage to spike, which causes concerns in some enviroments. In order to drastically reduce CPU usage while running debugs, use a debug-filter, and log activity to a log file.

1. Create the log file, which is saved under the directory log.
   

   Nexus-5000#debug logfile spanning-tree.txt
   Nexus-5548P-L3# dir log:
   31 Nov 06 12:46:35 2012 dmesg
   ----- cut to save space----
   7626 Nov 08 22:41:58 2012 messages
   0 Nov 08 23:05:40 2012 spanning-tree.txt
   4194304 Nov 08 22:39:05 2012 startupdebug

2. Run the debug.
   

   Nexus-5000# debug spanning-tree bpdu_rx interface e1/30 
   <<< setup your spanning-tree for bpdus
   Nexus-5000# copy log:spanning-tree.txt bootflash:
   

   
   Example from the log file:
   

   2012 Nov 8 23:08:24.238953 stp: BPDU RX: vb 1 vlan 300, ifi 0x1a01d000
   (Ethernet1/30)
   2012 Nov 8 23:08:24.239095 stp: BPDU Rx: Received BPDU on vb 1 vlan 300 
   port Ethernet1/30 pkt_len 60 bpdu_len 36 netstack flags 0x00ed enc_type ieee
   2012 Nov 8 23:08:35.968453 stp: RSTP(300): Ethernet1/30 superior msg
   2012 Nov 8 23:08:35.968466 stp: RSTP(300): Ethernet1/30 rcvd info remaining 6
   2012 Nov 8 23:08:36.928415 stp: BPDU RX: vb 1 vlan 300, ifi 0x1a01d000 
   (Ethernet1/30)
   2012 Nov 8 23:08:36.928437 stp: BPDU Rx: Received BPDU on vb 1 vlan 300 
   port Ethernet1/30 pkt_len 60 bpdu_len 36 netstack flags 0x00ed enc_type ieee
   2012 Nov 8 23:08:36.928476 stp: RSTP(300): msg on Ethernet1/30
   ....................        cut to save space        ............

Nexus 5000 Did Not Process BPDUs

In order to troubleshoot this problem, check the event history to determine if the Nexus 5000 Series switch assumed root. The Nexus 5000 assumes root if it either did not process BPDUs or it did not receive them. In order to investigate which is the cause, you should determine if there are other switches attached to the designated bridge that had this problem as well. If no other bridges had the problem, it is most likely that the Nexus 5000 did not process the BPDUs. If other bridges did have the problem, it is most likely that the bridge did not send the BPDUs.

Note: Things to keep in mind when troubleshooting STP and virtual Port Channel (vPC). Only the vPC Primary sends BPDUs. When the vPC secondary is the STP root, the Primary still sends the BPDUs. If the root is connected via a vPC, only the Primary increments Rx BPDU counters, even when the Secondary receives them.