The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes how to troubleshoot the MAC Flaps/Loop on Cisco Catalyst Switches.
Cisco recommends that you have a fundamental knowledge of basic Switching concepts and an understanding of Spanning Tree Protocol (STP) and its features on Cisco Catalyst Switches.
The information in this document is based on Cisco Catalyst Switches with all versions (this document is not restricted to any specific software or hardware versions).
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
This document serves as a guide that lays out a systematic approach to troubleshooting MAC flaps or loop issues on the Cisco Catalyst switches. MAC flaps/loops are disruptions in a network caused by inconsistencies in the MAC address tables of switches. This document not only provides steps to identify and resolve these issues but also includes practical examples for better understanding.
A MAC flap occurs when a switch receives a frame with the same MAC source address but from a different interface than the one it initially learned it from. This causes the switch to flap between ports, updating its MAC address table with the new interface. This situation can cause instability in the network and lead to performance issues.
In a Cisco switch, MAC flapping is typically logged as a message similar to this:
"%SW_MATM-4-MACFLAP_NOTIF: Host xxxx.xxxx.xxxx in vlan x is flapping between port (1) and port (2)"
In this example, the MAC address xxxx.xxxx.xxxx
was first learned on interface port (1), then seen on interface port (2), causing a MAC flap.
The most common cause of MAC flapping is a Layer 2 loop in the network, often due to a misconfiguration of STP or issues with redundant links. Other causes can include faulty hardware, software bugs, or even security issues such as MAC spoofing.
Troubleshooting MAC flaps often involves identifying and resolving any loops in the network, checking device configurations, or updating device firmware/software.
%SW_MATM-4-MACFLAP_NOTIF: Host [mac_address] in vlan [vlan_id] is flapping between port [port_id_1] and port [port_id_2]
show interfaces
or show mac address-table
in order to see which devices are connected to the interfaces and where the MAC address is being learned.show mac address-table
command on each Layer 2 switch in the path.show spanning-tree
in order to check the STP configuration. Also, check for Topology Change Notifications (TCNs) using the command show spanning-tree detail | include ieee|occur|from|is
.show mac address-table
command in order to look for duplicate MAC addresses on your network.Bug Search Tool: https://bst.cloudapps.cisco.com/bugsearch
Bug Search Tool Help: https://www.cisco.com/c/en/us/support/web/tools/bst/bsthelp/index.html#search
The eWLC controller is experiencing a loss of connectivity to the gateway, and packet drops are preventing APs from joining the controller.
MAC flapping was identified on the switch (Switch1) that is connected to the eWLC.
*Aug 5 05:52:50.750: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.5e00.0101 in vlan 4 is flapping between port Gi1/0/11 and port Te1/1/2
*Aug 5 05:53:03.327: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.5e00.0101 in vlan 4 is flapping between port Gi1/0/11 and port Te1/1/2
*Aug 5 05:53:21.466: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.5e00.0101 in vlan 4 is flapping between port Gi1/0/11 and port Te1/1/2
MAC Learning:
Enter the command show mac address-table address <mac>
in order to check the MAC address learned on the port.
Switch1#show mac address-table address 0000.5e00.0101
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
4 0000.5e00.0101 DYNAMIC Gi1/0/11
4 0000.5e00.0101 DYNAMIC Te1/1/2
Configuration of Ports Gi1/0/11 and Te1/1/2:
Enter the command show running-config interface <interface-number>
in order to check the interface configuration.
interface GigabitEthernet1/0/11
switchport trunk native vlan 4
switchport mode trunk
end
interface TenGigabitEthernet1/1/2
switchport mode trunk
end
CDP Neighbors of Ports Gi1/0/11 and Te1/1/2:
Enter the command show cdp neighbors <interface-number>
in order to check the details of connected devices.
Switch1#show cdp neighbors gi1/0/11
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
eWLC Gig 1/0/11 130 R T C9115AXI- Gig 0 < ----- eWLC Controller
Switch1#show cdp neighbors gi1/1/2
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
Switch2
Ten 1/1/2 163 R S I C9500-16X Ten 1/0/3 < ----- Uplink Switch
MAC Learning on Switch2 (Uplink Switch):
Enter the command show mac address-table address <mac>
in order to check the MAC address learned on the port.
Switch2#show mac address-table address 0000.5E00.0101
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
4 0000.5e00.0101 STATIC Vl4 < ----- VRRP MAC of Vlan4
4 0000.5e00.0101 DYNAMIC Te1/0/13 < ----- Learning from Switch1 (eWLC connected Switch)
Switch2#show vrrp vlan 4
Vlan4 - Group 1 - Address-Family IPv4
State is MASTER
State duration 5 days 4 hours 22 mins
Virtual IP address is x.x.x.x
Virtual MAC address is 0000.5E00.0101 < ------ VRRP MAC of Vlan4
Advertisement interval is 1000 msec
It was verified that the Virtual Router Redundancy Protocol (VRRP) ID of Switch 2 and the eWLC were the same, which resulted in the generation of the same virtual MAC by the VRRP.
The issue was resolved after changing the VRRP instance on the WLC, which was causing a duplicate MAC on the switch leading to a loss of connectivity to the gateway and packet drops, which prevented the APs from joining the controller.
Some of the servers are either inaccessible or experiencing significant latency/drops.
1. Noticed MAC flapping occurring on the Core switch.
Nov 14 08:36:34.637: %SW_MATM-4-MACFLAP_NOTIF: Host xxxx.xxxx.xxxx in vlan 1 is flapping between port Twe1/0/19 and port Twe1/0/5
Nov 14 08:36:34.838: %SW_MATM-4-MACFLAP_NOTIF: Host yyyy.yyyy.yyyy in vlan 1 is flapping between port Twe1/0/17 and port Twe1/0/19
Nov 14 08:36:34.882: %SW_MATM-4-MACFLAP_NOTIF: Host zzzz.zzzz.zzzz in vlan 1 is flapping between port Po1 and port Twe1/0/19
2. Chosen the MAC address yyyy.yyyy.yyyy
for the troubleshooting process.
MAC Learning:
Enter the command show mac address-table address <mac>
in order to check the MAC address learned on the port.
Core-SW#show mac address-table address yyyy.yyyy.yyyy
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
1 yyyy.yyyy.yyyy DYNAMIC Twe1/0/17
CDP Neighbors of Ports Twe 1/0/17 and Twe 1/0/17:
Enter the command show cdp neighbors <interface-number>
in order to check the details of connected devices.
Core-SW#show cdp neighbors Twe 1/0/17
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
2960X-SW1
Twe 1/0/17 162 S I WS-C2960X Gig 1/0/51
Core-SW#show cdp neighbors Twe 1/0/19
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
2960S-SW1
Twe 1/0/19 120 S I WS-C2960S Gig 2/0/28
Logs from 2960X-SW1 Connected to Core-SW Twe1/0/17:
MAC yyyy.yyyy.yyyy
is flapping between port Gi1/0/51 and Gi2/0/46 (9200L).
2960X-SW1#show mac address-table address yyyy.yyyy.yyyy
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
1 yyyy.yyyy.yyyy DYNAMIC Gi1/0/51
2960X-SW1#show mac address-table address yyyy.yyyy.yyyy
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
1 yyyy.yyyy.yyyy DYNAMIC Gi2/0/46
2960X-SW1#show run interface gi 1/0/51
Building configuration...
Current configuration : 62 bytes
!
interface GigabitEthernet1/0/51
switchport mode trunk
end
2960X-SW1#show run interface gi 2/0/46
Building configuration...
Current configuration : 62 bytes
!
interface GigabitEthernet2/0/46
switchport mode trunk
end
Logs from 9200L:
(This appears to be the valid port for this MAC address.)
9200L#show mac address-table address yyyy.yyyy.yyyy
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
1 yyyy.yyyy.yyyy DYNAMIC Gi1/0/1
9200L#show run interface gi 1/0/1
Building configuration...
Current configuration : 62 bytes
!
interface GigabitEthernet1/0/1
switchport mode access
end
2960S-SW1 Connected to Core-SW Twe1/0/19:
(Appears to be a loop path.) The port on the Core-SW was shut down in order to mitigate the loop.
However, MAC flaps were still being observed on the Core-SW.
Logs from 2960S-SW1:
Nov 14 08:36:34.637: %SW_MATM-4-MACFLAP_NOTIF: Host xxxx.xxxx.xxxx in vlan 1 is flapping between port Gi3/0/27 and port Gi2/0/26
Nov 14 08:36:34.838: %SW_MATM-4-MACFLAP_NOTIF: Host yyyy.yyyy.yyyy in vlan 1 is flapping between port Gi3/0/27 and port Gi2/0/26
Nov 14 08:36:34.882: %SW_MATM-4-MACFLAP_NOTIF: Host zzzz.zzzz.zzzz in vlan 1 is flapping between port Gi3/0/27 and port Gi2/0/26
2960S-SW1#show run interface gi 3/0/27
Building configuration...
Current configuration : 62 bytes
!
interface GigabitEthernet3/0/27
switchport mode trunk
end
2960S-SW1#show cdp neighbor gi 3/0/27
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
2960X-SW2
Gig 3/0/27 176 S I WS-C2960X Gig 1/0/27
Logs from 2960X-SW2:
2960X-SW2#show run interface gi 3/0/15
Building configuration...
Current configuration : 39 bytes
!
interface GigabitEthernet3/0/15
end
2960X-SW2#show cdp neighbor gi 3/0/15
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
SG300 Gig 3/0/15 157 S I SG300-28P gi18
2960X-SW2#config terminal
2960X-SW2(config)#interface gi 3/0/15
2960X-SW2(config-if)#shutdown
MAC flaps were seen due to the SG300 (unmanaged) switch connected to the network.
The MAC flapping issue was resolved by shutting down the port connected to the unmanaged switch SG300.
STP Portfast:
STP PortFast causes a Layer 2 LAN port to enter the forwarding state immediately, bypassing the listening and learning states. STP PortFast prevents the generation of STP TCNs, which are not meaningful from ports that do not receive STP Bridge Protocol Data Units (BPDUs). Configure STP PortFast only on ports that are connected to end host devices that terminate VLANs and from which the port must never receive STP BPDUs, such as Workstations, Servers, Ports on routers that are not configured to support bridging.
BPDU Guard:
STP BPDU Guard complements the functionality of STP PortFast. On STP PortFast-enabled ports, STP BPDU Guard protects Layer 2 loops that STP cannot provide when STP PortFast is enabled. STP BPDU Guard shuts down ports that receive BPDUs.
Root Guard:
Root guard prevents ports from becoming STP root ports. Use STP Root Guard in order to prevent unsuitable ports from becoming STP root ports. An example of an unsuitable port is a port that links to a device that is outside direct network administrative control.
Loop Guard:
Loop guard is a Cisco proprietary optimization for the STP. Loop guard protects Layer 2 networks from loops that occur when something prevents the normal forwarding of BPDUs on point-to-point links (for example, a network interface malfunction or a busy CPU). Loop guard complements the protection against unidirectional link failures provided by Unidirectional Link Detection (UDLD). Loop guard isolates failures and lets STP converge to a stable topology with the failed component excluded from the STP topology.
BPDU Filter:
This disables the STP. BPDUs are neither sent nor processed upon receipt. It is common with service providers, not necessarily enterprise networks.
UDLD Aggressive:
The Cisco-proprietary UDLD protocol monitors the physical configuration of the links between devices and ports that support UDLD. UDLD detects the existence of unidirectional links. UDLD can operate in either normal or aggressive mode. Normal-mode UDLD classifies a link as unidirectional if the received UDLD packets do not contain information that is correct for the neighbor device. In addition to the functionality of normal mode UDLD, aggressive mode UDLD puts ports into the err-disabled state if the relationship between two previously synchronized neighbors cannot be re-established.
Storm Control:
Traffic storm control is implemented in hardware and does not affect the overall performance of the switch. Typically, end-stations such as PCs and servers are the source of broadcast traffic that can be suppressed. In order to avoid unnecessary processing of excess broadcast traffic, enable traffic storm control for broadcast traffic on access ports that connect to end stations and on ports that connect to key network nodes.
Revision | Publish Date | Comments |
---|---|---|
1.0 |
22-Feb-2024 |
Initial Release |