The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes how to configure Switched Port Analyzer (SPAN) on Cisco Application Centric Infrastructure (ACI).
In general, there are three types of SPAN. Local SPAN, Remote SPAN (RSPAN) and Encapsulated Remote SPAN (ERSPAN). The differences between these SPANs are mainly the destination of copy packets. Cisco ACI supports Local SPAN and ERSPAN.
Note: This document assumes that readers are already familiar with SPAN in general such as Local SPAN and ERSPAN differences.
Cisco ACI has three types of SPAN; Fabric SPAN
, Tenant SPAN
and Access SPAN
. The difference between each SPANs is the source of copy packets.
As mentioned previously,
Fabric SPAN
is to capture packets that come in and go out from interfaces between Leaf and Spine switches
.Access SPAN
is to capture packets that come in and go out from interfaces between Leaf switches and external devices
.Tenant SPAN
is to capture packets that come in and go out from EndPoint Group (EPG) on ACI Leaf switches
.This SPAN name corresponds to where to be configured on Cisco ACI GUI.
Fabric > Fabric Policies
Fabric > Access Policies
Tenants > {each tenant}
As for the destination of each SPAN, only Access SPAN
is capable of both Local SPAN
and ERSPAN
. The other two SPAN (Fabric
and Tenant
) are only capable of ERSPAN
.
Please review the Limitations & Guidelines from Cisco APIC Troubleshooting Guide. It is mentioned in Troubleshooting Tools and Methodology > Using SPAN
.
This section introduces brief examples that relate to the configuration for each SPAN Type. There are specific sample cases on how to select the span type in the later section.
SPAN Configuration is also described in Cisco APIC Troubleshooting Guide: Troubleshooting Tools and methodology > Using SPAN.
The UI can appear different than the current versions but the config approach is the same.
Where:
Navigate to FABRIC > ACCESS POLICIES > Troubleshoot Policies > SPAN
.
SPAN Source Groups
SPAN Destination Groups
SPAN Source Group
ties Destination
and Sources
.
How:
SPAN Source Group
(SRC_GRP1).SPAN Source
(SRC1) under SPAN Source Group
(SRC_GRP1).SPAN Source
(SRC1).Note: Please refer to the picture for details of each parameter.
SPAN Destination Group
(DST_EPG).SPAN Destination
(DST).SPAN Destination
(DST)Note: Please refer to the picture for details of each parameter.
SPAN Destination Group
is tied to an appropriate SPAN Source Group
.Admin State
is Enabled.Note: SPAN stops when you select Disabled on this Admin State. There is no need to delete all policies if you re-use them later.
Also please ensure that the destination IP for ERSPAN is learned as an endpoint under the specified destination EPG. In the previously mentioned example, 192.168.254.1 has to be learned under Tenant TK > Application profile SPAN_APP > EPG SPAN
. Or the destination IP can be configured as a static EndPoint under this EPG if the destination device is a silent host.
Fabric > ACCESS POLICIES > Troubleshoot Policies > SPAN
- SPAN Source Groups
- SPAN Destination Groups
SPAN Source Group
ties Destination
and Sources
.
SPAN Source Group
(SRC_GRP1)SPAN Source
(SRC1) under SPAN Source Group
(SRC_GRP1)SPAN Source
(SRC1)SPAN Destination Group
(DST_Leaf1)SPAN Destination
(DST)SPAN Destination
(DST)SPAN Destination Group
is tied to an appropriate SPAN Source Group
.Ensure Admin State
is Enabled.
※ SPAN stops when you select Disabled on this Admin State. There is no need to delete all policies if you re-use them later.
The destination interface does not require any configuration by Interface Policy Groups. It works when you plug a cable into the interface on ACI Leaf.
Limitations:
You can use ACL filters on access span sources. This feature provides the ability to SPAN a particular flow or flow of traffic in/out of a SPAN source.
Users can apply the SPAN Acl(s) to a source when there is a need to SPAN flow specific traffic.
It is not supported in Fabric SPAN and Tenant Span source groups/sources.
Care must be taken when you add filter entries in a filter group since it could add tcam entries for every source that currently uses the filter group.
A Filter Group can be associated to:
-Span Source: the filter group is used to filter traffic on ALL interfaces defined under this Span Source.
-Span Source Group: the filter group (say x) is used to filter traffic on ALL interfaces defined under each of the Span Source(s) of this Span Source Group.
In this configuration snapshot, the filter group is applied to the Span source group.
In the case where a particular Span Source already associates with a Filter Group (say y), that filter group (y) is used instead to filter group on all interfaces under this specific Span Source
- A Filter group that is applied at a source group automatically applies to all sources in that source group.
- A Filter group that is applied at a source is applicable to that source only.
- A filter group is applied at both the source group and a source in that source group, the filter group applied at the source takes precedence.
- A filter group applied at a source is deleted, filter group applied at the parent source group is automatically applied.
- A filter group applied at a source group is deleted, it is deleted from all sources currently that inherit in that source group.
Tenants > {tenant name} > Troubleshoot Policies > SPAN
- SPAN Source Groups
- SPAN Destination Groups
※ SPAN Source Group ties Destination
and Sources
.
SPAN Source Group
(SRC_GRP)SPAN Source
(SRC_A) under SPAN Source Group
(SRC_GRP)SPAN Source
(SRC_A)SPAN Destination Group
(DST_GRP)SPAN Destination
(DST_A)SPAN Destination
(DST_A)SPAN Destination Group
is tied to an appropriate SPAN Source Group
.Admin State
is Enabled.Fabric > FABRIC POLICIES > Troubleshoot Policies > SPAN
- Fabric
- SPAN Destination Groups
※ SPAN Source Group
ties Destination
and Sources
SPAN Source Group
(SRC_GRP)SPAN Source
(SRC_A) under SPAN Source Group
(SRC_GRP)SPAN Source
(SRC_A)SPAN Destination Group
(DST_GRP)SPAN Destination
(DST_A)SPAN Destination
(DST_A)SPAN Destination Group
is tied to an appropriate SPAN Source Group
.Admin State
is Enabled.Admin State
. There is no need to delete all policies if you re-use them later.Although it is described in a later section "ERSPAN Version (type)", you can tell ERSPAN version II is used for Fabric SPAN and version I is used for Tenant and Access SPAN.
Fabric > ACCESS POLICIES > Troubleshoot Policies > SPAN > SPAN Source Groups > Operational tab
Fabric > FABRIC POLICIES > Troubleshoot Policies > SPAN > SPAN Source Groups > Operational tab
Tenants > {tenant name} > Troubleshoot Policies > SPAN > SPAN Source Groups > Operational tab
Please ensure Operational State is up.
SPAN Configuration Policy
or Fabric > INVENTORY > Node > Span Sessions > { SPAN session name }
Please ensure Operational State is up.
SPAN Session naming convention:
- Fabric SPAN: fabric_xxxx
- Access SPAN: infra_xxxx
- Tenant SPAN: tn_xxxx
In this section, detailed scenarios are described for each ACI SPAN type (Access, Tenant, Fabric
) The base topology for each scenario is mentioned in the previous section.
If you understand these scenarios, you can select the appropriate ACI SPAN type for your requirement such as packets on only specific interfaces must be captured or all packets on a specific EPG regardless of interfaces must be captured, and more.
In Cisco ACI, SPAN is configured with the source group
and destination group
. The Source group contains multiple source factors such as interfaces or EPG. The destination group contains destination information such as the destination interface for Local SPAN or destination IP for ESPAN.
After packets are captured, please see the section "How to Read SPAN Data" to decode captured packets.
Note: Please focus on VMs highlighted with a green light in each topology. Each scenario is to capture packets from these highlighted VMs.
Source Group
Destination Group
Access SPAN can specify multiple interfaces for a single SPAN session. It can capture all packets that come in or go out from specified interfaces regardless of their EPG.
When multiple interfaces are specified as a source group from multiple Leaf switches, the destination group must be ERSPAN, not Local SPAN.
In this example, it copies packets from all VMs on EPG1 and EPG2.
CLI Check Point
destination-ip
" is destination IP for ERSPANorigin-ip
" is source IP for ERSPANIn this example, Leaf1 e1/34 is removed from the SPAN Source Group configured at previous Case1.
The key point in this example is that Access SPAN can specify source interfaces regardless of EPG.
CLI Check Point
This example shows that Access SPAN also can specify a specific EPG on the source ports. This is useful when multiple EPGs flow on a single interface and it is required to capture traffic only for EPG1 on this interface.
Since EPG1 is not deployed on Leaf2, SPAN for Leaf2 fails with faults F1553 and F1561. However, SPAN on Leaf1 still works.
Also, two VLAN filters are automatically added for the SPAN session on Leaf1 because EPG1 uses two VLANs (VLAN-751,752) on Leaf1.
Please be noted that the VLAN ID on CLI (35, 39) is the internal VLAN so-called PI-VLAN(Platform Independent VLAN) which is not the actual ID on the wire. As shown in the picture, show vlan extended command shows the mapping of the actual encap VLAN ID and PI-VLAN.
This SPAN session allows us to capture packets only for EPG1(VLAN-752) on Leaf1 e1/11 even though EPG2 (VLAN-753) flows on the same interface.
CLI Check Point
When the vPC interface is configured as a source, a destination must be remote IP (ERSPAN) not the interface (Local SPAN)
Access SPAN can also use Local SPAN (that is a specific interface as a destination)
However, in this case, source interfaces must be on the same Leaf as the destination interface.
Access SPAN with Local SPAN can also use EPG Filter as well as ERSPAN.
It is similar to case 3 on Access SPAN (ERSPAN) but in this example, the one and only SPAN session on Leaf1 fails because EPG3 does not exist on Leaf1. So SPAN does not work at all.
EPG filter on Access SPAN works only when source ports are configured. If EPG is the only source to be specified, Tenant SPAN must be used instead of Access SPAN.
A vPC interface cannot be configured as a source with Local SPAN. Please use ERSPAN. Please refer to case4 for Access SPAN (ERSPAN).
If a destination I/F for SPAN already belongs to EPG, a fault "F1696 : Port has an invalid configuration of both EPG and span destination" is raised under the physical I/F.
But even with this fault, SPAN works without any problems. This fault is just a warning about extra traffic caused by SPAN since it can impact customers' normal EPG traffic on the same I/F.
Tenant SPAN uses EPG itself as a source while Access SPAN use EPG just for a filter.
The key point of Tenant SPAN is that you do not have to specify each individual port and ACI automatically detects appropriate VLANs on each Leaf switch. So this would be useful when all packets for specific EPG must be monitored and EndPoints for that EPG belong to multiple interfaces across Leaf switches.
Fabric SPAN specifies Fabric ports as a source where Fabric ports are interfaces between Leaf and Spine switches.
This SPAN is useful when it is required to copy packets between Leaf and Spine switches. However, packets between Leaf and Spine switches are encapsulated with iVxLAN header. So it is required a bit of a trick to read it. Please refer to "How to Read SPAN Data".
Note: iVxLAN header is an enhanced VxLAN header only for ACI Fabric internal use.
Fabric SPAN can use filters as well as Access SPAN. But the filter type is different. Fabric SPAN uses Virtual Routing and Forwarding (VRF) or BD as a filter.
In Cisco ACI, as described before, packets that go through Fabric ports are encapsulated with iVxLAN header. This iVxLAN header has VRF or BD information as Virtual Network Identifier (VNID). When packets are forwarded as Layer2 (L2), iVxLAN VNID stands for BD. When packets are forwarded as Layer3 (L3), iVxLAN VNID stands for VRF.
So when it is required to capture routed traffic on Fabric ports, use VRF as a filter.
As described in previous case 2, Fabric SPAN can use BD as a filter.
When it is required to capture bridged traffic on Fabric ports, use BD as a filter.
Note: Only a single filter of either BD or VRF can be configured at a time.
Just run a packet capture application such as tcpdump, wireshark
on it. It is not required to configure the ERSPAN destination session or anything.
Please ensure to run a capture tool on the interface with the destination IP for ERSPAN since SPAN packets are forwarded to the destination IP.
The received packet is encapsulated with a GRE header. Please refer to this section "How to Read ERSPAN Data" on how to decode the ERSPAN GRE header.
Please ensure to run a capture tool on the interface that connects to the SPAN Destination interface on ACI Leaf.
Raw packets are received in this interface. it is not required to deal with the ERSPAN header.
ERSPAN encapsulates copied packets to forward them to the remote destination. GRE is used for this encapsulation. The protocol type for ERSPAN on the GRE header is 0x88be.
In Internet Engineering Task Force (IETF) document, the ERSPAN version is described as type instead of version.
There are three types of ERSPAN. I, II and III. ERSPAN Type is mentioned in this RFC draft. Also, this GRE RFC1701 can be helpful to understand each ERSPAN type as well.
Here is the packet format of each type:
Type I does not use the sequence field on the GRE header. It does not even use the ERSPAN header which must succeed the GRE header if it was ERSPAN type II and III. Broadcom Trident 2 only supports this ERSPAN type I.
If the sequence field is activated by the S bit, this must be ERSPAN type II or III. The version field on the ERSPAN header identifies the ERSPAN type. In ACI, type III is not supported as of 03/20/2016.
If a SPAN source group for Access or Tenant SPAN has sources on both 1st-gen and 2nd-gen nodes, the ERSPAN destination receives both ERSPAN Type I and II packets from each generation of nodes. However, Wireshark can decode only one of the ERSPAN Types at a time. By default, it only decodes ERSPAN Type II. If you enable the decode of ERSPAN Type I, Wireshark does not decode ERSPAN Type II. See the later section on how to decode ERSPAN Type I on Wireshark.
To avoid this type of issue, you can configure ERSPAN Type on a SPAN destination group.
By default, SPAN Version is Version 2 and Enforce SPAN Version is unchecked. This means that if the source node is 2nd gen or later which supports ERSPAN Type II, it generates ERSPAN with Type II. If the source node is 1st gen which does not support ERSPAN Type II (except for Fabric SPAN), it falls back to Type I since the Enforce SPAN Version is not checked. As a result, the ERSPAN destination receives a mixed type of ERSPAN.
This table explains each combination for Access and Tenant SPAN.
SPAN Version |
Enforce SPAN Version |
1st gen source node |
2nd gen source node |
Version 2 |
Unchecked |
Uses Type I |
Uses Type II |
Version 2 |
Checked |
Fails |
Uses Type II |
Version 1 |
Unchecked |
Uses Type I |
Uses Type I |
Version 1 |
Checked |
Uses Type I |
Uses Type I |
Packets need to be decoded since it is encapsulated by ERSPAN Type I. This can be done with Wireshark. Please refer to the section "How to Decode ERSPAN Type 1".
Wireshark automatically decodes ERSPAN Type II. However, it is still encapsulated by iVxLAN header.
By default, Wireshark does not understand the iVxLAN header since it is ACI internal header. Please refer to "How to Decode iVxLAN Header".
Option 1. Navigate to Edit > Preference > Protocols > ERSPAN
and check FORCE to decode the fake ERSPAN frame.
user1@linux# tshark -f 'proto GRE' -nV -i eth0 -o erspan.fake_erspan:true
Note: Please ensure to disable this option when you read ERSPAN type II or III.
Option 2. Navigate to Decode As > Network > ICMP (if it’s ICMP)
.
iVxLAN header uses destination port 48879. So, you can decode iVxLAN header as well as VxLAN if you configure UDP destination port 48879 as VxLAN on Wireshark.
Analyze > Decode As > Transport > UDP destination (48879) > VxLAN
.Apply
.Note: There are communication packets between APICs on Fabric ports. Those packets are not encapsulated by iVxLAN header.
When you take an erspan capture on a user network that runs Precision Time Protocol (PTP) sometimes it is seen that Wireshark does not interpret the data due to an unknown ethertype within the GRE encap (0x8988). 0x8988 is the ethertype for the time tag that is inserted into dataplane packets when PTP is enabled. Decode the ethertype 0x8988 as "Cisco ttag" to expose the details of the packet.
Revision | Publish Date | Comments |
---|---|---|
2.0 |
06-Aug-2024 |
Renamed to include Call-to-Action word
Removed 'follow'
Updated MDF Tag |
1.0 |
01-Feb-2023 |
Initial Release |