The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes the use of Embedded Logic Analyzer Module (ELAM) tool within the Unified Computing System (UCS) 4th Generation Fabric Interconnect (FI) 6454, and how to best use it.
There are no prerequisites for this document.
Cisco recommends that you have knowledge of these topics:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
UCS 4th Gen FI has the capability to run ELAM captures. An ELAM capture comes embedded on the ASIC.
ELAM tool allows real time view of the packets being forwarded at the ASIC level. You can view the details of a packet such as:
ELAM provides details of packet forwarding. It is nondisruptive to the data plane.
Log in to UCS via Command Line Interface (CLI).
Run these commands:
#connect nxos a|b
#attach module 1
#debug platform internal tah elam asic 0
#trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1
#set outer ...
#start
#report
Note: 4th Gen FI is a single rack unit with one single module (module 1) with one ASIC (asic 0) and one slice (slice 0). See the output below.
RCH-SV-FFAIII-A(nx-os)# show hardware internal tah interface ethernet 1/30
#########################################
IfIndex: 436222464
DstIndex: 6028
IfType: 26
Interface name Ethernet1/30
Asic: 0
Asic: 0 <<<<<
AsicPort: 49
SrcId: 98
Slice: 0 <<<<<
PortOnSlice: 49
Table entries for interface Ethernet1/30
For ELAMs where the trigger is based on packet attributes "lu-a2d 1" is used. Value 6 and 1 will be used for "in-select" and "out-select" respectively for out interest
The "set outer" command is our filter, this is where we define and tell the FI what packet we want to capture, there are a bunch of options and we can be as granular as needed:
module-1(TAH-elam-insel6)# set outer ?
arp ARP Fields
fcoe FCoE Fields
ipv4 IPv4 Fields
ipv6 IPv6 Fields
l2 All Layer 2 Fields
l4 L4 Fields
module-1(TAH-elam-insel6)# set outer l2 ?
cfi CFI Setting
cntag_vld CNTag Information Valid
cos Class of Service
dst_mac Destination MAC Address
qtag_vld VLAN Tag Information Valid
snap_vld SNAP Header Information Valid
src_mac Source MAC Address
vlan VLAN Id (Present only in case of FEX)
vntag_dvif VNTAG Destination vif
vntag_looped VNTAG Header Looped Valid
vntag_pointer VNTAG Header Pointer Valid
vntag_svif VNTAG Source vif
vntag_vld VNTAG Information Valid
module-1(TAH-elam-insel6)# set outer ipv4 ?
checksum Checksum
dscp Diff. Serv. Code Point
dst_ip Destination IP Address
ecn Explicit Congestion Ntfn
fragment-off Fragments Offset
header-len Header Length
more-frags More Fragments Available
next-protocol Next Protocol
packet-len Packet Total Length
pyld-len Payload Length
src_ip Source IP Address
ttl Time to Live
version Version
Once the filters have been defined, run the command start to run the ELAM tool. If nothing that fullfills the filter condition has been captured then, this is seen:
module-1(TAH-elam-insel6)# report
ELAM not triggered yet on slot - 1, asic - 0, slice - 0
Note: The "set" command will survive across ELAMs, a good practice is to run a "reset" command everytime we intend to capture traffic with different IP, MAC, etc.
1. Ping from VM 172.16.35.31 to gateway 172.16.35.126:
RCH-SV-FFAIII-A(nx-os)# attach module 1
module-1# debug platform internal tah elam asic 0
module-1(TAH-elam)# trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1
param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 1
module-1(TAH-elam-insel6)# set outer ipv4 src_ip 172.16.35.31 dst_ip 172.16.35.126
module-1(TAH-elam-insel6)# start
GBL_C++: [MSG] rocky_elam_wrapper_init:54:asic type 8 inst 0 slice 0 a_to_d 1 insel 6 outsel 1
GBL_C++: [MSG] rocky_elam_wrapper_enable:149:asic type 8 inst 0 slice 0 a_to_d 1
GBL_C++: [MSG] - writing data=0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000005608118F800000
000000000000000000560811BF00000000000000000000001000000000000000000000000000000000000000000000000000000000001
GBL_C++: [MSG] - writing mask=0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000007FFFFFFF800000
0000000000000000007FFFFFFF80000000000000000000003800000000000000000000000000000000000000000000000000000000001
module-1(TAH-elam-insel6)# report
HOMEWOOD ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 0
============================
Incoming Interface: Eth1/33
Src Idx : 0x1002, Src BD : 35
Outgoing Interface Info: dmod 1, dpid 4
Dst Idx : 0x604, Dst BD : 35
Packet Type: IPv4
Dst MAC address: 8C:60:4F:CD:FD:7C
Src MAC address: 00:25:C5:00:00:1E
.1q Tag0 VLAN: 35, cos = 0x1
Dst IPv4 address: 172.16.35.126
Src IPv4 address: 172.16.35.31
Ver = 4, DSCP = 0, Don't Fragment = 0
Proto = 1, TTL = 64, More Fragments = 0
Hdr len = 20, Pkt len = 84, Checksum = 0x5f19
L4 Protocol : 1
ICMP type : 8
ICMP code : 0
Drop Info:
----------
LUA:
LUB:
LUC:
LUD:
Final Drops:
The packet with src_ip 172.16.35.31 and dst_ip 172.16.35.126 is on VLAN 35 and arrived on port 1/33 (incoming interface) and was destined (outgoing interface) to interface "dpid4" ...the what? "dpid" is the ASIC port internal identifier and the mapping can be found with "show interface hardware-mappings":
Incoming Interface: Eth1/33
Src Idx : 0x1002, Src BD : 35
interface Ethernet1/33
description S: Server, Port-channel 1025
no pinning server sticky
switchport mode fex-fabric
priority-flow-control mode on
fex associate 1
channel-group 1025
no shutdown
Outgoing Interface Info: dmod 1, dpid 4
Dst Idx : 0x604, Dst BD : 35
RCH-SV-FFAIII-A(nx-os)# show interface hardware-mappings
Legends:
SMod - Source Mod. 0 is N/A
Unit - Unit on which port resides. N/A for port channels
HPort - Hardware Port Number or Hardware Trunk Id:
HName - Hardware port name. None means N/A
FPort - Fabric facing port number. 255 means N/A
NPort - Front panel port number
VPort - Virtual Port Number. -1 means N/A
Slice - Slice Number. N/A for BCM systems
SPort - Port Number wrt Slice. N/A for BCM systems
SrcId - Source Id Number. N/A for BCM systems
------------------------------------------------------------------------
Name Ifindex Smod Unit HPort FPort NPort VPort Slice SPort SrcId
------------------------------------------------------------------------
.
Eth1/13 1a001800 1 0 4 255 48 -1 0 4 8
This "dpid 4" also corresponds to what the "show hardware internal tah interface ethernet 1/13" suggests:
RCH-SV-FFAIII-A(nx-os)# show hardware internal tah interface ethernet 1/13
#########################################
IfIndex: 436213760
DstIndex: 6096
IfType: 26
Interface name Ethernet1/13
Asic: 0
Asic: 0
AsicPort: 4 <<<<<
SrcId: 8
Slice: 0
PortOnSlice: 4 <<<<<
The packet was identified as an Internet Control Message Protocol (ICMP) by the ELAM Layer 4 (L4) Protocol. Refer to list of IANA protocol numbers. You can also filter with a specific MTU size. ELAM triggers only when the exact MTU is hit.
module-1(TAH-elam-insel6)# set outer ipv4 src_ip 172.16.35.31 dst_ip 172.16.35.126 packet-len 1500
Dst IPv4 address: 172.16.35.126
Src IPv4 address: 172.16.35.31
Ver = 4, DSCP = 0, Don't Fragment = 1
Proto = 1, TTL = 64, More Fragments = 0
Hdr len = 20, Pkt len = 1500, Checksum = 0x1758
L4 Protocol : 1
ICMP type : 8
ICMP code : 0
ARP request from the Virtual Machine (VM) to upstream network, with the MAC addresses set as filter:
RCH-SV-FFAIII-B(nx-os)# attach module 1
module-1# debug platform internal tah elam asic 0
module-1(TAH-elam)# trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1
param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 1
module-1(TAH-elam-insel6)# set outer l2 src_mac 00:25:c5:00:00:1e dst_mac ff:ff:ff:ff:ff:ff
module-1(TAH-elam-insel6)# start
GBL_C++: [MSG] rocky_elam_wrapper_init:36:asic type 8 inst 0 slice 0 a_to_d 1 insel 6 outsel 1
GBL_C++: [MSG] rocky_elam_wrapper_enable:95:asic type 8 inst 0 slice 0 a_to_d 1
GBL_C++: [MSG] - writing data=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000015820463E004B8A00003C0000000000000000000002000000000000000000000000000000000000000000000000000000000001
GBL_C++: [MSG] - writing mask=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
0000001FFFFFFFFFFFFFFFFFFFE0000000000000000000003800000000000000000000000000000000000000000000000000000000001
module-1(TAH-elam-insel6)# report
HOMEWOOD ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 0
============================
Incoming Interface: Eth1/33
Src Idx : 0x1002, Src BD : 35
Outgoing Interface Info: dmod 1, dpid 4
Dst Idx : 0x604, Dst BD : 35
Packet Type: ARP
Dst MAC address: FF:FF:FF:FF:FF:FF
Src MAC address: 00:25:C5:00:00:1E
.1q Tag0 VLAN: 35, cos = 0x1
Target Hardware address: 00:00:00:00:00:00
Sender Hardware address: 00:25:C5:00:00:1E
Target Protocol address: 172.16.35.110
Sender Protocol address: 172.16.35.31
ARP opcode: 1
Drop Info:
----------
LUA:
LUB:
LUC:
LUD:
Final Drops:
The packet is identified by the system as an ARP, this is especially useful when there is an incomplete ARP entry either at the VM or gateway level.
If applicable, the TCP/UDP ports will be listed as well, SSH is tested here:
RCH-SV-FFAIII-B(nx-os)# attach module 1
module-1# debug platform internal tah elam asic 0
module-1(TAH-elam)# trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1
param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 1
module-1(TAH-elam-insel6)# set outer ipv4 src_ip 172.16.35.126 dst_ip 172.16.35.31
module-1(TAH-elam-insel6)# start
GBL_C++: [MSG] rocky_elam_wrapper_init:36:asic type 8 inst 0 slice 0 a_to_d 1 insel 6 outsel 1
GBL_C++: [MSG] rocky_elam_wrapper_enable:95:asic type 8 inst 0 slice 0 a_to_d 1
GBL_C++: [MSG] - writing data=0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000560811BF0000
000000000000000000005608118F80000000000000000000001000000000000000000000000000000000000000000000000000000000001
GBL_C++: [MSG] - writing mask=0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000007FFFFFFF8000
000000000000000000007FFFFFFF80000000000000000000003800000000000000000000000000000000000000000000000000000000001
module-1(TAH-elam-insel6)# report
HOMEWOOD ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 0
============================
Incoming Interface: Eth1/14
Src Idx : 0x604, Src BD : 35
Outgoing Interface Info: dmod 1, dpid 44
Dst Idx : 0x1002, Dst BD : 35
Packet Type: IPv4
Dst MAC address: 00:25:C5:00:00:1E
Src MAC address: 8C:60:4F:CD:FD:7C
.1q Tag0 VLAN: 35, cos = 0x0
Dst IPv4 address: 172.16.35.31
Src IPv4 address: 172.16.35.126
Ver = 4, DSCP = 0, Don't Fragment = 0
Proto = 6, TTL = 64, More Fragments = 0
Hdr len = 20, Pkt len = 60, Checksum = 0x27f5
L4 Protocol : 6
TCP Dst Port : 22
TCP Src Port : 15067
Drop Info:
----------
LUA:
LUB:
LUC:
LUD:
Final Drops:
Drops are recorded as well. The FI drops an ARP request:
RCH-SV-FFAIII-B(nx-os)# attach module 1
module-1# debug platform internal tah elam asic 0
module-1(TAH-elam)# trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1
param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 1
module-1(TAH-elam-insel6)# set outer l2 src_mac 00:25:c5:00:00:1e dst_mac ff:ff:ff:ff:ff:ff
module-1(TAH-elam-insel6)# start
GBL_C++: [MSG] rocky_elam_wrapper_init:54:asic type 8 inst 0 slice 0 a_to_d 1 insel 6 outsel 1
GBL_C++: [MSG] rocky_elam_wrapper_enable:149:asic type 8 inst 0 slice 0 a_to_d 1
GBL_C++: [MSG] - writing data=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000000000971400007BFFFFFFFFFFFC00001
GBL_C++: [MSG] - writing mask=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000003FFFFFFFFFFFFFFFFFFFFFFFC00001
module-1(TAH-elam-insel6)# report
HOMEWOOD ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 0
============================
Incoming Interface: Eth1/18
Src Idx : 0x603, Src BD : 35
Outgoing Interface Info: dmod 0, dpid 0
Dst Idx : 0x0, Dst BD : 35
Packet Type: ARP
Dst MAC address: FF:FF:FF:FF:FF:FF
Src MAC address: 00:25:C5:00:00:1E
.1q Tag0 VLAN: 35, cos = 0x1
Target Hardware address: 00:00:00:00:00:00
Sender Hardware address: 00:25:C5:00:00:1E
Target Protocol address: 172.16.35.99
Sender Protocol address: 172.16.35.31
ARP opcode: 1
Drop Info:
----------
LUA:
LUB:
LUC:
LUD:
MC_RPF_FAIILURE
SRC_VLAN_MBR
Final Drops:
MC_RPF_FAIILURE
SRC_VLAN_MBR
The FI receives an ARP request on port 1/18 (this is an uplink port) with a source MAC of 00:25:c5:00:00:1e which is learned locally on a virtual Ethernet (vEth) ports. This condition triggers a Reverse Path Forward (RPF) drop. Notice that the Outgoing Interface Info reports dpid 0 which is a drop.
VLAN 35 is not allowed on port 1/18 and this also triggered drop SRC_VLAN_MBR.
RCH-SV-FFAIII-A(nx-os)# show run interface ethernet 1/18
interface Ethernet1/18
description U: Uplink
pinning border
switchport mode trunk
switchport trunk allowed vlan 1
channel-group 105 mode active