The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes the steps used to perform an ELAM (Embedded Logic Analyzer Module) on a Cisco Nexus 7000/7700 F3 module.
Cisco recommends that you have a familiarity with the Cisco Nexus Operating System (NX-OS) and basic Nexus architecture before you proceed with the information that is described in this document.
ELAM can only be done by the network-admin role. Please ensure to log in as a user with network-admin privilege.
The information in this document is based on these software and hardware versions:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
The ELAM assists in troubleshooting network forwarding issues by capturing real-time packets without disruptions, and it does not impact performance or control-plane resources. The ELAM is a powerful, granular, and non-intrusive tool most commonly used by Cisco Technical Assistance Center (TAC) engineers. However, it is crucial to know that the ELAM tool captures only one packet at a time - the first packet received after ELAM starts. If you need to capture all packets of a flow, please use SPAN or ERSPAN.
The ELAM can answer questions such as:
In this article, Host1 connected to N77-F3 port E5/23, sends traffic to Host2. ELAM is used to capture the individual frame from Host1 to Host2.
To run an ELAM on the N7K, you first need to log in as a user with network-admin, and then you need to connect to the module.
N77-F3# attach module 5
Attaching to module 5 ...
To exit type 'exit', to abort type '$.'
Last login: Thu Jan 18 05:31:04 pst 2024 from 127.1.1.3 on pts/0
The Nexus 7000 operates as a fully distributed switch, with forwarding decisions made by the forwarding engine of ingress linecard.
In this article, the traffic of interest is expected to ingress the switch via port 5/23. In the example of the N7K, the module 5 is an F3 module.
N77-F3# show module 5
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
5 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
Mod Sw Hw
--- --------------- ------
5 8.4(9) 1.3
For F3 modules, perform the ELAM on the Layer 2 (L2) Forwarding Engine (FE) with the internal codename Flanker.
module-5# show hardware internal dev-port-map
--------------------------------------------------------------
CARD_TYPE: 24 port 40G
>Front Panel ports:24
--------------------------------------------------------------
Device name Dev role Abbr num_inst:
--------------------------------------------------------------
> Flanker Fwd Driver DEV_LAYER_2_LOOKUP L2LKP 12
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF
22 10 10 10 10 0,1
23 11 11 11 11 0,1 >>>Port 23 belongs to FE instance 11
24 11 11 11 11 0,1
+-----------------------------------------------------------------------+
In this output, it is evident that port E5/23 belongs to FE instance 11.
The Layer 2 FE Data Bus (DBUS) carries the original header information prior to the Layer 2 (L2) and Layer 3 (L3) lookups, whereas the Result Bus (RBUS) contains the results of both L3 and L2 lookups. For most troubleshooting scenarios, a Layer 2 ELAM capture is sufficient.
N77-F3# attach module 5
Attaching to module 5 ...
To exit type 'exit', to abort type '$.'
Last login: Thu Jan 18 05:31:04 pst 2024 from 127.1.1.3 on pts/0
module-5# elam asic flanker instance 11
module-5(fln-elam)# ?
layer2 ELAMs for layer 2
layer3 ELAMs for layer 3
module-5(fln-elam)# layer2
In this example, Host1 on VLAN 100 (with an IP address of 192.168.1.1 and MAC address of 8c60.4fc7.c5bc), linked to port E5/23, sends an Address Resolution Protocol (ARP) request. This request is intended to resolve the MAC address for another host on the same VLAN 100, which has an IP address of 192.168.1.2.
The Flanker ASIC supports ELAM triggers for various frame types. The ELAM trigger must correspond with the frame type. If the frame is an ARP frame, the trigger must also be set to the ARP option. An ARP frame cannot be captured by the Other L2 trigger. If you use ELAM to capture an MPLS frame, select IPv4 or IPv6 instead of MPLS. For more details, please refer to the bug section.
module-5(fln-l2-elam)# trigger dbus ?
arp ARP Frame Format >>>capture ARP packet. Other L2 does not work for ARP
fc Fc hdr Frame Format
ipv4 IPV4 Frame Format >>>capture IPv4 frame
ipv6 IPV6 Frame Format >>>capture IPv6 frame
mpls MPLS
other L2 hdr Frame Format >>>capture non-ip l2 frame
rarp RARP Frame Format
In this example, the frame is captured based on the target IP address field of the ARP frame, so only this value is specified.
Flanker necessitates that triggers are set for both the DBUS and the RBUS. The RBUS trigger is simplified,trig
matches the same criteria as the DBUS trigger.
module-5(fln-l2-elam)# trigger dbus arp ingress if target-ip-address 192.168.1.2
module-5(fln-l2-elam)# trigger rbus ingress if trig
Now you have configured the trigger, you can start the capture.
module-5(fln-l2-elam)# start
To verify if ELAM has captured any packets, you can run the status
command. The term Armed indicates that no matching packets have been captured yet.
module-5(fln-l2-elam)# status
ELAM Slot 5 instance 11: L2 DBUS Configuration: trigger dbus arp ingress if target-ip-address 192.168.1.2
L2 DBUS: Armed >>>no matched packet
ELAM Slot 5 instance 11: L2 RBUS Configuration: trigger rbus ingress if trig
L2 RBUS: Armed >>>no matched packet
Ping from Host1 (192.168.1.1) to 192.168.1.2. Since there is no ARP entry on Host1, Host1 sends out the ARP request in a broadcast packet. Once the FE receives the ARP frame, it checks the trigger. If there is a match, ELAM captures the forwarding decision of this frame, and the ELAM status is then displayed as Triggered.
module-5(fln-l2-elam)# status
ELAM Slot 5 instance 11: L2 DBUS Configuration: trigger dbus arp ingress if target-ip-address 192.168.1.2
L2 DBUS: Triggered >>Packet hit
ELAM Slot 5 instance 11: L2 RBUS Configuration: trigger rbus ingress if trig
L2 RBUS: Triggered >>Packet hit
The result is valid only if the DBUS and RBUS have captured the same packet. Therefore, it is necessary to check the sequence number in both DBUS and RBUS results. If they do not match, you can restart and capture them again until they align.
module-5(fln-l2-elam)# show dbus | in seq
sequence-number : 0x7 vl : 0x0
module-5(fln-l2-elam)# show rbus | in seq
l2-rbus-trigger : 0x1 sequence-number : 0x7
It is advisable to first check the DBUS output, as it contains data prior to any rewriting. Here is an example of an ARP ELAM capture. Note that some output has been omitted.
module-5(fln-l2-elam)# show dbus
cp = 0x20c6ad1c, buf = 0x20c6ad1c, end = 0x20c7706c
--------------------------------------------------------------------
Flanker Instance 11 - Capture Buffer On L2 DBUS:
<snip>
--------------------------------------------------------------------
L2 DBUS PRS MLH ARP/RARP
--------------------------------------------------------------------
valid : 0x1 requst-response : 0x1 >>>ARP request
(1:for ARP request,2: for ARP reply, 3:for RARP request, 4:for RARP reply)
port-id : 0x0 last-ethertype : 0x806 >>>Ethernet type, 0x0806 means ARP
packet-type : 0x0 l2-length-check : 0x0 >>>0 for ingress, 1 for egress
vqi : 0x0 packet-length : 0x40 >>>L2 ethernet frame totally length 64 byte
vlan : 0x64 destination-index : 0x0 >>>VLAN100
source-index : 0xb79 bundle-port : 0x0 >>>source port ltl index
status-is-1q : 0x0 trill-encap : 0x0 >>>0 means frame without vlan tag
sender-ip-address: 192.168.1.1 >>>sender-ip-address in ARP header
target-ip-address: 192.168.1.2 >>>target-ip-address in ARP header
sender-mac-address : 8c60.4fc7.c5bc >>>sender-mac-address in ARP header
target-mac-address : ffff.ffff.ffff >>>target-mac-address in ARP header
destination-mac-address : ffff.ffff.ffff >>>sestination mac in ethernet header
source-mac-address : 8c60.4fc7.c5bc >>>source mac in ethernet header
With the DBUS data, you can confirm that the frame is received on VLAN100 (vlan:0x64) with a source MAC address of 8c60.4fc7.c5bc and a destination MAC address of ffff.ffff.ffff. You can also identify that this is an ARP request frame originating from IP 192.168.1.1.
To verify the port on which the frame is received, use the PIXM
(Port Index Manager) command. This command displays the mapping of a Local Target Logic (LTL) to a front port or a group of front ports.
N77-F3# show system internal pixm info ltl 0xb79
------------------
Type LTL
---------------------------------
PHY_PORT Eth5/23
FLOOD_W_FPOE 0xc031
The output reveals that a source-index of 0xb79 maps to port E5/23. This verifies that the frame is received on port E5/23.
After confirming that ELAM has captured the frame of interest, you can verify the result of the forwarding decision using the RBUS data (note that some output has been omitted).
module-5(fln-l2-elam)# show rbus
--------------------------------------------------------------------
L2 RBUS INGRESS CONTENT
--------------------------------------------------------------------
di-ltl-index : 0xc031 l3-multicast-di : 0xc00 >>> destination ltl index
source-index : 0xb79 vlan : 0x64 >>> vlan id after rewritten
vqi : 0x0 di2-valid : 0x0 >>> use l3-multicast-di as di if this is 1
routed-frame : 0x0 copy-cause : 0x0 >>> 0x0 means N7K performs layer 2 switching
With the RBUS data, you can confirm that the frame is switched on VLAN 100 (0x64). To determine the egress port(s) from the di-ltl-index, use the pixm command again.
N77-F3# show system internal pixm info ltl 0xc031
Member info
------------------
IFIDX LTL
---------------------------------
Eth5/24 0x0b78
Eth5/23 0x0b79
The output shows that ports E5/23 and E5/24 both belong to LTL 0xc031. The ARP packet switches to both of these ports. As it is received from E5/23, it is only sent out from E5/24.
In this example, Host1 on VLAN 100 (with an IP address of 192.168.1.1/24 and MAC address of 8c60.4fc7.c5bc), is connected to port E5/23 and sends an Internet Control Message Protocol (ICMP) request to Host2. Host2 has an IP address of 192.168.2.2/24 and is on a different VLAN, VLAN200.
In this example, since Host1 and Host2 are in different VLANs, the ICMP packet from Host1 to Host2 is routed through Layer 3 on N77-F3. A Layer 2 ELAM is used to capture the ICMP request packet.
Both the source IP (192.168.1.2) and the destination IP (192.168.2.2) are combined as DBUS triggers. ELAM only captures the packet that matches all triggers.
N77-F3# attach module 5
Attaching to module 5 ...
To exit type 'exit', to abort type '$.'
Last login: Thu Jan 18 11:19:46 pst 2024 from 127.1.1.3 on pts/0
module-5# elam asic flanker instance 11
module-5(fln-elam)# layer2
module-5(fln-l2-elam)# trigger dbus ipv4 ingress if destination-ipv4-address 192.168.2.2 source-ipv4-address 192.168.1.2
module-5(fln-l2-elam)# trigger rbus ingress if trig
module-5(fln-l2-elam)# start
module-5(fln-l2-elam)# status
ELAM Slot 5 instance 11: L2 DBUS Configuration: trigger dbus ipv4 ingress if destination-ipv4-address 192.168.2.2 source-ipv4-address 192.168.1.2
L2 DBUS: Armed
ELAM Slot 5 instance 11: L2 RBUS Configuration: trigger rbus ingress if trig
L2 RBUS: Armed
Start a ping from Host1 (192.168.1.2) to Host2 (192.168.2.2). Once the packet that matches the triggers is received by FE instance 11, the ELAM status displays as Triggered.
module-5(fln-l2-elam)# status
ELAM Slot 5 instance 11: L2 DBUS Configuration: trigger dbus ipv4 ingress if destination-ipv4-address 192.168.2.2 source-ipv4-address 192.168.1.2
L2 DBUS: Triggered
ELAM Slot 5 instance 11: L2 RBUS Configuration: trigger rbus ingress if trig
L2 RBUS: Triggered
Ensure that the RBUS and DBUS have the same sequence number. This step is necessary for every capture.
module-5(fln-l2-elam)# show dbus | in seq
sequence-number : 0x74 vl : 0x0
module-5(fln-l2-elam)# show rbus | in seq
l2-rbus-trigger : 0x1 sequence-number : 0x74 >>same sequence number, valid elam result
Here is an example of an IPv4 ICMP ELAM capture. Note that some output has been omitted.
module-5(fln-l2-elam)# show dbus
--------------------------------------------------------------------
L2 DBUS PRS MLH IPV4
--------------------------------------------------------------------
l4-protocol : 0x1 df : 0x0 >>>L4 protocol id, 1 means icmp packet
ttl : 0xff l3-packet-length : 0x54 >>>ip total length is 84 in this packet, ttl is 255
port-id : 0x0 last-ethertype : 0x800 >>>Ethernet type, 0x0800 means IPv4
vqi : 0x0 packet-length : 0x66 >>>L2 frame length field
vlan : 0x64 destination-index : 0x0 >>>vlan id 100
source-index : 0xb79 bundle-port : 0x0 >>>source port ltl index
status-is-1q : 0x1 trill-encap : 0x0 >>>1 means frame with vlan tag
source-ipv4-address: 192.168.1.2 >>>Packet source IP
destination-ipv4-address: 192.168.2.2 >>>Packet destination IP
destination-mac-address : 003a.9c40.8ac3 >>>Packet destination mac
source-mac-address : 8c60.4fc7.c5bc >>>Packet source mac
With the DBUS data, you can confirm that the packet is received on VLAN100 (vlan:0x64) with a source IP of 192.168.1.2 and a destination IP of 192.168.2.2. You can also identify that this is an IPv4 ICMP packet.
To verify the port on which the frame is received, run the PIXM
(Port Index Manager) command. This command displays the mapping of a Local Target Logic (LTL) to a front port or a group of front ports.
N77-F3# show system internal pixm info ltl 0xb79
------------------
Type LTL
---------------------------------
PHY_PORT Eth5/23
FLOOD_W_FPOE 0xc032
FLOOD_W_FPOE 0xc031
FLOOD_W_FPOE 0xc029
The output shows that a source-index of 0xb79 maps to port E5/23. This confirms that the frame is received on port E5/23.
After confirming that ELAM has captured the ICMP packet of interest, you can verify the result of the forwarding decision using the RBUS data (note that some output has been omitted). From the RBUS data, you can see that the frame is routed from VLAN 100 (0x64) to VLAN200.
module-5(fln-l2-elam)# show rbus
--------------------------------------------------------------------
L2 RBUS INGRESS CONTENT
--------------------------------------------------------------------
segment-id-valid : 0x0 ttl-out : 0xfe >>>TTL is 254
di-ltl-index : 0xb78 l3-multicast-di : 0x0 >>>destination port ltl
source-index : 0xb79 vlan : 0xc8 >>>vlan id is 200
routed-frame : 0x1 copy-cause : 0x0 >>>routed on N7K
To determine the egress port(s) from the di-ltl-index, run the PIXM
command. The output shows that the egress port is E5/24.
N77-F3# show system internal pixm info ltl 0xb78
Member info
------------------
Type LTL
---------------------------------
PHY_PORT Eth5/24
FLOOD_W_FPOE 0xc032
FLOOD_W_FPOE 0xc031
FLOOD_W_FPOE 0xc029
The output from this command is helpful in understanding the purpose of an LTL if it does not match a physical port. Examples include Drop LTL and Inband LTL:
N77-F3# show system internal pixm info ltl-region
===========================================================
PIXM VDC 1 LTL MAP Version: 3
Description: LTL Map for Crossbow
===========================================================
LTL_TYPE SIZE START END
========================================================================
LIBLTLMAP_LTL_TYPE_SUP_ETH_INBAND 64 0xc00 0xc3f
-------------------------------------------------------------------
LIBLTLMAP_LTL_TYPE_DROP_DI_WO_HW_BITSET 0xcae
LIBLTLMAP_LTL_TYPE_DROP_DI 0xcad
Cisco bug ID | Bug title | Fix version |
Cisco bug ID CSCux73273 | Mpls trigger for ELAM on F3 is not working | No fixed version, use the workaround |
Cisco bug ID CSCvm65736 | N7k: ELAM release trigger clp_elam crash/LC reload | 7.3(3)D1(1) or 8.2(3) or 8.3(2) |
Revision | Publish Date | Comments |
---|---|---|
1.0 |
30-Jan-2024 |
Initial Release |