Contents

Introduction

Topology

Determine the Ingress Forwarding Engine

Configure the Trigger

Start the Capture

Interpret the Results

Additional Verification

Introduction

This document describes the steps used in order to perform an ELAM on Cisco Nexus 7700 (N7700) M3 modules, explains the most relevant outputs, and describes how to interpret the results.

Tip: Refer to the ELAM Overview document for an overview on ELAM.

Topology

In this example, a host on VLAN 2500 (10.0.5.101), port Eth4/1 sends an Internet Control Message Protocol (ICMP) request to a host on VLAN 55 (10.0.3.101), port Eth3/5. ELAM is used in order to capture this single packet from 10.0.5.101 to 10.0.3.101. It is important to remember that ELAM allows you to capture a single frame.

In order to perform an ELAM on the N7K, you must first connect to the appropriate module (this requires the network-admin privilege):

N7700# attach module 4

Attaching to module 4 ...

module-4#

Determine the Ingress Forwarding Engine

Traffic is expected to ingress the switch on port Eth4/1. When you check the modules in the system, you see that Module 4 is an M3 module. It is important to remember that the N7K is fully-distributed, and that the modules, not the supervisor, make the forwarding decisions for dataplane traffic.

N7700# show module

Mod  Ports  Module-Type                         Model              Status

---  -----  ----------------------------------- ------------------ ----------

1    12     100 Gbps Ethernet Module            N77-F312CK-26      ok

3    48     1/10 Gbps Ethernet Module           N77-M348XP-23L     ok

4    24     10/40 Gbps Ethernet Module          N77-M324FQ-25L     ok

5    0      Supervisor Module-2                 N77-SUP2E          active *

6    0      Supervisor Module-2                 N77-SUP2E          ha-standby

7    24     10/40 Gbps Ethernet Module          N77-F324FQ-25      ok

Mod  Sw               Hw

---  ---------------  ------

1    7.3(0)DX(1)      1.1

3    7.3(0)DX(1)      1.1

4    7.3(0)DX(1)      1.0

5    7.3(0)DX(1)      1.2

6    7.3(0)DX(1)      1.2

7    7.3(0)DX(1)      1.0

For M-Series modules, perform the ELAM on the Layer 2 (L2) Forwarding Engine (FE) with internal codename F4. Note that the L2 FE Data Bus (DBUS) contains the original header information before the L2 and Layer 3 (L3) lookups, and the Result Bus (RBUS) contains the results after both L3 and L2 lookups.

N7K M3 modules can use multiple FEs for each module, so you must determine the F4 ASIC that is used for the FE on port Eth4/1. Enter this command in order to verify this:

module-4# show hardware internal dev-port-map

(some output omitted)

--------------------------------------------------------------

CARD_TYPE:       24 port 40G

>Front Panel ports:24

--------------------------------------------------------------

Device name             Dev role              Abbr num_inst:

--------------------------------------------------------------

> SLF L3 Driver          DEV_LAYER_3_LOOKUP     L3LKP  4

> SLF L2FWD driver       DEV_LAYER_2_LOOKUP     L2LKP  4

+-----------------------------------------------------------------------+

+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+

+-----------------------------------------------------------------------+

FP port |  PHYS | MAC_0 | RWR_0 | L2LKP | L3LKP | QUEUE |SWICHF

1               0       0       0       0       0       0,1

2               0       0       0       0       0       0,1

3               0       0       0       0       0       0,1

In the output, you can see that port Eth4/1 is on F4 (L2LKP) instance 0. On the N77-M312CQ-26L module, there are 6 F4 ASICs with 2 ports in each port group. On the N77-M324FQ-25L module, there are 4 F4 ASICs with 6 ports in each port group. The N77-M348XP-23L module has 2 F4 ASICs with 12 ports in each port group.

Note: Just like F-series modules, M3 module ELAM syntax uses 0-based values. This is not the case for M1 and M2 modules, which use 1-based values.

module-4# elam asic f4 instance 0

module-4(f4-elam)# layer2

module-4(f4-l2-elam)#

Configure the Trigger

The F4 ASIC supports ELAM triggers for IPv4, IPv6, and others. The ELAM trigger must align with the frame type. If the frame is an IPv4 frame, then the trigger must also be IPv4. An IPv4 frame is not captured with an other trigger. The same logic applies to IPv6.

With Nexus Operating Systems (NX-OS), you can use the question mark character in order to separate the ELAM trigger:

module-4(f4-l2-elam)# trigger dbus ipv4 ingress if ?

(some output omitted)

destination-index         Destination-index

destination-ipv4-address  Destination ipv4 address

destination-ipv4-mask     Destination ipv4 mask

destination-mac-address   Destination mac address

l4-protocol               L4 protocol

source-index              Source-index

source-ipv4-address       Source ipv4 address

source-ipv4-mask          Source ipv4 mask

source-mac-address        Source mac address

For this example, the frame is captured according to the source and destination IPv4 addresses, so only those values are specified.

F4 requires separate triggers for the DBUS and the RBUS.

Here is the DBUS trigger:

module-4(f4-l2-elam)# trigger dbus ipv4 ingress if source-ipv4-address

10.0.5.101 destination-ipv4-address 10.0.3.101

Here is the RBUS trigger:

module-4(f4-l2-elam)# trigger rbus ingress result if tr 1

Start the Capture

Now that the ingress FE is selected and you configured the trigger, you can start the capture:

module-4(f4-l2-elam)# start

In order to check the status of the ELAM, enter the status command:

module-4(f4-l2-elam)# status

ELAM Slot 4 instance 0: L2 DBUS/LBD Configuration: trigger dbus ipv4 ingress if

source-ipv4-address 10.0.5.101 destination-ipv4-address 10.0.3.101

L2 DBUS/LBD: Configured

ELAM Slot 4 instance 0: L2 RBUS Configuration: trigger rbus ingress result if tr 1

L2 RBUS: Configured

L2 BIS: Unconfigured

L2 BPL: Unconfigured

L2 EGR: Unconfigured

L2 PLI: Unconfigured

L2 PLE: Unconfigured

Once the frame that matches the trigger is received by the FE, the ELAM status shows as Triggered:

module-4(f4-l2-elam)# status

ELAM Slot 4 instance 1: L2 DBUS/LBD Configuration: trigger dbus ipv4 ingress if

source-ipv4-address 10.0.5.101 destination-ipv4-address 10.0.3.101

L2 DBUS/LBD: Triggered

ELAM Slot 4 instance 1: L2 RBUS Configuration: trigger rbus ingress result if tr 1

L2 RBUS: Triggered

L2 BIS: Unconfigured

L2 BPL: Unconfigured

L2 EGR: Unconfigured

L2 PLI: Unconfigured

L2 PLE: Unconfigured

7

Interpret the Results

In order to display the ELAM results, enter the show dbus and show rbus commands. If there is a high volume of traffic that matches the same triggers, the DBUS and RBUS might trigger on different frames. Therefore, it is important to check the internal sequence numbers on the DBUS and RBUS data in order to ensure that they match:

module-4(f4-l2-elam)# show dbus | i seq

port-id             : 0x0             sequence-number : 0x868

module-4(f4-l2-elam)# show rbus | i seq

de-bri-rslt-valid   : 0x1             sequence-number : 0x868

Here is the excerpt from the ELAM data that is most relevant to this example (some output is omitted):

module-4(f4-l2-elam)# show dbus

--------------------------------------------------------------------

LBD IPV4

--------------------------------------------------------------------

ttl                 : 0xff            l3-packet-length    : 0x54

destination-address: 10.0.3.101

source-address: 10.0.5.101

--------------------------------------------------------------------

packet-length       : 0x66            vlan                : 0x9c4

segid-lsb           : 0x0             source-index        : 0xe05

destination-mac-address : 8c60.4f07.ac65

source-mac-address : 8c60.4fb7.3dc2

port-id             : 0x0             sequence-number     : 0x868

module-4(f4-l2-elam)# show rbus

--------------------------------------------------------------------

L2 RBUS RSLT CAP DATA

--------------------------------------------------------------------

de-bri-rslt-valid   : 0x1             sequence-number     : 0x868

vlan                : 0x37            rbh                 : 0x65

cos                 : 0x0             destination-index   : 0x9ed

With the DBUS data, you can verify that the frame is received on VLAN 2500 with a source MAC address of 8c60.4fb6.3dc2 and a destination MAC address of 8c60.4f07.ac65. You can also see that this is an IPv4 frame that is sourced from 10.0.5.101, and is destined to 10.0.3.101.

Tip: There are several other useful fields that are not included in this output, such as Type of Service (TOS) value, IP flags, IP length, and L2 frame length.

In order to verify on which port the frame is received, enter the SRC_INDEX command (the source Local Target Logic (LTL)). Enter this command in order to map an LTL to a port or group of ports for the N7K:

N7700# show system internal pixm info ltl 0xe05

Member info

------------------

Type            LTL

---------------------------------

PHY_PORT       Eth4/1

FLOOD_W_FPOE   0xc031

The output shows that the SRC_INDEX of 0xe05 maps to port Eth4/1. This confirms that the frame is received on port Eth4/1.

With the RBUS data, you can verify that the frame is routed to VLAN 55. Notice that the TTL starts as 0xff in the DBUS data. Additionally, you can confirm the egress port from the DEST_INDEX (destination LTL):

N7K# show system internal pixm info ltl 0x9ed

Member info

------------------

Type            LTL

---------------------------------

PHY_PORT       Eth3/5

FLOOD_W_FPOE   0x8017

FLOOD_W_FPOE   0x8016

The output shows that the DEST_INDEX of 0x9ed maps to port Eth3/5. This confirms that the frame is sent from port Eth3/5.

Additional Verification

In order verify how the switch allocates the LTL pool, enter the show system internal pixm info ltl-region command. The output from this command is useful in order to understand the purpose of an LTL if it is not matched to a physical port. A good example of this is a Drop LTL:

N7700# show system internal pixm info ltl 0xcad

0x0cad is Drop DI LTL

N7700# show system internal pixm info ltl-region

(some output omitted)

===========================================================

PIXM VDC 1 LTL MAP Version: 3

Description: LTL Map for Crossbow

===========================================================

LTL_TYPE                                           SIZE    START   END

========================================================================

LIBLTLMAP_LTL_TYPE_PHY_PORT                        3072    0x0     0xbff

LIBLTLMAP_LTL_TYPE_SUP_ETH_INBAND                  64      0xc00   0xc3f

LIBLTLMAP_LTL_TYPE_UCAST_VPC_VDC_SI                32      0xc40   0xc5f

LIBLTLMAP_LTL_TYPE_EXCEPTION_SPAN                  32      0xc60   0xc7f

LIBLTLMAP_LTL_TYPE_UCAST_GENERIC                   48      0xc80   0xcaf

-------------------------------------------------------------------

SUB-TYPE                                           LTL

-------------------------------------------------------------------

LIBLTLMAP_LTL_TYPE_UCAST_GENERIC_NOT_USED          0xcaf

LIBLTLMAP_LTL_TYPE_DROP_DI_WO_HW_BITSET            0xcae

LIBLTLMAP_LTL_TYPE_DROP_DI                         0xcad

LIBLTLMAP_LTL_TYPE_SUP_DIAG_SI_V5                  0xcac

LIBLTLMAP_LTL_TYPE_RESERVED_ERSPAN_LTL             0xcab

-------------------------------------------------------------------

LIBLTLMAP_LTL_TYPE_LC_CPU                          192     0xcb0   0xd6f

LIBLTLMAP_LTL_TYPE_UCAST_RESERVED                  144     0xd70   0xdff

LIBLTLMAP_LTL_TYPE_PC                              1536    0xe00   0x13ff

LIBLTLMAP_LTL_TYPE_DYNAMIC_UCAST                   5120    0x1400  0x27ff

LIBLTLMAP_LTL_TYPE_MCAST_RESERVED                  48      0x2800  0x282f

LIBLTLMAP_LTL_TYPE_DYNAMIC_MCAST                   38848   0x2830  0xbfef

LIBLTLMAP_LTL_TYPE_SAC_FLOOD                       16      0xbff0  0xbfff

LIBLTLMAP_LTL_TYPE_FLOOD_WITH_FPOE                 16384   0xc000  0xffff