Contents

Introduction

Prerequisites

Requirements

Components Used

Problem

Solution

Introduction

This document describes a problem encountered with Nexus 5010/5020 switches caused by a hardware issue in the Altos ASIC (error message %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag detected major event: Port Failure), and also provides a solution to the problem.

Prerequisites

Requirements

Cisco recommends that you have knowledge of the Nexus CLI.

Components Used

The information in this document is based on Cisco Nexus 5010/5020 switches only. It does not affect Cisco Nexus 5548/5596 switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem

Multiple interfaces on Card 2 are down, and you see this alert:

N5020 %$ VDC-1 %$ %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag detected major event

The alert suggests a card failure, but some ports are up. Even though the Nexus 5020 switch is online, the Fiber Channel (FC) module in Slot 2 is offline. Enter the show module command in order to view the status of the modules:

Mod Ports  Module-Type                      Model                  Status

--- -----  -------------------------------- ---------------------- ------------

1    40     40x10GE/Supervisor               N5K-C5020P-BF-SUP      active *

2    8      8x1/2/4G FC Module               N5K-M1008              offline <<<<<<

Mod  Sw              Hw      World-Wide-Name(s) (WWN)

---  --------------  ------  --------------------------------------------------

1    4.2(1)N2(1)     1.3     --

2    4.2(1)N2(1)     1.0     77:9f:b7:62:2f:6c:69:62 to 00:00:00:b8:27:0a:08:2c

Enter the show environment command in order to view the module environment data.

Mod Model                   Power     Power       Power     Power       Status

Requested Requested   Allocated Allocated

(Watts)   (Amp)       (Watts)   (Amp)

--- ----------------------  -------   ----------  --------- ----------  ----------

1    N5K-C5020P-BF-SUP      625.20    52.10       625.20    52.10       powered-up

2    N5K-M1008              9.96      0.83        9.96      0.83        fail/shutdown

Enter the show logging nvram command in order to view this output:

N5020 %$ VDC-1 %$ %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag detected major event:

Port failure: Ethernet1/1

N5020 %$ VDC-1 %$ last message repeated 2 times

N5020 %$ VDC-1 %$ %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag detected major event:

Port failure: Ethernet1/2

N5020 %$ VDC-1 %$ last message repeated 7 times

N5020 %$ VDC-1 %$ %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag detected major event:

Port failure: Ethernet1/5

N5020 %$ VDC-1 %$ last message repeated 3 times

N5020 %$ VDC-1 %$ %NOHMS-2-NOHMS_DIAG_ERROR: Module 1: Runtime diag detected major event:

Port failure: Ethernet1/13

As you can see from the logs, several ports failed the runtime diagnostics. Also, two ports from every Gatos ASIC report a "Hardware failure" because the fabric is down. Enter the show interface brief command in order to view this output:

--------------------------------------------------------------------------------

Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port

Interface                                                                   Ch #

--------------------------------------------------------------------------------

Eth1/1        1      eth  fabric down    Hardware failure            10G(D) 138

Eth1/2        1      eth  fabric down    Hardware failure            10G(D) 138

Eth1/3        1      eth  fabric up      none                        10G(D) 138

Eth1/4        1      eth  fabric up      none                        10G(D) 138

Eth1/5        1      eth  fabric down    Hardware failure            10G(D) 140

Eth1/6        1      eth  fabric down    Hardware failure            10G(D) 140

Eth1/7        1      eth  fabric up      none                        10G(D) 140

Eth1/8        1      eth  fabric up      none                        10G(D) 140

The Gatos ASIC reports failures for some of the ports and disables them. Enter the show hardware internal gatos event-history error command in order to view this output:

1)  Event:E_DEBUG, length:81, at 775734 usecs after Fri May 24 15:28:10 2013

[101] xcvr_set_port_to_hw_failure(): Sending nohms failure notif for port xgb1/13

2)  Event:E_DEBUG, length:44, at 775726 usecs after Fri May 24 15:28:10 2013[100] CODE-PATH:

xcvr_set_port_to_hw_failure

935)  Event:E_DEBUG, length:34, at 434695 usecs after Fri May 24 15:28:06 2013[100] CODE-PATH:

xcvr_port_disable

936)  Event:E_DEBUG, length:38, at 434653 usecs after Fri May 24 15:28:06 2013[100] CODE-PATH:

xcvr_set_port_disable

937)  Event:E_DEBUG, length:81, at 408233 usecs after Fri May 24 15:28:06 2013

[101] xcvr_set_port_to_hw_failure(): Sending nohms failure notif for port xgb1/30

938) Event:E_DEBUG, length:44, at 408224 usecs after Fri May 24 15:28:06 2013 [100] CODE-PATH:

xcvr_set_port_to_hw_failure

From the Altos ASIC, there are numerous "error interrupt" messages due to synchronization issues that cause Fabric Interconnects (FI) resets. Enter the show hardware internal altos event-history errors command in order to view this output:

1)  Event:E_DEBUG, length:131, at 959201 usecs after Fri May 24 14:19:20 2013

[100] Threshold reached for error interrupt - ALT_FIC3_INT_3_XGXS_rx2_loss_of_sync, flags:

0xa8, fabric port: 15, Action: fi-reset

2)  Event:E_DEBUG, length:122, at 372727 usecs after Fri May 24 14:15:05 2013

[100] Threshold reached for interrupt - ALT_FIC6_INT_0_XGXS_EXT_serdes_rx2_sync, masking it

(threshold=3 period=10 msecs)

453)  Event:E_DEBUG, length:122, at 658189 usecs after Fri May 24 03:38:48 2013

[100] Threshold reached for interrupt - ALT_FIC6_INT_1_XGXS_EXT_serdes_rx0_sync, masking it

(threshold=3 period=10 msecs)

454) Event:E_DEBUG, length:129, at 658137 usecs after Fri May 24 03:38:48 2013

[100] Threshold reached for error interrupt - ALT_FIC6_INT_1_XGXS_rx2_code_eerror, flags:

0xa8, fabric port: 25, Action: fi-reset

Solution

The problem is due to a hardware issue in the Altos ASIC. Contact the Cisco Technical Assistance Center (TAC) in order to replace the Nexus 5000 Series switch.