THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Revision | Publish Date | Comments |
---|---|---|
1.1 |
01-Sep-22 |
Updated the Problem Description, Problem Symptom, and Workaround/Solution Sections |
1.0 |
20-Jul-22 |
Initial Release |
Affected OS Type | Affected Software Product | Affected Release | Affected Release Number | Comments |
---|---|---|---|---|
NON-IOS |
NX-OS System Software |
9 |
9.2(1), 9.2(2), 9.2(3), 9.3(1), 9.3(2), 9.3(3), 9.3(4), 9.3(5), 9.3(6), 9.3(7), 9.3(7a), 9.3(8) |
|
NON-IOS |
NX-OS System Software |
10 |
10.1(1), 10.1(2), 10.2(1), 10.2(2) |
Defect ID | Headline |
---|---|
CSCvz65993 | tahoe0 brought down resulting in inband connectivity failure |
Nexus 9000 switches might experience traffic disruptions and control plane instability when running release 9.2(1) through 9.3(8) or 10.1(1) through 10.2(2).
In affected releases later than Release 9.3(1), the device might undergo a kernel panic due to a L2ACLRedirect or Rewrite-engine GOLD test failure.
A known fix for Cisco bug ID CSCvz65993 is available in the software releases identified in this software field notice. Cisco recommends that you upgrade to the fixed version or later.
The impact seen for Nexus 3000 and Nexus 9000 devices that experience this issue is detailed in this section.
Cloudscale Nexus 9200/9300 - Suffixes EX, FX, FX2, FX3, and GX
These syslogs will be seen on the device when the issue occurs:
%DIAG_PORT_LB-2-L2ACLREDIRECT_LOOPBACK_TEST_INTERMEDIATE_FAIL: Warning! Module:1 Test:L2ACLRedirect Loopback failed 5 consecutive times. Upon reaching consecutive failure threshold corrective action would be taken.
%DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test <L2ACLRedirect> has been disabled as a part of default EEM action
%DIAG_PORT_LB-2-L2ACLREDIRECT_LOOPBACK_TEST_FAIL: Module:1 Test:L2ACLRedirect Loopback failed 10 consecutive times. Faulty module: Error:Loopback test failed. Packets lost on the SUP in the receive direction
In order to recover the device from this condition, a manual reload of the device must be performed.
When this issue occurs in affected releases after Release 9.3(1), control plane traffic will be impacted until the unit undergoes a diagnostic-triggered kernel panic. When this occurs, the previous syslogs and the next syslog will be seen on the device:
%KERN-1-SYSTEM_MSG: [19791125.834518] BUG: unable to handle kernel NULL pointer dereference at (null) - kernel
These reset reasons will be seen on the device:
----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) ---
At 503414 usecs after Tue Jan 1 00:00:00 2022 Reason: Kernel Panic Service: Version: 9.3(6) At 126444 usecs after Tue Jun 1 00:00:00 2022 Reason: Reset due to critical failure detected by Online diagnostic Service: GOLD Packet Test Failure Version:
In affected releases earlier than Release 9.3(1), devices will encounter indefinite control plane instability and will not kernel panic. In order to avoid this, configure this Embedded Event Manager (EEM) to do a forced reload. The EEM applet is only a workaround to avoid the indefinite control plane instability and will not resolve this issue permanently. For a permanent solution, an upgrade to the fixed version shown in the Workaround/Solution section is the only option.
event manager applet gold_l2acl override __L2ACLRedirect action 1 syslog priority emergencies msg L2ACL_test_failed_reloading action 2 reload force
Nexus 9500 - All Generations of LCs
When this issue occurs on Nexus 9500s, control plane instability will be seen on the device. The internal management channels (EOBC/EPC) will be impacted which might result in modules within the chassis being shut off.
In affected releases, kernel panic might occur on the active supervisor which results in supervisor switchover. This syslog will be seen on the device when the issue occurs:
%PLATFORM-2-MOD_ALL_PWRDN_NOSC: All fabric, line-card modules powered down due to non-availabilitySS of system controller modules
If the device does not automatically recover from this condition, a manual restart of the device should be performed.
Nexus 3000 R-Series
When this issue occurs on Nexus 3000 R-Series, these syslogs will be seen on the device:
%DIAG_PORT_LB-2-L2ACLREDIRECT_LOOPBACK_TEST_INTERMEDIATE_FAIL: Warning! Module:1 Test:L2ACLRedirect Loopback failed 5 consecutive times. Upon reaching consecutive failure threshold corrective action would be taken.
%DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test <L2ACLRedirect> has been disabled as a part of default EEM action
%DIAG_PORT_LB-2-L2ACLREDIRECT_LOOPBACK_TEST_FAIL: Module:1 Test:L2ACLRedirect Loopback failed 10 consecutive times. Faulty module: Error:Loopback test failed. Packets lost on the SUP in the receive direction
Indefinite control plane impact might be seen on devices that run the impacted versions. This EEM can be configured to force a reload. The EEM applet is only a workaround to avoid the indefinite control plane instability and will not resolve this issue permanently. For a permanent solution, an upgrade to the fixed version shown in the Workaround/Solution section is the only option.
event manager applet gold_l2acl override __L2ACLRedirect action 1 syslog priority emergencies msg L2ACL_test_failed_reloading action 2 reload force
Nexus 3100 and First-Generation Merchant-Silicon 9300
When this issue occurs on Nexus 3100 Series and first-generation merchant-silicon Nexus 9300 on affected releases, these syslogs will be seen on the device:
%DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test <RewriteEngineLoopback> has been disabled as a part of default EEM action
%DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:1 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module: Error:Loopback test failed. Packets lost on the SUP in the receive direction
When this issue occurs in affected releases after Release 9.3(1), control plane traffic will be impacted until the unit undergoes a diagnostic triggered kernel panic. When this occurs, the previous syslogs plus the next syslog will be seen on the device:
%KERN-1-SYSTEM_MSG: [24014796.045164] BUG: unable to handle kernel NULL pointer dereference at (null) - kernel
In affected releases before Release 9.3(1), devices will encounter indefinite control plane instability and will not kernel panic. In order to avoid this, configure this EEM to do a forced reload. The EEM applet is only a workaround to avoid the indefinite control plane instability and will not resolve this issue permanently. For a permanent solution, an upgrade to the fixed version shown in the Workaround/Solution section is the only option.
Note: This EEM will work only for Nexus 3100 and first-generation Nexus 9000.
event manager applet goldrewriteacl override __RewriteEngineLoopback action 1 syslog priority emergencies msg Rewrite_engine_failed_reloading action 2 reload module 1 force
Nexus 3000
When this issue occurs on Nexus 3000 Series on affected releases, these syslogs will be seen on the device:
%DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test <RewriteEngineLoopback> has been disabled as a part of default EEM action
%DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:1 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module: Error:Loopback test failed. Packets lost on the SUP in the receive direction
When this issue occurs in affected releases, control plane traffic will be impacted until the unit is recovered by a manual reload.
Customers should upgrade impacted Product Identifiers (PIDs) to Release 9.3(9) or Release 10.2(3) in order to address this condition.
PID | Release |
---|---|
N3K-C3048 | 9.3(9) |
N3K-C3064 | 9.3(9) |
N3K-C3132C-Z | 9.3(9) |
N3K-C3132Q | 9.3(9) |
N3K-C3164Q | 9.3(9) |
N3K-C3172PQ | 9.3(9) |
N3K-C3172TQ | 9.3(9) |
N3K-C3172PQ-XL | 9.3(9) |
N3K-C3172TQ-XL | 9.3(9) |
N3K-C3232C | 9.3(9) |
N3K-C3264C-E | 9.3(9) |
N3K-C3264 | 9.3(9) |
N3K-C3524 | 9.3(9) |
N3K-C3524-X | 9.3(9) |
N3K-C3524-XL | 9.3(9) or 10.2(3) |
N3K-C3548 | 9.3(9) |
N3K-C3548-XL | 9.3(9) of 10.2(3) |
N3K-C31108PC-V | 9.3(9) |
N3K-C31108TC-V | 9.3(9) |
N3K-C31128PQ | 9.3(9) |
N3K-C36180YC-R | 9.3(9) or 10.2(3) |
N3K-C3636-R | 9.3(9) or 10.2(3) |
N9K-C9372PX | 9.3(9) |
N9K-C9372PX-E | 9.3(9) |
N9K-C9372TX | 9.3(9) |
N9K-C9372TX-E | 9.3(9) |
N9K-C9332PQ | 9.3(9) |
N9K-C93120TX | 9.3(9) |
N9K-C9396PX | 9.3(9) |
N9K-C9396TX | 9.3(9) |
N9K-C93120TX | 9.3(9) |
N9K-C93128TX | 9.3(9) |
N9K-C92160YC-X | 9.3(9) |
N9K-C9236C | 9.3(9) |
N9K-C92300YC | 9.3(9) |
N9K-C92304QC | 9.3(9) |
N9K-C9272Q | 9.3(9) |
N9K-C92348GC-X | 9.3(9) or 10.2(3) |
N9K-C93180YC-EX-24 | 9.3(9) or 10.2(3) |
N9K-C93180YC-EX | 9.3(9) or 10.2(3) |
N9K-C93180LC-EX | 9.3(9) or 10.2(3) |
N9K-C93108TC-EX | 9.3(9) or 10.2(3) |
N9K-C93108TC-EX-24 | 9.3(9) or 10.2(3) |
N9K-C93108TC-FX | 9.3(9) or 10.2(3) |
N9K-C93108TC-FX-24 | 9.3(9) or 10.2(3) |
N9K-C93180YC-FX | 9.3(9) or 10.2(3) |
N9K-C93180YC-FX-24 | 9.3(9) or 10.2(3) |
N9K-C9348GC-FXP | 9.3(9) or 10.2(3) |
N9K-C9364C | 9.3(9) or 10.2(3) |
N9K-C9332C | 9.3(9) or 10.2(3) |
N9K-C9336C-FX2 | 9.3(9) or 10.2(3) |
N9K-C93240YC-FX2 | 9.3(9) or 10.2(3) |
N9K-C93360YC-FX2 | 9.3(9) or 10.2(3) |
N9K-C93216TC-FX2 | 9.3(9) or 10.2(3) |
N9K-C9336C-FX2-E | 10.2(3) |
N9K-C93180YC-FX3 | 9.3(9) or 10.2(3) |
N9K-C93180YC-FX3S | 9.3(9) or 10.2(3) |
N9K-C93108TC-FX3P | 9.3(9) or 10.2(3) |
N9K-C93600CD-GX | 9.3(9) or 10.2(3) |
N9K-C9364C-GX | 9.3(9) or 10.2(3) |
N9K-C9316D-GX | 9.3(9) or 10.2(3) |
N9K-C9504-FM | 9.3(9) |
N9K-C9508-FM | 9.3(9) |
N9K-C9516-FM | 9.3(9) |
N9K-C9504-FM-E | 9.3(9) |
N9K-C9508-FM-E | 9.3(9) |
N9K-C9516-FM-E | 9.3(9) |
N9K-C9508-FM-E2 | 9.3(9) or 10.2(3) |
N9K-C9516-FM-E2 | 9.3(9) or 10.2(3) |
N9K-C9504-FM-R | 9.3(9) or 10.2(3) |
N9K-C9508-R | 9.3(9) or 10.2(3) |
N9K-C9504-FM-G | 10.2(3) |
N9K-C9508-FM-G | 10.2(3) |
For information on how to upgrade the software, see:
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.
Unleash the Power of TAC's Virtual Assistance