THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Revision | Publish Date | Comments |
---|---|---|
1.0 |
15-Apr-21 |
Initial Release |
1.1 |
10-Jun-21 |
Updated the Products Affected and Problem Symptom Sections |
Affected Product ID | Comments |
---|---|
DS-C9148T-24IK9 |
|
DS-C9148T-24PITK9 |
|
DS-C9148T-48PITK9 |
|
DS-C9148T-24PITK9 |
|
DS-C9148T-24IK9 |
|
DS-C9148T-48PITK9 |
|
DS-C9148T-48PETK9 |
|
DS-C9148T-24EK9 |
|
DS-C9148T-24PETK9 |
|
DS-C9148T-24EK9 |
|
DS-C9148T-24PETK9 |
|
DS-C9148T-48PETK9 |
|
DS-C9148T-24PETK9 |
|
DS-C9148T-24EK9 |
|
DS-C9148T-48PETK9 |
|
DS-C9148T-48PITK9 |
|
DS-C9148T-24IK9 |
|
DS-C9148T-24PITK9 |
|
DS-C9148T-K9 |
|
DS-C9396T-48ETK9 |
|
DS-C9396T-48EK9 |
|
DS-C9396T-96ETK9 |
|
DS-C9396T-K9 |
|
DS-C9396T-96ETK9 |
|
DS-C9396T-48EK9 |
|
DS-C9396T-48ETK9 |
|
DS-C9396T-96ITK9 |
|
DS-C9396T-48IK9 |
|
DS-C9396T-48ITK9 |
|
DS-C9396T-48IK9 |
|
DS-C9396T-48ITK9 |
|
DS-C9396T-96ITK9 |
|
DS-C9396T-48ITK9 |
|
DS-C9396T-48IK9 |
|
DS-C9396T-96ITK9 |
Defect ID | Headline |
---|---|
CSCvx52194 | MDS 9396T, 9148T reloads during ISSU or internal switching errors or some ports fail loop back tests |
A Multilayer Director Switch (MDS) updates the supervisor BIOS automatically as part of an NX-OS In Service Software Upgrade (ISSU) and, under limited circumstances, as part of an In Service Software Downgrade (ISSD).
The MDS 9148T (DS-C9148T) and MDS 9396T (DS-C9396T) Fibre Channel fabric switches do not operate correctly after a nondisruptive upgrade to certain BIOS versions. Any one of these symptoms will occur:
This issue is due to installation of BIOS code bundled in NX-OS 8.4(2b) and 8.5(1) software releases. The versions of BIOS in these two releases are not ISSU-compatible with earlier versions. The process of nondisruptively updating the supervisor CPU BIOS to these versions causes the CPU to enter an indeterminate state. The behavior of the CPU in this state is unpredictable and one of the symptoms described in the Problem Symptom section will occur.
This table compares the working and ISSU-incompatible BIOS versions.
Working BIOS Version | ISSU-Incompatible BIOS Version |
---|---|
0.01 to 0.16 | 0.17 to 1.00 |
This table shows which supervisor CPU BIOS version is bundled in each NX-OS version.
NX-OS Version | Supervisor CPU BIOS Version |
---|---|
8.3(1) | 0.12 |
8.3(2) | 0.12 |
8.4(1) | 0.15 |
8.4(1a) | 0.15 |
8.4(2) | 0.16 |
8.4(2a) | 0.16 |
8.4(2b) | 0.18 |
8.5(1) | 0.19 |
The issue affects the switch only once – the first time the BIOS is updated to an ISSU-incompatible version by an ISSU. The first ISSU with these BIOS versions to NX-OS versions will trigger this issue.
Current BIOS Version | Issue Triggered by ISSU/D to |
---|---|
0.01 to 0.16 | 8.4(2b), 8.5(1) |
A switch might not run the bundled BIOS version when it is shipped from the factory. Always check the current BIOS version as described in the How to Identify Affected Products section.
Any one of these symptoms might occur when an ISSU/ISSD is executed from the working version of the BIOS to the ISSU-incompatible version:
show version
command display shows a switch reset by “f32xbar_usd hap reset”:
Last reset at 364747 usecs after Sun Feb 21 01:51:59 2021 Reason: Reset triggered due to HA policy of Reset System version: 8.4(2b) Service: f32xbar_usd hap reset
show version
command display shows a switch reset by “f32mac hap reset”:
Last reset at 767868 usecs after Sun Feb 21 02:15:19 2021 Reason: Reset triggered due to HA policy of Reset System version: 8.4(2b) Service: f32mac hap reset
show logging nvram
:
%MODULE-2-MOD_DIAG_FAIL: hardware is not accessible in device DEV_TOR_ARB
This is a fatal error which leads to a switch reset. The last reset information in the show version
command output shows a reset caused by a “lc_port_mgr hap reset”:
Last reset at 547800 usecs after Sun Feb 21 02:12:21 2021 Reason: Reset triggered due to HA policy of Reset System version: 8.4(2b) Service: lc_port_mgr hap reset
show hardware internal errors
command:
86 egr EPR2: F32_EPR2_CNT_BAD_SF_OUT 0000000000000142 1-16 - 489 egr EPR2: F32_EPR2_CNT_BAD_PKT_OUT 0000000000000142 1-16 - 492 egr EPR2: F32_EPR2_CNT_BAD_PKT_DROP 0000000000000163 1-16 - 494 egr EPR2: F32_EPR2_CNT_INVALID_KEY_OUT 0000000000000163 1-16 - 502 egr EPR2: F32_EPR2_CNT_OTHER_DROP 0000000000000163 1-16 - 514 egr EPR3: F32_EPR3_CNT_ERR_FI 0000000000000259 1-16 - 516 egr EPR3: F32_EPR3_CNT_ERR_CRC_ERR 0000000000000322 1-16 - 517 egr EPR3: F32_EPR3_CNT_ERR_CSUM_ERR 0000000000000169 1-16 - 518 egr EPR3: F32_EPR3_CNT_ERR_PSR 0000000000000164 1-16 –
show logging logfile
command:
%SYSTEMHEALTH-2-OHMS_MOD_PORT_LB_TEST_FAILED: Module 1 Port 49 has failed loop back tests.
Note: Do not confuse this issue with these three defects. These defects have very similar symptoms, but different causes and fixes. The issue described in this field notice is only caused by an upgrade to a BIOS to the identified versions and only on the identified platforms. If these symptoms are seen in circumstances other than described in this field notice, then refer to these defects for the solution.
For Switches That Have Been Affected
Switches that have been affected by this issue must be cold booted with the reload
CLI command or power-cycled. This will restore the CPU to a normal state. It needs to be completed only once after the BIOS upgrade.
If the switch has already been cold booted or power-cycled after the ISSU for any reason (which includes symptoms 1, 2, or 3 in the Problem Symptom section), then it is not necessary to repeat this step. The issue will not reoccur after this.
A nondisruptive reload, such as another ISSU, will not resolve this issue.
For Switches That Have Not Been Affected
New versions of NX-OS will address this issue and allow software updates for these platforms to be nondisruptive. The first such version is Cisco NX-OS Release 8.4(2c) which is due for release in the second calendar quarter of 2021.
Switches that have not been upgraded to an ISSU-incompatible BIOS version should not be nondisruptively upgraded to NX-OS Release 8.4(2b) or 8.5(1). An ISSU/ISSD to other versions of NX-OS as listed in the Background section will be nondisruptive since BIOS will not be updated to an ISSU-incompatible version.
If it is unavoidable to upgrade to an ISSU-incompatible BIOS version of Cisco NX-OS (such as for specific code fixes) it must be done disruptively, as follows:
configure
command.no boot system
command.boot kickstart m9300-s2ek9-kickstart-mz.8.4.2b.bin
command.boot system m9300-s2ek9-mz.8.4.2b.bin
command.copy running-config startup-config
command.reload
command.Note: BIOS is updated automatically only during an ISSU. This procedure will not update the BIOS version as it does not use an ISSU. Due to this, the switch can still be affected by this issue at the next ISSU if the target NX-OS version does not have the fix.
The switch will function normally after it has been cold booted, even with an ISSU-incompatible BIOS version.
In order to determine if a switch will be affected by this issue during an ISSU, verify the currently installed BIOS version with the show version
CLI command. The highlighted lines in this example show that BIOS Version 0.19 is currently installed:
Software
BIOS: version 0.19.0
In order to determine if a switch has been affected by an ISSU, verify the BIOS version history with the show logging onboard boot-uptime
CLI command. An entry is made with both the primary and secondary BIOS versions each time the switch boots. The highlighted lines in this example show that the switch booted at 16:23:50 with BIOS Version 0.15 and was upgraded by an ISSU to Version 0.19 at 14:16:46:
Wed Mar 17 14:16:46 2021: Boot Record ------------------------------------------------------- Boot Time..........: Wed Mar 17 14:16:46 2021 Slot Number........: 1 Serial Number......: JAE2208.... Bios Version.......: v0.19.0(09/29/2020) Alt Bios Version...: v0.19.0(09/29/2020) Firmware Version...: 8.5(1) [build 8.5(1)] Wed Mar 10 16:23:50 2021: Boot Record ------------------------------------------------------- Boot Time..........: Wed Mar 10 16:23:50 2021 Slot Number........: 1 Serial Number......: JAE2208.... Bios Version.......: v0.15.0(09/29/2020) Alt Bios Version...: v0.15.0(09/29/2020) Firmware Version...: 8.4(1a)) [build 8.4(1a)]
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.
Unleash the Power of TAC's Virtual Assistance