Field Notice: FN72237 - Cisco MDS Fabric Switch – Some Hosts Might Lose Connectivity After a Switch Software Update - Software Upgrade Recommended
Available Languages
Notice
THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Products Affected
| Affected Product Name | Description | Comments |
|---|---|---|
| 9148S-12PSK9-NFR | MDS 9148S, 12 active ports, 16G SW SFPs not for resale | |
| DS-C9132T-K9 | MDS 9132T 32G 1 RU FC switch, w/ 8 active FC ports, No Fans | |
| DS-C9148S-12PK9 | MDS 9148S 16G FC switch, w/ 12 active ports | |
| DS-C9148S-12PK9= | MDS 9148S 16G FC switch, w/ 12 active ports | |
| DS-C9148S-48PK9 | MDS 9148S 16G FC switch, w/ 48 active ports | |
| DS-C9148S-48PK9= | MDS 9148S 16G FC switch, w/ 48 active ports | |
| DS-C9148S-CD12P8K9 | MDS 9148S 16G FC switch, w/ 12 active ports + 8G SW SFPs | |
| DS-C9148S-CD12PSK9 | MDS 9148S 16G FC switch, w/ 12 active ports + 16G SW SFPs | |
| DS-C9148S-D12P8K9 | MDS 9148S 16G FC switch, w/ 12 active ports + 8G SW SFPs | |
| DS-C9148S-D12PSK9 | MDS 9148S 16G FC switch, w/ 12 active ports + 16G SW SFPs | |
| DS-C9148S-D12S-ENT | MDS 9148S w/ 12 active ports + 16G SFP + Ent License | |
| DS-C9148S-D24P8K9 | MDS 9148S 16G FC switch, w/ 24 active ports + 8G SW SFPs | |
| DS-C9148S-D24PSK9 | MDS 9148S 16G FC switch, w/ 24 active ports + 16G SW SFPs | |
| DS-C9148S-D48P8K9 | MDS 9148S 16G FC switch, w/ 48 active ports + 8G SW SFPs | |
| DS-C9148S-D48PSK9 | MDS 9148S 16G FC switch, w/ 48 active ports + 16G SW SFPs | |
| DS-C9148S-D8P8K9 | MDS 9148S 16G FC switch, w/ 8 active ports + 8G SW SFPs | |
| DS-C9148S-K9 | MDS 9148S 16G FC 48-port switch (w/ 12 ports active) | |
| DS-C9148S-K9= | MDS 9148S HW base (12 ports active), spare | |
| DS-C9148T-K9 | MDS 9148T 48P 32G Fibre Channel switch w/ 24 Ports activated | |
| DS-C9220I-K9 | MDS 9220I 1RU Multiservice switch, BASE PID | |
| DS-C9396S-48E8K9 | ^MDS 9396S, w/ 48 active ports + 8G SFPs (port-side exhaust) | |
| DS-C9396S-48EK9 | ^MDS 9396S switch, w/ 48 active ports (port-side exhaust) | |
| DS-C9396S-48ESK9 | ^MDS 9396S, w/ 48 active ports + 16G SFPs (port-side exhaust) | |
| DS-C9396S-48IK9 | ^MDS 9396S switch, w/ 48 active ports (port-side intake) | |
| DS-C9396S-96E8K9 | ^MDS 9396S, w/ 96 active ports + 8G SFPs (port-side exhaust) | |
| DS-C9396S-96EK9 | ^MDS 9396S switch, w/ 96 active ports (port-side exhaust) | |
| DS-C9396S-96ESK9 | ^MDS 9396S, w/ 96 active ports + 16G SFPs (port-side exhaust) | |
| DS-C9396S-96IK9 | ^MDS 9396S switch, w/ 96 active ports (port-side intake) | |
| DS-C9396S-K9 | ^MDS 9396S HW base (48 ports active) | |
| DS-C9396S-K9= | ^MDS 9396S HW base (48 ports active) | |
| DS-C9396T-K9 | MDS 9396T 96P 32G Fibre Channel switch w/ 48 activated ports | |
| UCS-EP-MDS9148S-1 | MDS 9148S 16G FC switch, w/ 12 active ports + 8G SW SFPs | |
| UCS-EP-MDS9148S-16 | MDS 9148S 16G FC switch, w/ 12 active ports + 16G SW SFPs | |
| UCS-EP-MDS9396S-16 | ^Cisco MDS 9396S 16G Switch; 48 enabled ports, 48x16G SW SFP |
Defect Information
| Defect ID | Headline |
| CSCvm01925 | Closing FCNS logins during ISSU/D or FCNS restart causes end devices to lose connectivity |
| CSCvz09012 | End devices encounter errors or do not respond after MDS Fabric switch ISSU |
Problem Description
After a nondisruptive software update of a fabric switch via an In Service Software Upgrade/In Service Software Downgrade (ISSU/ISSD), hosts might lose connectivity to storage devices, either temporarily or until manual intervention is performed.
It is difficult to predict which end devices will be affected by these defects, how they will be affected, when they will be affected, or the extent of any disruption.
Background
When end devices connect to the fabric, they establish a connection with the local switch’s Fibre Channel Name Server (FCNS). This connection is used to query the FCNS about other devices that are zoned with this device. Problems with this connection can occur as a result of one of two defects:
CSCvm01925: The FCNS connection is closed by the switch during an ISSU/ISSD. FCNS queries can no longer be made by the end device unless it opens a new connection.
CSCvz09012: The FCNS connection is not restored on the switch side after an ISSU/ISSD. The switch will still reply to small FCNS queries, but also attempt to make a new connection from the FCNS to the end device.
In either case, host or storage devices attached to the switch might not reopen the FCNS connection or might recover disruptively. This could lead to undesired I/O disruption between the device and other devices it is zoned with.
Problem Symptom
Symptoms will vary depending on the end device model and even the software version. These can range from devices flapping their links to the switch to devices permanently setting their links to the errored state.
Storage device ports are typically more affected than host ports.
There are no errors or other symptoms logged on the switch.
Workaround/Solution
Upgrade the switch software to a fixed version to clear any current symptoms and prevent this issue from recurring.
Note: some platforms are affected by more than one Field Notice. Refer to the table below to determine the minimum version of Cisco MDS NX-OS to address each individual Field Notice.
|
Field Notice: |
|||
|
Summary: |
Memory leak reload |
FCNS issue after ISSU |
Machine check reload |
|
Affected MDS Platforms: |
9148S 9250i 9396S |
9132T 9148S 9148T 9220i 9396S 9396T |
9396S 9700 32 Gbps FC module |
|
Cisco MDS NX-OS train |
Fixed Cisco MDS NX-OS Versions |
||
|
8.4 train: |
8.4(2c) and later |
9220i: None 1 others: 8.4(2d) and later |
8.4(2d) and later |
|
8.5 train: |
8.5(1) |
None |
8.5(1) |
|
9.2 train: |
9.2(1a) |
9.2(1a) and 9.2(2) |
None |
|
9.3 train: |
9.3(1) and later |
9.3(1) and later |
9.3(1) and later |
1 Cisco MDS 9220i is not supported in the 8.4 train.
Refer to the table below to determine the minimum version of Cisco MDS NX-OS to address FN72223, FN72237, and FN72346 by platform.
|
Minimum Cisco MDS NX-OS Version to Resolve FN72223, FN72237, and FN72346 |
||||||||
|
Cisco MDS NX-OS train |
9700 32 Gbps FC module |
9396T |
9148T |
9132T |
9396S |
9148S |
9250i |
9220i |
|
8.4 train: |
8.4(2d) |
8.4(2d) |
8.4(2d) |
8.4(2d) |
8.4(2d) |
8.4(2d) |
8.4(2c) |
None 1 |
|
8.5 train: |
8.5(1) |
None |
None |
None |
None |
None |
8.5(1) |
None |
|
9.2 train: |
None |
9.2(1a) and 9.2(2) |
9.2(1a) and 9.2(2) |
9.2(1a) and 9.2(2) |
None |
9.2(1a) |
9.2(1a) |
9.2(2) |
|
9.3 train: |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
1 Cisco MDS 9220i is not supported in the 8.4 train.
If the switch was last powered up with Cisco MDS NX-OS 8.4(1) or later then upgrade nondisruptively to a fixed version of Cisco MDS NX-OS. This can be done using the install all command. This will recover FCNS connections from attached devices and the fixed NX-OS will prevent the issue in the future.
If the switch was booted with a version earlier than Cisco MDS NX-OS 8.4(1) then the upgrade must be done disruptively to avoid any possible related issues that might be difficult to troubleshoot in the future. This will cause the attached devices to build new FCNS connections and the fixed NX-OS will prevent the issue in the future.
In order to disruptively upgrade a switch, follow these steps:
- Download the new kickstart and system images to the switch boot flash.
- In a terminal session, enter these CLI commands:
switch# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
switch(config)# no boot system
switch(config)# boot kickstart new_kickstart_image_name
switch(config)# boot system new_system_image_name
switch(config)# end
switch# copy running-config startup-config
switch# reload
This command will reboot the system. (y/n)? [n] y
- The switch will reboot at this point and be operational again in several minutes.
How to Identify Affected Products
Either of these defects can occur during an ISSU/ISSD. Which defect will apply depends on the NX-OS version running at the start of the ISSU/ISSD. The mapping of affected initial NX-OS release to defect is listed in the following table:
|
NX-OS Version |
Defect ID |
|
|
CSCvm01925 |
CSCvz09012 |
|
|
6.2 train: |
|
|
|
6.2(27) and earlier |
Impacted |
N/A |
|
6.2(29) and later |
Fixed |
Impacted |
|
7.3 train: all versions |
Impacted |
N/A |
|
8.1 train: all versions |
Impacted |
N/A |
|
8.2 train: all versions |
Impacted |
N/A |
|
8.3 train: all versions |
Impacted |
N/A |
|
8.4 train: |
|
|
|
8.4(2c) and earlier |
Fixed |
Impacted |
|
8.4(2d) and later |
Fixed |
Fixed |
|
8.5 train: all versions |
Fixed |
Impacted |
|
9.2 train: |
|
|
|
9.2(1) |
Fixed |
Impacted |
|
9.2(2) |
Fixed |
Fixed |
In order to verify which Cisco NX-OS release currently runs on a switch, enter the show version CLI command. The release is shown in the "system" field:
switch# show version
...
Software
BIOS: version 2.1.17
loader: version N/A
kickstart: version 8.4(2)
system: version 8.4(2)
For More Information
A "train" refers to a major NX-OS release and is a set of Cisco NX-OS incremental minor versions. For example, Cisco MDS NX-OS 8.2 is a train comprised of NX-OS versions 8.2(1) and 8.2(2).
Revision History
| Version | Description | Section | Date |
| 1.2 | Updated the release numbers for the 9.2 train. | Workaround/Solution | 2023-OCT-06 |
| 1.1 | Updated the Workaround/Solution Section | — | 2022-JUN-14 |
| 1.0 | Initial Release | — | 2021-SEP-27 |
For More Information
For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:
Receive Email Notification About New Field Notices
To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications
Feedback
Unleash the Power of TAC's Virtual Assistance