Field Notice: FN72223 - MDS 9148S, MDS 9396S and MDS 9250i Switches That Run on NX-OS Releases 8.4(2), 8.4(2a), 8.4(2b), 9.2(1) or 9.2(2) Might Spontaneously Reload - Software Upgrade Recommended
Available Languages
Notice
THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Products Affected
| Affected Product Name | Description | Comments |
|---|---|---|
| DS-C9148S-12PK9 | MDS 9148S 16G FC switch, w/ 12 active ports | |
| DS-C9148S-48PK9 | MDS 9148S 16G FC switch, w/ 48 active ports | |
| DS-C9148S-CD12P8K9 | MDS 9148S 16G FC switch, w/ 12 active ports + 8G SW SFPs | |
| DS-C9148S-CD12PSK9 | MDS 9148S 16G FC switch, w/ 12 active ports + 16G SW SFPs | |
| DS-C9148S-D12P8K9 | MDS 9148S 16G FC switch, w/ 12 active ports + 8G SW SFPs | |
| DS-C9148S-D12PSK9 | MDS 9148S 16G FC switch, w/ 12 active ports + 16G SW SFPs | |
| DS-C9148S-D12S-ENT | MDS 9148S w/ 12 active ports + 16G SFP + Ent License | |
| DS-C9148S-D24P8K9 | MDS 9148S 16G FC switch, w/ 24 active ports + 8G SW SFPs | |
| DS-C9148S-D24PSK9 | MDS 9148S 16G FC switch, w/ 24 active ports + 16G SW SFPs | |
| DS-C9148S-D48P8K9 | MDS 9148S 16G FC switch, w/ 48 active ports + 8G SW SFPs | |
| DS-C9148S-D48PSK9 | MDS 9148S 16G FC switch, w/ 48 active ports + 16G SW SFPs | |
| DS-C9148S-D8P8K9 | MDS 9148S 16G FC switch, w/ 8 active ports + 8G SW SFPs | |
| DS-C9148S-K9 | MDS 9148S 16G FC 48-port switch (w/ 12 ports active) | |
| DS-C9250I-22PK9 | MDS 9250i 20-port FC, 2-port FCIP/iSCSI | |
| DS-C9250I-D12S-ENT | MDS 9250i w/ 12 active ports + 16G SFP + Ent License | |
| DS-C9250I-K9 | MDS 9250i 50 port switch base config(20xFC, 8xFCoE, 2xFCIP) | |
| DS-C9250I-K9++ | MDS 9250i 50 port switch base config(20xFC, 8xFCoE, 2xFCIP) | |
| DS-C9250ID16GSFPK9 | MDS 9250i 50 port switch base config bundle,20x16G FC optics | |
| DS-C9250ID8GSFPK9 | MDS 9250i 50 port switch base config bundle,20x8G FC optics | |
| DS-C9250IH-K9 | MDS 9250i 50 port switch HP config(20xFC, 8xFCoE, 2xFCIP) | |
| DS-C9396S-48E8K9 | ^MDS 9396S, w/ 48 active ports + 8G SFPs (port-side exhaust) | |
| DS-C9396S-48EK9 | ^MDS 9396S switch, w/ 48 active ports (port-side exhaust) | |
| DS-C9396S-48ESK9 | ^MDS 9396S, w/ 48 active ports + 16G SFPs (port-side exhaust) | |
| DS-C9396S-48IK9 | ^MDS 9396S switch, w/ 48 active ports (port-side intake) | |
| DS-C9396S-96E8K9 | ^MDS 9396S, w/ 96 active ports + 8G SFPs (port-side exhaust) | |
| DS-C9396S-96EK9 | ^MDS 9396S switch, w/ 96 active ports (port-side exhaust) | |
| DS-C9396S-96ESK9 | ^MDS 9396S, w/ 96 active ports + 16G SFPs (port-side exhaust) | |
| DS-C9396S-96IK9 | ^MDS 9396S switch, w/ 96 active ports (port-side intake) | |
| DS-C9396S-K9 | ^MDS 9396S HW base (48 ports active) | |
| UCS-EP-MDS9148S-1 | MDS 9148S 16G FC switch, w/ 12 active ports + 8G SW SFPs | |
| UCS-EP-MDS9148S-16 | MDS 9148S 16G FC switch, w/ 12 active ports + 16G SW SFPs | |
| UCS-EP-MDS9396S-16 | ^Cisco MDS 9396S 16G Switch; 48 enabled ports, 48x16G SW SFP |
Defect Information
| Defect ID | Headline |
| CSCvw91665 | MDS crashes with "Anon_Resident_Mem 0 KB being killed due to lack of memory" |
Problem Description
A small memory leak occurs at each log in to a Multilayer Data Switch (MDS) supervisor that runs Cisco MDS NX-OS Releases 8.4(2), 8.4(2a), 8.4(2b), 9.2(1), or 9.2(2) only. The supervisor will reboot when the memory pool is exhausted.
Only MDS 9148S and MDS 9250i platforms have been reported to reach complete memory exhaustion and reboot. These platforms are single supervisor switches and the supervisor reboot causes the whole switch to disruptively reload.
Background
This issue is caused by a small leak in one of the memory pools on the supervisor each time a new process is started. Most services in Cisco NX-OS (such as Fibre Channel Name Server (FCNS), SNMP, and Zone) are started at boot time and run continuously. The most common reason processes are started during switch operation is when a log in occurs. This creates several processes, which triggers this memory leak.
MDS 9148S has the smallest memory pool. An idle MDS 9148S (with only irregular interactive log ins used for occasional switch maintenance) might run for one to two years without rebooting. However, a switch with frequent log ins per hour will be significantly affected. This can occur when a monitoring platform or script automatically logs in to the switch at high frequency. This can lead to exhaustion in as little as two weeks for an MDS 9148S.
Although reboots due to this issue have been reported from the field only for MDS 9148S and MDS 9250i switches, all MDS platforms are subject to this memory leak and theoretically could eventually reboot. Practically, this is not likely since the rate of log ins is limited by NX-OS and the memory pool is many times bigger on the other platforms. This has been verified by lab testing.
Problem Symptom
The switch logs the "Proc ... being killed due to lack of memory" syslog and subsequently reloads. These syslogs from the show logging nvram CLI command can be used to identify this issue:
%KERN-0-SYSTEM_MSG: [18590637.381749] LR = 0xb7c78908 - kernel
%SYSMGR-2-SERVICE_CRASHED: Service "snmpd" (PID 3609) hasn't caught signal 9 (no core).
%KERN-1-SYSTEM_MSG: [18590637.520017] Proc swapper/0 (0) with Total_VM 0 KB Resident_Mem 0 KB Anon_Resident_Mem 0 KB being killed due to lack of memory - kernel
Note: The name of the crashed service and PID ("snmpd" and "PID 3609" in the previous example) are random and not important.
In order to identify if a system is affected, enter the show system internal kernel meminfo CLI command to display the current memory usage.
switch# show system internal kernel meminfo
MemTotal: 4088568 kB
MemFree: 2732056 kB
MemAvailable: 2834504 kB
Buffers: 6136 kB
Cached: 517872 kB
SwapCached: 0 kB
Active: 654628 kB
Inactive: 250816 kB
Active(anon): 542076 kB
Inactive(anon): 250124 kB
Active(file): 112552 kB
Inactive(file): 692 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 3604476 kB
HighFree: 2384812 kB
LowTotal: 484092 kB
LowFree: 347244 kB
...
On MDS 9148S, MDS 9250i, and MDS 9396S platforms, monitor the ratio of the "LowFree" field to the "LowTotal" field. On all other MDS platforms, monitor the "MemFree" and "MemTotal" fields.
Once the "LowFree" or "MemFree" becomes severely depleted (that is, free memory becomes less than 20% of the total memory), recovery action should be taken. This must be done before the free memory reaches 0% in order to prevent an unscheduled, disruptive reload.
Workaround/Solution
Upgrade the switch software to a fixed version to clear any current symptoms and prevent this issue from recurring.
Note: Some platforms are affected by more than one Field Notice. Refer to the table below to determine the minimum version of Cisco MDS NX-OS to address each individual Field Notice.
|
Field Notice: |
|||
|
Summary: |
Memory leak reload |
FCNS issue after ISSU |
Machine check reload |
|
Affected MDS Platforms: |
9148S 9250i 9396S |
9132T 9148S 9148T 9220i 9396S 9396T |
9396S 9700 32 Gbps FC module |
|
Cisco MDS NX-OS train |
Fixed Cisco MDS NX-OS Versions |
||
|
8.4 train: |
8.4(2c) and later |
9220i: None 1 others: 8.4(2d) and later |
8.4(2d) and later |
|
8.5 train: |
8.5(1) |
None |
8.5(1) |
|
9.2 train: |
9.2(1a) |
9.2(1a) and 9.2(2) |
None |
|
9.3 train: |
9.3(1) and later |
9.3(1) and later |
9.3(1) and later |
1 Cisco MDS 9220i is not supported in the 8.4 train.
Refer to the table below to determine the minimum version of Cisco MDS NX-OS to address FN72223, FN72237, and FN72346 by platform.
|
Minimum Cisco MDS NX-OS Version to Resolve FN72223, FN72237, and FN72346 |
||||||||
|
Cisco MDS NX-OS train |
9700 32 Gbps FC module |
9396T |
9148T |
9132T |
9396S |
9148S |
9250i |
9220i |
|
8.4 train: |
8.4(2d) |
8.4(2d) |
8.4(2d) |
8.4(2d) |
8.4(2d) |
8.4(2d) |
8.4(2c) |
None 1 |
|
8.5 train: |
8.5(1) |
None |
None |
None |
None |
None |
8.5(1) |
None |
|
9.2 train: |
None |
9.2(1a) and 9.2(2) |
9.2(1a) and 9.2(2) |
9.2(1a) and 9.2(2) |
None |
9.2(1a) |
9.2(1a) |
9.2(1a) and 9.2(2) |
|
9.3 train: |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
9.3(1) |
1 Cisco MDS 9220i is not supported in the 8.4 train.
In order to resolve this issue, perform an In Service Software Upgrade (ISSU) or In Service Software Downgrade (ISSD) to a fixed release of Cisco NX-OS. This can be any release of Cisco NX-OS except these affected releases: 8.4(2), 8.4(2a), 8.4(2b), 9.2(1), or 9.2(2). See MDS Software Downloads for a list of software downloads.
In order to temporarily recover from this situation on MDS Fabric switches, proactively execute a nondisruptive reload with the reload system non-disruptive CLI command. On MDS Director switches, perform a nondisruptive supervisor switchover with the system switchover CLI command. Alternatively, an ISSU (or ISSD) to another release of NX-OS on all platform types will also reclaim the leaked memory.
How to Identify Affected Products
In order to verify which release currently runs on Cisco NX-OS, enter the show version CLI command. The relevant release is shown in the "system" field:
switch# show version
...
Software
BIOS: version 2.1.17
loader: version N/A
kickstart: version 8.4(2)
system: version 8.4(2)Revision History
| Version | Description | Section | Date |
| 1.6 | Updated the release numbers for the 9.2 train. | Workaround/Solution | 2023-OCT-06 |
| 1.5 | Updated the Products Affected Section | — | 2022-JUL-07 |
| 1.4 | Updated the Workaround/Solution Section | — | 2022-JUN-14 |
| 1.3 | Updated the Problem Description and Workaround/Solution Sections | — | 2022-MAY-20 |
| 1.2 | Updated the Background Section | — | 2021-SEP-02 |
| 1.1 | Updated the Workaround/Solution Section | — | 2021-AUG-10 |
| 1.0 | Initial Release | — | 2021-AUG-05 |
For More Information
For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:
Receive Email Notification About New Field Notices
To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications
Feedback
Unleash the Power of TAC's Virtual Assistance