Field Notice: FN74042 - Some Cisco ASR 9000 Series Aggregation Services Router Products May Experience Kernel Panic, Silent Reloading, or Hanging - BIOS/Firmware Upgrade Recommended

Available Languages

Updated:November 6, 2023

Document ID:FN74042

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Affected Product Name	Description	Comments
A99-12X100GE	ASR 9900 12-port 100GE Line Card
A9K-24X10GE-1G-CM	ASR 9000 24-port 10GE & 1GE dual rate Consumption Model LC
A9K-24X10GE-1G-SE	ASR 9000 24-port 10GE & 1GE dual rate SE LC
A9K-24X10GE-1G-TR	ASR 9000 24-port 10GE & 1GE dual rate -TR LC
A9K-48X10GE-1G-CM	ASR 9000 48-port 10G &1G dual rate Consumption model LC
A9K-48X10GE-1G-SE	ASR 9000 48-port 10GE & 1GE dual rate SE LC
A9K-48X10GE-1G-TR	ASR 9000 48-port 10GE & 1GE dual rate -TR LC
A9K-4X100GE	ASR 9000 4-port 100GE LAN Line Card
A9K-RSP880-LT-SE	ASR 9000 Route Switch Processor 880-LT for Service Edge
A9K-RSP880-LT-TR	ASR 9000 Route Switch Processor 880-LT for Packet Trans
ASR-9901	ASR 9901 Compact Chassis, 2RU	ASR-9901-RP ASR-9901-LC

Affected Software Product	Affected Release	Affected Release Number	Comments
IOS XR Software	7	7.8.1, 7.8.2, 7.9.1, 7.9.2

Defect Information

Defect ID	Headline
CSCwf51196	XR-VM Kernel Panic/Silent Reload/Hang due to Intel Microcode Errata BDE38

Problem Description

The Cisco ASR 9000 Series Aggregation Services Router products that are listed in the Products Affected section may experience kernel panic, may experience a silent reload with or without kernel dump being stored on the device, or may become unresponsive or hang if they are running one of the following releases of Cisco IOS XR Software:

7.8.1
7.8.2
7.9.1
7.9.2

Background

Cisco IOS XR Software releases 7.8.1, 7.8.2, 7.9.1, and 7.9.2 load the default BIOS instead of the latest BIOS version, exposing the CPU to BDE38 Errata. As a result, devices might experience kernel panic, may experience a silent reload with or without kernel dump, and may hang or become unresponsive under normal working conditions.

Following table shows the affected BIOS version for each of the affected product identifiers (PIDs):

PID	Impacted Primary-BIOS Version	Non-Impacted Primary-BIOS Version with SMU
A9K-48X10GE-1G-SE/TR/CM A9K-24x10GE-1G-SE/TR/CM	18.34	18.33
A99-12x100GE	9.34	9.33
A9K-4X100GE	9.34	9.33
ASR-9901-RP	22.27	22.26
ASR-9901-LC	23.23	23.22
A9K-RSP880-LT-SE A9K-RSP880-LT-TR	17.41	17.4

Problem Symptom

On the affected products that are experiencing the issue reported in this field notice, kernel core dump files (for example, default-sdr--1.20230716-005550.core.host.0_RSP0.kdump) may be created on the hard disk. If the RP on a Cisco ASR9901 router hangs, the router will be unresponsive until it is power-cycled. If the line card or RP hangs, the card will be in SW_INACTIVE state, as shown in the following example:

RP/0/RSP0/CPU0:xrg-403-asr9910#sh platform
Mon Aug  7 07:39:58.386 UTC
Node              Type                       State             Config state
--------------------------------------------------------------------------------
0/RSP0/CPU0       A99-RSP-SE(Active)         IOS XR RUN        NSHUT
0/RSP1/CPU0       A99-RSP-SE(Standby)        IOS XR RUN        NSHUT
0/FT0             ASR-9910-FAN               OPERATIONAL       NSHUT
0/FT1             ASR-9910-FAN               OPERATIONAL       NSHUT
0/0               A9K-24X10GE-1G-SE          SW_INACTIVE       NSHUT
...

Workaround/Solution

An affected system that experiences kernel panic or a silent reload will recover on its own. If an affected device hangs or is unresponsive, then a power cycle of the affected device, route switch processor (RSP), or line card is required.

Cisco recommends installing the SMUs for the products that are listed in the Products Affected section if they are running an affected software release. After SMU activation, a forced Field Programmable Device (FPD) upgrade is required for the affected device, RSP, or line card.

Upgrade the BIOS of an RSP and a line card with the force keyword, as shown in the following example:

#upgrade hw-module location 0/RSP0 fpd Primary-BIOS force
#upgrade hw-module location 0/0 fpd Primary-BIOS force

Once the FPD is upgraded, a power cycle in admin mode is required only for impacted PIDs (RSP or line card).

admin hw-module location 0/RSP0 reload noprompt
admin hw-module location 0/0 reload noprompt

After SMU activation and FPD force upgrade, use the CLI commands that are provided in the Problem Symptom section to validate the CPU microcode revision.

How to Identify Affected Products

To determine the FPD version, use the show hw-module fpd command. The following example shows the output from an affected ASR 9901 platform:

RP/0/RSP0/CPU0:ASR-9901-A#show hw-module fpd
Mon Aug  7 07:43:35.678 UTC

Auto-upgrade:Enabled
                                                                         FPD Versions
                                                                        ==============
Location   Card type             HWver FPD device       ATR Status    Running   Programd
-----------------------------------------------------------------------------------------
0/RSP0     ASR-9901-RP           1.0   CBC                  CURRENT    54.11     54.11
0/RSP0     ASR-9901-RP           1.0   Drax-FPGA            CURRENT     0.38      0.38
0/RSP0     ASR-9901-RP           1.0   IPU-FPGA             CURRENT     2.05      2.05
0/RSP0     ASR-9901-RP           1.0   IPU-FSBL             CURRENT     1.104     1.104
0/RSP0     ASR-9901-RP           1.0   IPU-Linux            CURRENT     1.104     1.104
0/RSP0     ASR-9901-RP           1.0   Primary-BIOS         CURRENT    22.27     22.27
0/RSP0     ASR-9901-RP           1.0   SSDa-MICRON          N/A         7.05      7.05
0/RSP0     ASR-9901-RP           1.0   SSDb-MICRON          N/A         7.05      7.05
0/0        ASR-9901-LC           1.0   CBC                  CURRENT    55.07     55.07
0/0        ASR-9901-LC           1.0   Gamora-FPGA          CURRENT     0.36      0.36
0/0        ASR-9901-LC           1.0   IPU-FPGA             CURRENT     1.10      1.10
0/0        ASR-9901-LC           1.0   IPU-FSBL             CURRENT     1.104     1.104
0/0        ASR-9901-LC           1.0   IPU-Linux            CURRENT     1.104     1.104
0/0        ASR-9901-LC           1.0   Primary-BIOS         CURRENT    23.23     23.23
0/0        ASR-9901-LC           1.0   SSDa-MICRON          N/A         7.05      7.05
0/PT0      A9K-1600W-AC          0.0   PM0-PO-PriMCU        CURRENT    17.137    17.137
0/PT0      A9K-1600W-AC          0.0   PM1-PO-PriMCU        CURRENT    17.137    17.137
RP/0/RSP0/CPU0:ASR-9901-A#

Revision History

Version	Description	Section	Date
1.0	Initial Release	—	2023-NOV-06

For More Information

For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:

Receive Email Notification About New Field Notices

To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)