The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes "SW_INACTIVE" in CIsco IOS® XR and how to troubleshoot it.
There are no specific requirements for this document.
This document is not restricted to specific hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
"SW_INACTIVE" state in Linecards and Router Switch Processor/ Router Processor (RSP/RPs) for Cisco IOS XR platforms, for example, the next output displays the error.
RP/0/RSP0/CPU0:XR#admin show platform
Location Card Type HW State SW State Config State
----------------------------------------------------------------------------
0/1 A9K-MOD400-TR OPERATIONAL OPERATIONAL NSHUT
0/0 A9K-MOD400-TR OPERATIONAL SW_INACTIVE NSHUT >>>
0/RSP0 A9K-RSP880-SE OPERATIONAL OPERATIONAL NSHUT
0/RSP1 A9K-RSP880-SE OPERATIONAL OPERATIONAL NSHUT
0/FT0 ASR-9010-FAN-V2 OPERATIONAL N/A NSHUT
0/FT1 ASR-9010-FAN-V2 OPERATIONAL N/A NSHUT
0/PT0 A9K-DC-PEM-V3 OPERATIONAL N/A NSHUT
0/PT1 A9K-DC-PEM-V3 OPERATIONAL N/A NSHUT
In Cisco IOS XR platforms, 64 bits (also know as eXR), the next set of states compose the Finite State Machine (FSM) for a linecard/RSP/RP to boot.
Said FSM is valid for versions higher than 7.1.1 and other NCS5500 versions.
The states are the next:
The "SW_INACTIVE" (software Inactive) state, means that the software is not ready to be run on the device. The node does not have enough information to continue with the next state and is requesting information to the active RSP/RP to bake the image.
In other words, "SW_INACTIVE" is the state where the node (linecard/RSP/RP) synchronizes with the active (primary) node.
Why does the linecard needs to synchronize with the active node?
The next output shows some of the files that the linecard downloads to start the FSM and boot to IOS_XR_RUN state.
The main files are:
Note: Linecards and Standby Route Processors use the same process checking ISO and all the required information to boot. Peripheral Component Interconnect Express (PCIE) protocol was built to handle point-to-point connections for non-core components. One of the first FPGA that can start the communication and establish control path for the linecards is IOFPGA and MIFPGA. These FPGAs are focused in PCIE communications.
To archive the mentioned client-server model, Cisco IOS XR needs to:
0/RSP1/ADMIN0:canbus_driver[3903]: %PLATFORM-CANB_SERVER-7-CBC_POST_RESET_NOTIFICATION : Node 0/1 CBC-0, reset reason CPU_RESET_POR (0x05000000)
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_OFF, event_reason_str 'HW Event Powered OFF' for card 0/1
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_ON, event_reason_str 'HW Event Powered ON' for card 0/1
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_OK, event_reason_str 'HW Event OK' for card 0/1
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/1 hardware state going to Operational
0/RSP1/ADMIN0:canbus_driver[3903]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/1 CBC-0, reset reason CPU_RESET_POR (0x05000000)
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_RESET, event_reason_str 'HW Event RESET' for card 0/1
0/RSP1/ADMIN0:canbus_driver[3903]: %PLATFORM-CANB_SERVER-7-CBC_POST_RESET_NOTIFICATION : Node 0/1 CBC-0, reset reason CPU_RESET_POR (0x05000000)
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_OFF, event_reason_str 'HW Event Powered OFF' for card 0/1
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_ON, event_reason_str 'HW Event Powered ON' for card 0/1
Note: This state does not mean that the linecard has a hardware failure. Note that "HW_EVENT_OK" means the linecard hardware is working as expected, FPGA and all PCIE show no issue.
Warning: Interrupting boot process generates boot process to restart and modules take time to finish. Be aware of the logs displayed in the CLI. If the logs do not show any error, let the linecard/module continues with the process.
In case the router detects an error in the boot process, the router automatically performs recovery actions.
2. If the linecard shows this logs, do not remove it from the chassis. The linecard is in the downloading process and needs to finish.
0/0/ADMIN0: inst_agent[3930]: %INFRA-INSTAGENT-4-XR_PART_PREP_REQ : Received SDR/XR partition request. Looking for available matching partition. If not found, new one will be created after copying relevant image and RPMs
0/0/ADMIN0: fpd_agent[3927]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :Golden FPGA is unprotected :DECLARE :0/0: FPD IPU-FP
Golden FPGA is below minimum version, Perform force fpd upgrade for IPU-FPGA
0/0/ADMIN0: fpd_agent[3927]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :Golden FPGA is unprotected :DECLARE :0/0: FPD IPU-FS
Golden FPGA is below minimum version, Perform force fpd upgrade for IPU-FSBL
0/0/ADMIN0: fpd_agent[3927]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :Golden FPGA is unprotected :DECLARE :0/0: F
IPU-Linux Golden FPGA is below minimum version, Perform force fpd upgrade for IPU-Linux
0/0/ADMIN0: inst_agent[3930]: %INFRA-INSTAGENT-4-XR_PART_PREP_IMG : SDR/XR image baking in progress
Note: After this, FPD gets upgraded and the linecard must boot with no problem. In case an issue is observed, check the Action Items and Information to open a case with TAC sections.
3. In case there is no log from the recently inserted linecard please go to the section Information to Collect.
For "SW_INACTIVE" focus on the XR VM and Admin VM.
Consider the next, when issuing the show platform command:
This example displays the output for both show platform commands:
RP/0/RSP0/CPU0:XR# show platform
Node Type State Config state
--------------------------------------------------------------------------------
0/RSP0 A9K-RSP5-SE SW_INACTIVE NSHUT
0/RSP1/CPU0 A9K-RSP5-SE(Active) IOS XR RUN NSHUT
0/FT0 ASR-9006-FAN-V2 OPERATIONAL NSHUT
0/FT1 ASR-9006-FAN-V2 OPERATIONAL NSHUT
0/0/CPU0 A9K-24X10GE-1G-SE IOS XR RUN NSHUT
0/1/CPU0 A9K-24X10GE-1G-SE IOS XR RUN NSHUT
0/2 A9K-20HG-FLEX-SE SW_INACTIVE NSHUT
0/3 A9K-20HG-FLEX-SE SW_INACTIVE NSHUT
0/PT0 A9K-DC-PEM-V2 OPERATIONAL NSHUT
RP/0/RSP0/CPU0:XR# admin show platform
Location Card Type HW State SW State Config State
----------------------------------------------------------------------------
0/0 A9K-24X10GE-1G-SE OPERATIONAL OPERATIONAL NSHUT
0/1 A9K-24X10GE-1G-SE OPERATIONAL OPERATIONAL NSHUT
0/2 A9K-20HG-FLEX-SE OPERATIONAL OPERATIONAL NSHUT
0/3 A9K-20HG-FLEX-SE OPERATIONAL OPERATIONAL NSHUT
0/RSP0 A9K-RSP5-SE OPERATIONAL OPERATIONAL NSHUT
0/RSP1 A9K-RSP5-SE OPERATIONAL OPERATIONAL NSHUT
0/FT0 ASR-9006-FAN-V2 OPERATIONAL N/A NSHUT
0/FT1 ASR-9006-FAN-V2 OPERATIONAL N/A NSHUT
0/PT0 A9K-DC-PEM-V2 OPERATIONAL N/A NSHUT
Note: The shown output displays information from a different router than the one used in further show commands.
This example displays the output for both show platform commands:
RP/0/RSP0/CPU0:XR# show platform
Node Type State Config state
--------------------------------------------------------------------------------
0/RSP0 A9K-RSP5-SE IOS XR RUN NSHUT
0/RSP1/CPU0 A9K-RSP5-SE(Active) IOS XR RUN NSHUT
0/FT0 ASR-9006-FAN-V2 OPERATIONAL NSHUT
0/FT1 ASR-9006-FAN-V2 OPERATIONAL NSHUT
0/0/CPU0 A9K-24X10GE-1G-SE IOS XR RUN NSHUT
0/1/CPU0 A9K-24X10GE-1G-SE IOS XR RUN NSHUT
0/2 A9K-20HG-FLEX-SE IOS XR RUN NSHUT
0/3 A9K-20HG-FLEX-SE IOS XR RUN NSHUT
0/PT0 A9K-DC-PEM-V2 OPERATIONAL NSHUT
RP/0/RSP0/CPU0:XR# admin show platform
Location Card Type HW State SW State Config State
----------------------------------------------------------------------------
0/0 A9K-24X10GE-1G-SE OPERATIONAL SW_INACTIVE NSHUT
0/1 A9K-24X10GE-1G-SE OPERATIONAL OPERATIONAL NSHUT
0/2 A9K-20HG-FLEX-SE OPERATIONAL OPERATIONAL NSHUT
0/3 A9K-20HG-FLEX-SE OPERATIONAL OPERATIONAL NSHUT
0/RSP0 A9K-RSP5-SE OPERATIONAL OPERATIONAL NSHUT
0/RSP1 A9K-RSP5-SE OPERATIONAL OPERATIONAL NSHUT
0/FT0 ASR-9006-FAN-V2 OPERATIONAL N/A NSHUT
0/FT1 ASR-9006-FAN-V2 OPERATIONAL N/A NSHUT
0/PT0 A9K-DC-PEM-V2 OPERATIONAL N/A NSHUT
Once a boot loop is identified or the linecard is not going to Baking, check:
As mentioned, the linecard needs to download the file and boot. For this there are 2 options:
First Option: Connect to the console of the linecard:
For this, issue the next commands:
RP/0/RP0/CPU0:XR#admin
sysadmin-vm:0_RP0#run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ attachCon 0/X --> X is the linecard slot number
Example shows how to connect to linecard 0/0/CPU0:
RP/0/RP0/CPU0:XR#admin
sysadmin-vm:0_RP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ attachCon 0/0
===============================================
==== Connecting to Line Card =====
===============================================
Line Card: No 0
Press <Ctrl-W> to disconnect
Enabling 16550 on uart 0 baud rate 115200
host login:
Note: Select enter and insert the host log in information, information used when provisioning the router for the first time.
Second Option: Check PCIE logs in the Admin VM:
In admin issue:
sysadmin-vm:0_RSP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd/
This command displays the linecards that are inserted in the chassis.
Example shows the command in a chassis with only one linecard inserted:
[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd
LC0
Once the "SW_INACTIVE" linecard is identified and displayed in the output, proceed to issue the next command:
[sysadmin-vm:0_RSP0:/]$ ls /misc/scratch/pcie/loggerd/LC0
Note: When the commands are issued, consider that it displays different linecards. In such case, issue the command with the linecard of interest. For this example LC0.
The command displays the files in the linecard. Note any file with pslot.
Warning: When issuing the command, consider users can lose access to the CLI. In case the CLI gets frozen, select Ctrl-W.
Example outputs for the pslot found in linecard 0/0/CPU0:
[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd/LC0
[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd/LC0
first last pslot_2_uart_0_w0
[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd/LC0/pslot_2_uart_0_w0
The next information is displayed after selecting enter.
===============================================
==== Connecting to Line Card =====
===============================================
Line Card: No 0
Press <Ctrl-W> to disconnect
Enabling 16550 on uart 0 baud rate 115200
TriggerCpRmonInit
ASR9K Init Starting ASR9k initialization ...
Reading both MB and DB cookie
Board Type:0x3d1013
Starlord LC setting i2c block 7
The base address of i2c_mux4to1 is at dc30e000
zl init starlord lc
Initializing Zl clock to 322MHz
Missing Parameter SERVER_URL
##########################################################
System Bootstrap, Version 23.278 [ASR9K x86 ROMMON],
Copyright (c) 1994-2020 by Cisco Systems, Inc.
Compiled on Sat 03/14/2020 14:37:06.76
BOARD_TYPE : 0x3d1013
Rommon : 23.278 (Primary)
IPU FPGA(PL) : 1.10.1 (Primary)
IPU INIT(HW.FPD) : 1.10.1
IPU FSBL(BOOT.BIN) : 1.104.0
IPU LINUX(IMAGE.FPD) : 1.104.0
GAMORA FPGA : 0.36.1
CBC0 : Part 1=55.7, Part 2=55.7, Act Part=1
Product Number : ASR-9901-LC
Slot Number : 2
##########################################################
Got EMT Mode as Disk Boot
<snip>
Note: In some cases, with the linecard in "SW_INACTIVE", the PCIE displays the boot loop. Proceed to log the session and collect these outputs for further troubleshooting.
Issue the next commands:
RP/0/RP0/CPU0:PE2#admin
sysadmin-vm:0_RP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ls -lah /misc/disk1/tftpboot
The next output is the output of this command:
sysadmin-vm:0_RSP0# run ls -lah /misc/disk1/tftpboot
total 3.2G
drwxrwxrwx. 3 root root 4.0K Jan 31 02:14 .
drwxrwxrwx. 12 root root 4.0K Jan 31 01:49 ..
-rw-r--r--. 1 root root 119M Jan 31 01:46 asr9k-common-7.3.2
-rw-r--r--. 1 root root 120M Dec 18 15:32 asr9k-common-7.5.2
-rw-r--r--. 1 root root 1.4G Jan 31 01:36 asr9k-mini-x64-7.3.2
-rw-r--r--. 1 root root 1.5G Dec 18 15:25 asr9k-mini-x64-7.5.2
drwxr-xr-x. 3 root root 4.0K Mar 16 2022 boot
-rwxr-xr-x. 1 root root 637 Jan 31 01:57 grub.cfg
-rw-r--r--. 1 root root 906K Jan 31 01:48 grub.efi
lrwxrwxrwx. 1 root root 22 Jan 31 01:57 system_image.iso -> ./asr9k-mini-x64-7.3.2
-rwxrwxrwx. 1 root root 1007 Jan 31 01:58 zapdisk.log
From the output, consider the next:
To review the DHCP configuration, issue the next commands:
RP/0/RP0/CPU0:XR#admin
sysadmin-vm:0_RP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$cat /etc/dhcp/dhcpd.conf
The example for this command outputs:
RP/0/RP0/CPU0:XR#admin
sysadmin-vm:0_RP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$cat /etc/dhcp/dhcpd.conf
ddns-update-style none;
default-lease-time 86400;
max-lease-time 604800;
authoritative;
subnet 192.168.0.0 netmask 255.0.0.0 {
pool {
max-lease-time 600;
range 192.168.0.128 192.168.0.191;
allow unknown-clients;
}
filename "grub.efi";
next-server 192.168.0.1;
option subnet-mask 255.0.0.0;
option broadcast-address 192.168.0.255;
option ip-forwarding off;
}
Like in the previous scenario, check if this file displays information, if so, please proceed to Information to Open a Case with TAC section.
If no output is seen, please proceed to the Action Item.
Tip: In some cases, and depending on the scenario fpd auto-upgrade enable needs to be configured to both Admin VM and XR VM.
"I see the files in /misc/disk1/tftpboot and /etc/dhcp/dhcpd.conf displays information": Add the fpd auto-upgrade in the XR VM (for more information click: System Management Configuration Guide)
Once this is configured, Online Insertion and Removal (OIR) the Linecard and check the logs that this one displays.
"I do not see any file in /misc/disk1/tftpboot but/etc/dhcp/dhcpd.conf displays information": This can be an issue with the Active Route Processor not being able to share the files.
sysadmin-vm:0_RSP0# hw-module location <Linecard in SW_INACTIVE state> bootmedia network reload
In this example, the command is issued for linecard 0/0/CPU0:
sysadmin-vm:0_RSP0# hw-module location 0/0 bootmedia network reload
Check the logs with the same troubleshooting steps. If the logs are similar, continue to Information required to open a case with TAC section:
"I do not see files in misc/disk1/tftpboot nor information is displayed in /etc/dhcp/dhcpd.conf": This scenario needs further checking from TAC. Proceed to Information required to open a case with TAC section.
Tip: When opening a case, collect all the information and attach it to the case (when opened). This avoids any delay in the investigation
Logs to be collected:
In XR VM:
show logging
Route Processor/Linecard console logs; in case the issue is presented in the standby Route Processor, consider connecting the console cable to this node and log the output that is being displayed.
show hw-module fpd location all
In XR VM:
show controllers switch statistics location <Route Processor/Linecard>
show controllers switch statistics detail location <active Route Processor> <switch port no>
show alarms
Files to attach:
In Admin VM:
Show tech-support OS
Show tech-support canbus
Show tech-support control-ethernet
Show tech-support ctrace
Note: Admin show techs: To copy the admin show tech to the default XR_PLANE, issue the next command in admin mode: copy harddisk:/showtech/ harddisk:/showtech/ location 0/RSP/VM1. In case there are issues copying the file, click this link: Show Tech Support files to XR VM.
Revision | Publish Date | Comments |
---|---|---|
1.0 |
16-Feb-2024 |
Initial Release |