Introduction
This document describes how to avoid situations in which the Cisco NCS 560 route processor (RP) becomes unresponsive after a reload or a power cycle.
Background Information
In most cases, the NCS 560 deployments are in remote locations, in large numbers, and where human intervention must be kept to a minimum.
In these deployments, it is also important to fulfil the mandatory SMU installation requirement based on the installed release.
Note: From Cisco IOS XR Release 7.2.2 and onwards, you are no longer required to install SMUs separately.
SMU Summary Table
FPD Upgrade MOP
FPDs must be upgraded to ensure that FPDs are always at a version equal to or higher than the minimum required version required for that Cisco IOS® XR release.
ADM
|
1.06
|
IOFPGA
|
0.64
|
BIOS
|
0.16
|
This document assumes that:
- RP0 is the Active RP
- RP1 is the Standby RP
the prompt for various locations so that all commands to navigate the various location are omitted.
1. XR - RP0
RP/0/RP0/CPU0:ios#
2. XR - RP1
RP/0/RP1/CPU0:ios#
3. sysadmin - RP0
sysadmin-vm:0_RP0#
4. sysadmin - RP1
sysadmin-vm:0_RP1#
5. sysadmin - RP0 shell
[sysadmin-vm:0_RP0:~]$
6. sysadmin - RP1 shell
[sysadmin-vm:0_RP1:~]$
Use Case. System with Cisco IOS-XR Release 7.1.2. And Necessary SMUS Activated
These SMUs must be active and committed to the system prior to this process.
XR VM
RP/0/RP0/CPU0:ios#show install active summary
Active Packages: 10
ncs560-xr-7.1.2 version=7.1.2 [Boot image]
ncs560-mcast-2.0.0.0-r712
ncs560-mgbl-2.0.0.0-r712
ncs560-mpls-1.0.0.0-r712
ncs560-mpls-te-rsvp-2.0.0.0-r712
ncs560-ospf-2.0.0.0-r712
ncs560-k9sec-2.1.0.0-r712
ncs560-iosxr-fwding-2.1.0.1-r712.CSCvp67225
ncs560-os-support-2.1.0.1-r712.CSCvv41516
cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907 .xr
sysadmin
sysadmin-vm:0_RP0# show install active summary
Active Packages: 4
ncs560-sysadmin-7.1.2 version=7.1.2 [Boot image]
ncs560-sysadmin-ncs560-7.1.2.1-r712.CSCvv41516
cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907 .admin
cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907 .host
Verify FPD Versions on the RPs
RP/0/RP0/CPU0:ios#show hw-module fpd
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/5 A900-IMA8Z 0.0 IMFPGA NEED UPGD 17.02 17.02
0/14 A900-IMA8CS1Z-M 0.0 IMFPGA CURRENT 1.95 1.95
0/RP0 N560-RSP4-E 0.0 ADM NEED UPGD 1.04 1.04
0/RP0 N560-RSP4-E 0.0 IOFPGA NEED UPGD 0.53 0.53
0/RP0 N560-RSP4-E 0.0 PRIMARY-BIOS NEED UPGD 0.15 0.15
0/RP0 N560-RSP4-E 0.0 SATA NEED UPGD 1.30 1.30
0/RP1 N560-RSP4-E 0.0 ADM NEED UPGD 1.04 1.04
0/RP1 N560-RSP4-E 0.0 IOFPGA NEED UPGD 0.53 0.53
0/RP1 N560-RSP4-E 0.0 PRIMARY-BIOS NEED UPGD 0.15 0.15
0/RP1 N560-RSP4-E 0.0 SATA NEED UPGD 1.30 1.30
0/FT0 A907-FAN-E 1.0 PSOC CURRENT 1.65 1.65
Check TAM Area on Both RPs (Evaluate the Removal of the Section)
It is important to ensure that the TAM area is not corrupted; otherwise, an IOFPGA, FPD upgrade, and subsequent reloads might lead to an RMA. With this MOP, since FPD version 0.64 or higher is in use, this IOFPGA FPD upgrade process and RP reload leads to TAM corruption recovery.
RP0
sysadmin-vm:0_RP0# show control-driver info location 0/RP0
...
Card view from local RP:
TAM Memory is not corrupted
Can proceed with IOFPGA upgrade if required
...
RP1
sysadmin-vm:0_RP0# show control-driver info location 0/RP1
...
Card view from local RP:
TAM Memory is not corrupted
Can proceed with IOFPGA upgrade if required
...
Note: If TAM Memory is corrupted, it is not a problem if the target IOFPGA version is 0.64 or higher.
Sample Output of Corrupted TAM Memory
sysadmin-vm:0_RP1# show control-driver info location 0/RP0
<snip>
Card view from local RP:
TAM Memory is corrupted
Do not proceed with IOFPGA upgrade
<snip>
Note: If all prerequisites mentioned in this document are fulfilled, you can proceed with the upgrade. The “Do not proceed with IOFPGA upgrade” refers to the IOFPGA version earlier than 0.64.
Perform IOFPGA Upgrade on Both RPs
RP/0/RP0/CPU0:ios#upgrade hw-module location all fpd IOFPGA
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP0/CPU0:RONALDO-RF2a#0/RP0/ADMIN0:Oct 16 15:19:26.609 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location FPD name Force
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1 IOFPGA FALSE
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0 IOFPGA FALSE
Verify
RP/0/RP0/CPU0:ios#show hw-module fpd
Fri Oct 16 15:22:11.256 CEST
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/5 A900-IMA8Z 0.0 IMFPGA NEED UPGD 17.02 17.02
0/14 A900-IMA8CS1Z-M 0.0 IMFPGA CURRENT 1.95 1.95
0/RP0 N560-RSP4-E 0.0 ADM NEED UPGD 1.04 1.04
0/RP0 N560-RSP4-E 0.0 IOFPGA RLOAD REQ 0.53 0.64
0/RP0 N560-RSP4-E 0.0 PRIMARY-BIOS NEED UPGD 0.15 0.15
0/RP0 N560-RSP4-E 0.0 SATA NEED UPGD 1.30 1.30
0/RP1 N560-RSP4-E 0.0 ADM NEED UPGD 1.04 1.04
0/RP1 N560-RSP4-E 0.0 IOFPGA RLOAD REQ 0.53 0.64
0/RP1 N560-RSP4-E 0.0 PRIMARY-BIOS NEED UPGD 0.15 0.15
0/RP1 N560-RSP4-E 0.0 SATA NEED UPGD 1.30 1.30
0/FT0 A907-FAN-E 1.0 PSOC CURRENT 1.65 1.65
Reload the Active and Standby RPs for IOFPGA Upgrade to Complete
First, reload the standby RP:
sysadmin-vm:0_RP0# hw-module location 0/RP1 reload
Reload hardware module ? [no,yes] yes
After the redundancy is restored, reload the active RP (RP0) (it triggers a failover):
sysadmin-vm:0_RP0# hw-module location 0/RP0 reload
Reload hardware module ? [no,yes] yes
Verify ADM Version
On the new active RP (RP1):
[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.2
Bootloader Version : 0.0.9
Config Version : 1.4
On the new standby RP (RP0):
[sysadmin-vm:0_RP0:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.2
Bootloader Version : 0.0.9
Config Version : 1.4
Perform Other FPD Upgrade on Both RPs
1. ADM
RP/0/RP1/CPU0:ios#upgrade hw-module location all FPd ADM
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 15:57:14.332 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location FPD name Force
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1 ADM FALSE
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0 ADM FALSE
2. BIOS
RP/0/RP1/CPU0:ios#upgrade hw-module location all FPD PRIMARY-BIOS
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location FPD name Force
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1 PRIMARY-BIOS FALSE
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0 PRIMARY-BIOS FALSE
3. SATA
RP/0/RP1/CPU0:ios#upgrade hw-module location all FPD SATA
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location FPD name Force
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1 SATA FALSE
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0 SATA FALSE
Verify the Upgrades
RP/0/RP1/CPU0:ios#show hw-module fpd
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/5 A900-IMA8Z 0.0 IMFPGA NEED UPGD 17.02 17.02
0/14 A900-IMA8CS1Z-M 0.0 IMFPGA CURRENT 1.95 1.95
0/RP0 N560-RSP4-E 0.0 ADM RLOAD REQ 1.04 1.04
0/RP0 N560-RSP4-E 0.0 IOFPGA CURRENT 0.64 0.64
0/RP0 N560-RSP4-E 0.0 PRIMARY-BIOS RLOAD REQ 0.15 0.16
0/RP0 N560-RSP4-E 0.0 SATA RLOAD REQ 1.30 2.10
0/RP1 N560-RSP4-E 0.0 ADM RLOAD REQ 1.04 1.04
0/RP1 N560-RSP4-E 0.0 IOFPGA CURRENT 0.64 0.64
0/RP1 N560-RSP4-E 0.0 PRIMARY-BIOS RLOAD REQ 0.15 0.16
0/RP1 N560-RSP4-E 0.0 SATA RLOAD REQ 1.30 2.10
0/FT0 A907-FAN-E 1.0 PSOC CURRENT 1.65 1.65
Power Cycle Procedure to Complete Other FPD Upgrade
Note: If the ADM firmware version is 1.14.2, you must synchronize all filesystems on all VMs/host before you use the reload script.
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.1 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.4 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.4 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.1 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.1 chvrf 0 ssh 10.0.2.2 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.1 chvrf 0 ssh 10.0.2.2 sync
Note: There is no output for these commands.
In order to reload the chassis:
[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/reboot_system.sh
Initiating power-cycle of 0/RP0
Initiating power-cycle of 0/RP1
Note: If the ADM firmware version is 1.14.3, FPDs on the RP require a reload.
First, reload the standby RP (RP0):
sysadmin-vm:0_RP1# hw-module location 0/RP0 reload
Reload hardware module ? [no,yes] yes
After redundancy is restored, reload the active RP (RP1) (it triggers a failover):
sysadmin-vm:0_RP1# hw-module location 0/RP1 reload
Reload hardware module ? [no,yes] yes
Verify FPD Versions
RP/0/RP0/CPU0:ios#show hw-module fpd
Fri Oct 16 16:24:18.849 CEST
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/5 A900-IMA8Z 0.0 IMFPGA NEED UPGD 17.02 17.02
0/14 A900-IMA8CS1Z-M 0.0 IMFPGA CURRENT 1.95 1.95
0/RP0 N560-RSP4-E 0.0 ADM CURRENT 1.06 1.06
0/RP0 N560-RSP4-E 0.0 IOFPGA CURRENT 0.64 0.64
0/RP0 N560-RSP4-E 0.0 PRIMARY-BIOS CURRENT 0.16 0.16
0/RP0 N560-RSP4-E 0.0 SATA CURRENT 2.10 2.10
0/RP1 N560-RSP4-E 0.0 ADM CURRENT 1.06 1.06
0/RP1 N560-RSP4-E 0.0 IOFPGA CURRENT 0.64 0.64
0/RP1 N560-RSP4-E 0.0 PRIMARY-BIOS CURRENT 0.16 0.16
0/RP1 N560-RSP4-E 0.0 SATA CURRENT 2.10 2.10
0/FT0 A907-FAN-E 1.0 PSOC CURRENT 1.65 1.65
Verify ADM Version
[sysadmin-vm:0_RP0:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.3
Bootloader Version : 0.0.9
Config Version : 1.6
[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.3
Bootloader Version : 0.0.9
Config Version : 1.6
Upgrade Other FPDs on Other Locations (If Any)
RP/0/RP0/CPU0:ios#upgrade hw-module location 0/5 FPD IMFPGA
Fri Oct 16 16:28:15.006 CEST
upgrade command issued (use "show hw-module fpd" to check upgrade status)