Replacing the NCS 4000 Rack

Rolling Back a Multi Chassis System (1+1) to a Single Chassis System

This procedure describes the steps for rolling back a 1+1 MC system to an SC system.

Procedure

Step 1

Take note of the important configuration values of fabric planes on calvados in system admin mode.

sysadmin-vm:2_RP1:R3# show controller fabric plane all detail 
Wed Jan  8  13:59:51.628 UTC-05:30

Plane Admin Plane  Plane  up->dn  up->mcast Total   Down    PPU
Id    State State  Mode   counter   counter Bundles Bundles State
------------------------------------------------------------
0     UP    UP          MC       0         1     24       0  NA        
1     UP    UP          MC       0         1     24       0  NA        
2     UP    UP          MC       0         2     24       0  NA        
3     UP    UP          MC       0         1     24       0  NA        
sysadmin-vm:2_RP1:R3# show controller fabric link port s3 rx | in UP/UP | count
Wed Jan  8  14:00:12.240 UTC-05:30
Count: 1151 lines
sysadmin-vm:2_RP1:R3# show running-config controller 
Wed Jan  8  14:00:45.634 UTC-05:30
controller fabric plane 0
instance 0
location F0/FC0
!
instance 1
location F0/FC4
!
!
controller fabric plane 1
instance 0
location F0/FC1
!
instance 1
location F0/FC5
!
!
controller fabric plane 2
instance 0
location F1/FC0
!
instance 1
location F1/FC4
!
!       
controller fabric plane 3
instance 0
location F1/FC1
!
instance 1
location F1/FC5
!
!

sysadmin-vm:2_RP1:R3# show running-config chassis 
Wed Jan  8  14:01:53.275 UTC-05:30
chassis serial FLM213200B9
rack F0
!
chassis serial FLM213200BN
rack F1
!
chassis serial SAL18370JXS
rack 2
!
chassis serial SAL1925H76U
rack 0
!
Step 2

Shutdown fabric plane 0 using the following commands.

sysadmin-vm:2_RP1:R3(config)# controller fabric plane 0
sysadmin-vm:2_RP1:R3(config-plane-0)# shut
sysadmin-vm:2_RP1:R3(config-plane-0)# commit
Wed Jan  8  14:03:13.639 UTC-05:30
Commit complete.
Step 3

Remove the instance fabric configuration for plane 0 using the following command.

sysadmin-vm:2_RP1:R3(config)# no instance 0
sysadmin-vm:2_RP1:R3(config)# no instance 1
sysadmin-vm:2_RP1:R3(config-plane-0)# commit
Note 
Verify the NCS4016-FC2-M or NCS4009-FC2-M card for fabric plane 0 and wait for the card to reload automatically. The mode changes to SC. 
0/RP0/ADMIN0:Jan  8 14:16:15.852 IST: shelf_mgr[3118]: %INFRA-SHELF_MGR-4-CARD_RELOAD : Reloading card 0/FC0  
0/RP0/ADMIN0:Jan  8 14:16:15.852 IST: shelf_mgr[3118]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :AUTORESET :DECLARE :0/FC0: Automatic System Reset  
2/RP0/ADMIN0:Jan  8 14:16:16.045 IST: shelf_mgr[3177]: %INFRA-SHELF_MGR-4-CARD_RELOAD : Reloading card 2/FC0  
2/RP0/ADMIN0:Jan  8 14:16:16.048 IST: shelf_mgr[3177]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :AUTORESET :DECLARE :2/FC0: Automatic System Reset
Step 4

After 7 minutes, when the fabric cards become operational, un-shut the fabric plane 0. After un-shut, Admin State changes to UP and Fabric State changes to DOWN; and the traffic is not affected on the LCC0 and the plane 0 fabric cards in SC Mode.

Step 5

Follow the above steps for the other fabric planes 1, 2, 3 and check the fabric status.

sysadmin-vm:0_RP0:R3# show controller fabric plane all detail 
Wed Jan  8  18:31:01.271 UTC-05:30

Plane Admin Plane  Plane  up->dn  up->mcast Total   Down    PPU
Id    State State  Mode   counter   counter Bundles Bundles State
------------------------------------------------------------
0     UP    DN          SC       3         0      0       0  NA        
1     UP    DN          SC       3         1      0       0  NA        
2     UP    DN          SC       3         0      0       0  NA        
3     UP    DN          SC       2         1      0       0  NA
Step 6

Remove the chassis configuration for fabric plane 0, fabric plane 1, and LCC2 to shutdown the respective fabric and other LCC1, LCC2, LCC3 chassis. Only LCC0 remains live. 

sysadmin-vm:0_RP0:R3(config)# no chassis serial FLM213200B9
Wed Jan  8  18:37:12.290 UTC-05:30
sysadmin-vm:0_RP0:R3(config)# no chassis serial FLM213200BN
Wed Jan  8  18:37:20.647 UTC-05:30
sysadmin-vm:0_RP0:R3(config)# no chassis serial SAL18370JXS
Wed Jan  8  18:37:27.800 UTC-05:30
sysadmin-vm:0_RP0:R3(config)# commit
Wed Jan  8  18:37:31.194 UTC-05:30
Step 7

Check the fabric plane details in the system admin mode.

Note 
The fabric plane changes to UP state and does not impact traffic on LCC0.
 
sysadmin-vm:0_RP0:R3# show controller fabric plane all detail 
Wed Jan  8  18:39:49.109 UTC-05:30

Plane Admin Plane  Plane  up->dn  up->mcast Total   Down    PPU
Id    State State  Mode   counter   counter Bundles Bundles State
------------------------------------------------------------
0     UP    UP          SC       3         0      0       0  NA        
1     UP    UP          SC       3         1      0       0  NA        
2     UP    UP          SC       3         0      0       0  NA        
3     UP    UP          SC       2         1      0       0  NA
Step 8

Remove the control plane cables and other fabric plane cables.

Replacing the NCS 4000 Rack

This procedure describes the steps for replacing the NCS 4000 rack.

Procedure

Step 1

Check the running configuration of the NCS 4000 chassis.

sysadmin-vm:2_RP1:R3# show running-config chassis 
Wed Jan  8  14:01:53.275 UTC-05:30
chassis serial FLM213200B9
rack F0
!
chassis serial FLM213200BN
rack F1
!
chassis serial SAL18370JXS
rack 2
!
chassis serial SAL1925H76U
rack 0
!
Step 2

Verify the rack to be replaced is the source of mac-pool using show macpool command.

  • If the rack to be replaced is the source of mac-pool, the rack is marked as true.

    In the below example, rack1 is the source of mac addresses. It is marked as true. 

    sysadmin-vm:1_RP1# show macpool 
Sat Oct  30 13:25:07.475 UTC+00:00
Rack Serial   Rack Num  Mac Base           Mac Count   Selected    Allocated   
-------------------------------------------------------------------------------
FLM213101UR   F0        00:00:00:00:00:00  0           false       0           
FLM213200BC   F1        00:00:00:00:00:00  0           false       0           
FLM213300GV   F2        00:00:00:00:00:00  0           false       0           
SAL1806LW7T   6         74:ab:62:44:eb:20  2015        false       0           
SAL1824UHYD   3         b0:4a:d4:00:00:20  2015        false       0           
SAL1824UJ1F   4         aa:00:ea:00:00:20  2015        false       0           
SAL1834Z18B   5         f0:7f:06:e7:26:c0  2015        false       0           
SAL184121V3   0         f0:7f:06:e7:8c:80  2015        false       0           
SAL1917DZBZ   1         78:ba:f9:9b:72:40  2015        true        100         
SAL1940Q2A4   2         00:af:1f:d6:00:20  2015        false       0           
SAL2016PB3V   7         04:2a:e2:b3:b0:20  2015        false       0

  • If the rack to be replaced is the not source of mac-pool, the rack is marked as false.

    In the procedure, the rack to be replaced, rack2, is not the source of mac addresses.
Step 3

Enter the serial number of the chassis to remove the old rack using no chassis serial serialNumber command. 

sysadmin-vm:0_RP0:R3(config)# no chassis serial SAL18370JXS
Wed Jan  8  18:37:27.800 UTC-05:30
sysadmin-vm:0_RP0:R3(config)# commit
Wed Jan  8  18:37:31.194 UTC-05:30
Step 4

After deleting the chassis configuration, check if the chassis is shut down. If not, power off the rack.
Step 5

If the deleted rack is the source of the mac-pool:

  • Reload the complete system using the hw-module location all reload command.

  • If not, skip this step.
Step 6

Check if other racks are operational using show chassis . 

sysadmin-vm:0_RP0# show chassis
Wed Oct  16  09:41:31.116 UTC-05:00
Serial Num    Rack Num    Rack Type   Rack State  Data Plane  Ctrl Plane  
--------------------------------------------------------------------------
FLM213200BN   F1          FCC         OPERATIONAL CONN       CONN         
FLM213200B9   F0          FCC         OPERATIONAL CONN       CONN         
SAL1925H76U   0           LCC         OPERATIONAL CONN       CONN
Step 7

Remove the control plane connection SFP+ pluggables and cables from the RP0 and RP1 for the rack under removal.
Step 8

Remove the CXP2 pluggables and related fiber for all the fabric cards FC0, FC1, FC2, and FC3.
Step 9

Insert the required cards and pluggables for the line card.
Step 10

Insert the required fabric card without CXP2 pluggables and insert RPs without SFP+ pluggables.
Step 11

Power on the new rack and connect the Console Con0 and Con1 on RP0 and RP1.
Step 12

Check the rack id of the new rack using show chassis command in admin mode.

Note 
By default, the new rack displays rack id as 0.
Step 13

Add new chassis in system admin configuration mode.

sysadmin-vm:0_RP0:R3(config)# chassis serial SAL18370JXT rack 2
Wed Jan  8  18:37:27.800 UTC-05:30
sysadmin-vm:0_RP0:R3(config)# commit
Wed Jan  8  18:37:31.194 UTC-05:30
Step 14

Power up the new rack and check the rack id is changed according to requirement using show chassis command.

Step 15

When the rack is proper with rack id, switch off the rack.
Step 16

Reconnect the CXP2 pluggable and related fiber on all the fabric cards FC0, FC1, FC2, and FC3.
Step 17

Reconnect the SFP+ pluggable and related fiber on RP0 and RP1.
Step 18

Note down the new rack serial number written on the rack or displayed in Step 13 output.

Step 19

Provision the new rack serial number in existing MC system.

sysadmin-vm:0_RP0:R3(config)# chassis serial SAL18370JXT rack 2
Wed Jan  8  18:37:27.800 UTC-05:30
sysadmin-vm:0_RP0:R3(config)# commit
Wed Jan  8  18:37:31.194 UTC-05:30
Step 20

Power on the new rack. In this example, it is rack 2.
Step 21

Validate all the control plane and fabric connectivity.