High Availability (SSO) Deployment Guide
High Availability in Release 7.3 and 7.4
HA Connectivity Using Redundant Port on the 5500/7500/8500 WLC
High Availability Connectivity Using Redundant VLAN on WiSM-2 WLC
Introduction of New Interfaces for HA Interaction
Redundancy Management Interface
Configure HA from the Configuration Wizard
Important Guidelines before Initiating a WLC Upgrade in HA Setup
Download/Upload Facts in HA Setup
Failover Process in the HA Setup
Steps to Simulate Box Failover
SSO Deployment with Legacy Primary/Secondary/Tertiary HA
SSO Deployment in Mobility Setup
One WLC has a valid AP Count license and the other WLC has a HA SKU UDI
Both the WLCs have a valid AP Count license
One WLC has an Evaluation license and the other WLC has a HA SKU UDI or Permanent license
High Availability in Release 7.5
Redundancy Port Connectivity in 7.5
Supported HA Topologies in Release 7.5
5500/7500/8500 Series Controllers
5508, 7500 or 8500 Connected to VSS Pair
Supported HA Topologies for WiSM2 Controllers
WiSM2 in Different Chassis: Redundancy VLAN over L2 Network
WiSM2 in Different Chassis: VSS Pair
Client SSO (Client Stateful Switchover)
Client SSO Behavior and Limitations
High Availability in Release 8.0
Debug/Show Command Enhancements
Configurable Keep-alive and Peer-Search Parameters
Peer RMI ICMP Ping Replaced with UDP Messages
Standby WLC On-the-Fly Maintenance Mode (MTC)
Default Gateway Reachability Check Enhancement
SSO Support for Internal DHCP Server
SSO Support for Sleeping Clients
This document provides information on the theory of operation and configuration for the Cisco Unified Wireless LAN Controller (WLC) as it pertains to supporting stateful switchover of access points and clients (AP and Client SSO).
The new High Availability (HA) feature (that is, AP SSO) set within the Cisco Unified Wireless Network software release version 7.3 and 7.4 allows the access point (AP) to establish a CAPWAP tunnel with the Active WLC and share a mirror copy of the AP database with the Standby WLC. The APs do not go into the Discovery state when the Active WLC fails and the Standby WLC takes over the network as the Active WLC.
There is only one CAPWAP tunnel maintained at a time between the APs and the WLC that is in an Active state. The overall goal for the addition of AP SSO support to the Cisco Unified Wireless LAN is to reduce major downtime in wireless networks due to failure conditions that may occur due to box failover or network failover.
To support High Availability without impacting service, there needs to be support for seamless transition of clients and APs from the active controller to the standby controller. Release 7.5 supports Client Stateful Switch Over (Client SSO) in Wireless LAN controllers. Client SSO will be supported for clients which have already completed the authentication and DHCP phase and have started passing traffic. With Client SSO, a client's information is synced to the Standby WLC when the client associates to the WLC or the client’s parameters change. Fully authenticated clients, i.e. the ones in Run state, are synced to the Standby and thus, client re-association is avoided on switchover making the failover seamless for the APs as well as for the clients, resulting in zero client service downtime and no SSID outage.
The information in this document is based on these software and hardware versions:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
The new architecture for HA is for box-to-box redundancy. In other words, 1:1 where one WLC will be in an Active state and the second WLC will be in a Hot Standby state continuously monitoring the health of the Active WLC via a Redundant Port. Both the WLCs will share the same set of configurations including the IP address of the Management interface. The WLC in the Standby state does not need to be configured independently as the entire configuration (Bulk Configuration while boot up and Incremental Configuration in runtime) will be synced from the Active WLC to the Standby WLC via a Redundant Port. The AP's CAPWAP State (only APs which are in a run state) is also synced, and a mirror copy of the AP database is maintained on the Standby WLC. The APs do not go into the Discovery state when the Active WLC fails and the Standby WLC takes over the network's Active WLC.
There is no preempt functionality. When the previous Active WLC comes back, it will not take the role of the Active WLC, but will negotiate its state with the current Active WLC and transition to a Standby state. The Active and Standby decision is not an automated election process. The Active/Standby WLC is decided based on HA SKU (Manufacturing Ordered UDI) from release 7.3 onwards. A WLC with HA SKU UDI will always be the Standby WLC for the first time when it boots and pairs up with a WLC running a permanent count license. For existing WLCs having a permanent count license, the Active/Standby decision can be made based on manual configuration.
AP SSO is supported on 5500/7500/8500 and WiSM-2 WLCs. Release 7.3 only supports AP SSO that will ensure that the AP sessions are intact after switchover.
Client SSO is supported on 5500/7500/8500 and WiSM2 WLCs from release 7.5 onwards. For more information see High Availability in Release 7.5.
Here you can see the Redundant Port Connectivity between 5500 WLCs in an HA Setup:
Here you can see the Redundant Port Connectivity between Flex 7500 WLCs in an HA setup:
Note A direct physical connection between Active and Standby Redundant Ports is highly recommended. The distance between the connections can go up to 100 meters at per Ethernet cable standards.
This diagram shows HA Connectivity in a single chassis and extending Redundancy VLAN in a multiple chassis VSS setup:
Warning The Redundancy VLAN should be a non routable VLAN. In other words, no layer 3 interface should be created for this VLAN and can be allowed on VSL Link to extend HA setup between multiple chassis in VSS setup. It is important to make sure this VLAN is dedicated for the HA process and is not part of any Data VLAN, or else it may result in unpredictable results.
Note The Redundancy VLAN should be created like any normal Data VLAN on IOS® switches. Redundancy VLAN is configured for redundant port on WiSM-2 blades connected to a backplane. There is no need to configure an IP address for the Redundancy VLAN as it will receive an auto-generated IP which is discussed later in this document.
Note On Cisco WiSM2 and Cisco Catalyst 6500 Series Supervisor Engine 2T, if HA is enabled, post switchover, the APs might disconnect and reassociate with the WiSM2 controller. To prevent this from occurring, before you configure HA, we recommend that you verify–in the port channel–the details of both the active and standby Cisco WiSM2 controllers, that the ports are balanced in the same order, and the port channel hash distribution is using fixed algorithm. If they are not in order, you must change the port channel distribution to be fixed and reset Cisco WiSM2 from the Cisco Catalyst 6500 Series Supervisor Engine 2T. You can use the command show etherchannel port-channel
to verify the port channel member order and load value. You can use the config command port-channel hash-distribution fixed
to make the distribution fixed.
Note To support the active and standby WLCs in different data centers, in release 7.5, back-to-back redundancy port connectivity between peers is no longer mandatory and the redundancy ports can be connected via switches such that there is L2 adjacency between the two controllers. See Redundancy Port Connectivity in 7.5 for more information.
The IP address on this interface should be configured in the same subnet as the management interface. This interface will check the health of the Active WLC via network infrastructure once the Active WLC does not respond to Keepalive messages on the Redundant Port. This provides an additional health check of the network and Active WLC, and confirms if switchover should or should not be executed. Also, the Standby WLC uses this interface in order to source ICMP ping packets to check gateway reachability. This interface is also used in order to send notifications from the Active WLC to the Standby WLC in the event of Box failure or Manual Reset. The Standby WLC will use this interface in order to communicate to Syslog, the NTP server, and the TFTP server for any configuration upload.
This interface has a very important role in the new HA architecture. Bulk configuration during boot up and incremental configuration are synced from the Active WLC to the Standby WLC using the Redundant Port. WLCs in a HA setup will use this port to perform HA role negotiation. The Redundancy Port is also used in order to check peer reachability sending UDP keep-alive messages every 100 msec (default timer) from the Standby WLC to the Active WLC. Also, in the event of a box failure, the Active WLC will send notification to the Standby WLC via the Redundant Port. If the NTP server is not configured, a manual time sync is performed from the Active WLC to the Standby WLC on the Redundant Port. This port in case of standalone controller and redundancy VLAN in case of WISM-2 will be assigned an auto generated IP Address where last 2 octets are picked from the last 2 octets of Redundancy Management Interface (the first 2 octets are always 169.254).
1. Before you configure HA, it is mandatory to have both the controllers' management interface in the same subnet:
2. HA is disabled by default. Before you enable HA, it is mandatory to configure the Redundancy Management IP Address and Peer Redundancy Management IP Address. Both the interfaces should be in the same subnet as the Management Interface. In this example, 9.6.61.21 is the Redundancy Management IP Address for WLC 1, and 9.6.61.23 is the Redundancy Management IP Address for WLC 2. It also needs to be configured so that 9.6.61.23 is the Redundancy Management IP Address of WLC 2 and 9.6.61.21 is the Redundancy Management IP Address of WLC 1.
Use this CLI in order to configure the Redundancy and Peer Redundancy Management IP Address:
3. Configure one WLC as Primary (by default, the WLC HA Unit ID is Primary and should have a valid AP-BASE count license installed) and another WLC as Secondary (AP base count from the Primary WLC will be inherited by this unit) using the CLI in this step. In this example, WLC 1 is configured as Primary, and WLC 2 is configured as Secondary:
Note You do not need to configure the unit as Secondary if it is a factory ordered HA SKU that can be ordered from release 7.3 onwards. A factory ordered HA SKU is a default Secondary unit, and will take the role of the Standby WLC the first time it is paired with an Active WLC that has a valid AP Count License.
If you want to convert any existing WLC as a Standby WLC, do so using the config redundancy unit secondary command in the CLI. This CLI command will only work if the WLC which is intended to work as Standby has some number of permanent license count. This condition is only valid for the 5500 WLC, where a minimum of 50 AP Permanent licenses are needed to be converted to Standby. There is no restriction for other WLCs such as the WiSM2, 7500, and 8500.
4. After the WLCs are configured with Redundancy Management and Peer Redundancy Management IP Addresses and Redundant Units are configured, it is time to enable SSO. It is important to make sure that physical connections are up between both the controllers (that is, both the WLCs are connected back to back via the Redundant Port using an Ethernet cable) and the uplink is also connected to the infrastructure switch and the gateway is reachable from both the WLCs before SSO is enabled.
Once SSO is enabled, it will reboot the WLCs. While it boots, the WLCs negotiate the HA role as per the configuration via Redundant Port. If the WLCs cannot reach each other via Redundant Port or via the Redundant Management Interface, the WLC configured as Secondary may go in to Maintenance Mode. Maintenance Mode is discussed later in this document.
5. Use the CLI in this step in order to enable AP SSO. Remember that enabling AP SSO will initiate a WLC reboot.
6. Enabling SSO will reboot the WLCs in order to negotiate the HA role as per the configuration performed. Once the role is determined, configuration is synced from the Active WLC to the Standby WLC via the Redundant Port. Initially, the WLC configured as Secondary will report XML mismatch and will download the configuration from Active and reboot again. During the next reboot after role determination, it will validate the configuration again, report no XML mismatch, and process further in order to establish itself as the Standby WLC.
These are the boot-up logs from both the WLCs:
WLC 2 on first reboot after enabling SSO:
Note Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
WLC 2 on second reboot after downloading XML configuration from Active:
7. After SSO is enabled, WLC is rebooted, and the XML configuration is synced, WLC 1 will transition its state to Active and WLC 2 will transition its state to Standby HOT. From this point onwards, GUI/Telnet/SSH for WLC 2 on the management interface will not work, as all the configurations and management should be done from the Active WLC. If required, the Standby WLC (WLC 2, in this example) can only be managed via the Console or Service Port.
Also, once the Peer WLC transitions to the Standby Hot state, -Standby keyword is automatically appended to the Standby WLCs prompt name.
8. Complete these steps in order to check the redundancy status:
a. For WLC 1, go to Monitor > Redundancy > Summary:
b. For WLC 2, go to Console connection:
Note Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
1. On primary controller, disable SSO using the command:
Config redundancy mode disable
The Active and Standby WLCs reboot once this command is executed.
The standby controller, when it comes back after the reboot, has the same IP address on interfaces as the primary controller and all the ports disabled.
2. On the standby controller, re-enter the correct IP addresses corresponding to the management and dynamic interfaces and execute the following command:
Config port adminmode all enable
3. Save the configuration on the controller.
4. To re-enable SSO, execute the command Config redundancy sso on the primary and secondary controllers.
Both controllers reboot and pair up in the SSO mode. The standby will sync its configuration from the primary and come back in Hot-standby mode.
1. Before you configure HA, it is mandatory to have both the controllers' management interface in the same subnet:
2. HA is disabled by default. Before you enable HA, it is mandatory to configure the Redundancy Management IP Address and the Peer Redundancy Management IP Address. Both interfaces should be in the same subnet as the Management Interface. In this example, 9.6.61.21 is the Redundancy Management IP Address for WLC 1, and 9.6.61.23 is the Redundancy Management IP Address for WLC 2. It needs to be configured on WLC 2 where 9.6.61.23 is the Redundancy Management IP Address of WLC 2 and 9.6.61.21 is the Redundancy Management IP Address of WLC 1.
Enter the IP Address for both interfaces, and click Apply.
3. Configure one WLC as Primary and the other WLC as Secondary from the Redundant Unit drop-down list. In this example, WLC 1 is configured as Primary and WLC 2 is configured as Secondary. Once configured, click Apply.
Note You do not need to configure the unit as Secondary if it is a factory ordered HA SKU ordered from release 7.3 onwards. A factory ordered HA SKU is the default Secondary unit and will take the role of the Standby WLC the first time it is paired with an Active WLC with a valid AP Count License.
If you want to convert any existing WLC as a Standby WLC, do so by using the config redundancy unit secondary command in the CLI. This CLI only works if the WLC which is intended to work as standby has some number of permanent license count. This condition is only valid for the 5500 WLC, where a minimum of 50 AP Permanent licenses are needed to be converted to Standby. There is no restriction for other WLCs such as the WiSM2, 7500, and 8500.
4. After the WLCs are configured with Redundancy Management and Peer Redundancy Management IP Address and Redundant Units are configured, it is time to enable SSO. It is important to make sure that physical connections are up between both the controllers (that is, both the WLCs are connected back to back via Redundant Port using an Ethernet cable) and the uplink is also connected to the infrastructure switch and the gateway is reachable from both the WLCs before SSO is enabled.
Once SSO is enabled, it will reboot the WLCs. While it boots, the WLCs negotiate the HA role as per the configuration via Redundant Port. If the WLCs cannot reach each other via the Redundant Port or via the Redundant Management Interface, the WLC configured as Secondary may go in Maintenance Mode. Maintenance Mode is discussed later in this document.
5. In order to enable AP SSO, select Enabled from the drop-down list on both the WLCs, and click Apply. After you enable AP SSO, the WLCs reboot and the default information is populated in other fields like Peer Service Port IP, Peer Redundancy port IP, and so forth.
6. Enabling SSO will reboot the WLCs in order to negotiate the HA role as per the configuration performed. Once the role is determined, configuration is synced from the Active WLC to the Standby WLC via the Redundant Port. Initially WLC configured, as Secondary will report XML mismatch and will download the configuration from Active and reboot again. During the next reboot after role determination, it will validate the configuration again, report no XML mismatch, and will process further in order to establish itself as the Standby WLC.
These are the boot-up logs from both the WLCs:
WLC on first reboot after enabling SSO:
Note Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
WLC 2 on second reboot after downloading XML configuration from Active:
7. After SSO is enabled, WLC is rebooted, and the XML configuration is synced, WLC 1 transitions its state as Active and WLC 2 transitions its state to STANDBY HOT. From this point onwards, GUI/Telnet/SSH for WLC 2 on the management interface will not work, as all the configurations and management should be done from the Active WLC. If required, the Standby WLC (WLC 2, in this case) can only be managed via the Console or Service Port.
Also, once Peer WLC transitions to the STANDBY HOT state, the -Standby keyword is automatically appended to Standby WLCs prompt name.
8. Complete these steps in order to check the redundancy status:
a. For WLC 1, go to Monitor > Redundancy > Summary:
b. For WLC 2, go to Console connection:
Note Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
1. HA between two WLCs can also be enabled from the configuration wizard. It is mandatory to configure the Management IP Address of both the WLCs in same subnet before you enable HA.
2. Once the Management IP is configured, the wizard will prompt you to enable HA. Enter yes in order to enable HA, which is followed by the configuration of the Primary/Secondary Unit and the Redundancy Management and Peer Management IP Address.
3. After you enable HA from the configuration wizard, continue to configure these legacy wizard parameters:
The WLCs will reboot after you save the configuration at the end.
4. While booting, the WLCs will negotiate the HA role as per the configuration done. Once the role is determined, the configuration is synced from the Active WLC to the Standby WLC via the Redundant Port. Initially WLC is configured, as Secondary will report XML mismatch and will download the configuration from Active and reboot again. During the next reboot after role determination, it will validate the configuration again, report no XML mismatch, and process further in order to establish itself as the Standby WLC.
These are the boot-up logs from both the WLCs:
WLC 2 on first reboot after enabling HA:
WLC 2 on second reboot after downloading XML configuration from Active:
Note Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
5. After HA is enabled followed by WLC reboots and XML configuration is synced, WLC 1 will transition its state as Active and WLC 2 will transition its state as STANDBY HOT. From this point onwards GUI/Telnet/SSH for WLC 2 on management interface will not work, as all the configurations and management should be done from Active WLC. If required, the Standby WLC (WLC 2, in this case) can only be managed via the Console or Service Port.
Also, once the Peer WLC transitions to the STANDBY Hot state, the -Standby keyword is automatically appended to the Standby WLCs prompt name.
6. Complete these steps in order to check the redundancy status:
b. For WLC 2, go to Console connection:
Note Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
1. Before you configure HA, it is mandatory to have both the controllers' management interface in the same subnet.
2. Add both the controllers in Cisco Prime using their individual Management IP Address. Once added, both the WLCs can be viewed under Operate > Device Work Center.
3. HA is disabled by default. Before you enable HA, it is mandatory to configure the Redundancy Management IP Address and the Peer Redundancy Management IP Address. Both the interfaces should be in the same subnet as the Management Interface. In this example, 9.6.61.21 is the Redundancy Management IP Address for WLC 1 and 9.6.61.23 is the Redundancy Management IP Address of WLC 2. It needs to be configured on WLC 2 where 9.6.61.23 is the Redundancy Management IP Address of WLC 2 and 9.6.61.21 is the Redundancy Management IP Address of WLC 1.
In order to configure from Cisco Prime, go to Operate > Device Work Center, and select the controller by clicking on the check box in front of the device on which HA should be configured. Once selected, click the Configuration tab, which provides all the options needed to configure the WLC 1, and repeat the steps for WLC 2.
In order to configure the HA parameters for WLC 1, go to Redundancy > Global Configuration, enter the Redundancy and Peer Redundancy-Management IP address, and click Save.
In order to configure the HA parameters for WLC 2, go to Redundancy > Global Configuration, enter the Redundancy and Peer Redundancy-Management IP address, and click Save.
4. Configure one WLC as Primary and the other WLC as Secondary from the Redundant Unit drop-down list. In this example, WLC 1 is configured as Primary and WLC 2 is configured as Secondary. Once configured, click Save.
5. After the WLCs are configured with Redundancy Management and Peer Redundancy Management IP Address, and the Redundant Units are configured, it is time to enable SSO. Once SSO is enabled, it will reboot the WLCs. While booting, the WLCs negotiate the HA role as per configuration via Redundant Port. If the WLCs cannot reach each other via the Redundant Port or via the Redundant Management Interface, the WLC configured as secondary may go in to Maintenance Mode. Maintenance Mode is discussed later in this document.
6. Check the Enabled check box, in order to enable redundancy mode, and click Save. The WLCs will reboot once redundancy mode is enabled.
7. Enabling SSO will reboot the WLCs in order to negotiate the HA role as per the configuration performed. Once the role is determined, the configuration is synced from the Active WLC to the Standby WLC via the Redundant Port. Initially WLC configured, as Secondary will report XML mismatch and will download the configuration from Active and reboot again. In the next reboot after role determination, it will validate the configuration again, report no XML mismatch, and process further in order to establish itself as the Standby WLC.
These are the boot-up logs from both the WLCs:
WLC 2 on first reboot after enabling SSO:
Note Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
WLC 2 on second reboot after downloading XML configuration from Active:
8. After SSO is enabled followed by the WLC reboot and XML configuration is synced, WLC 1 will transition its state as Active and WLC 2 will transition its state as STANDBY HOT. From this point onwards, the GUI/Telnet/SSH for WLC 2 on the management interface will not work, as all the configurations and management should be done from the Active WLC. If required, the Standby WLC (WLC 2, in this case) can only be managed via the Console or Service Port.
Also, once the Peer WLC transitions to the STANDBY Hot state, the -Standby keyword is automatically appended to the Standby WLCs prompt name.
9. Once the HA pairing is formed, Cisco Prime removes/deletes the WLC 2 entry from its database as both the WLCs have the same management IP address. For the network, it is the one box which is active in the network.
Note From this image, it is clear that only WLC 1 (with an IP address of 9.6.61.2 and configured as Primary Unit) is active on Cisco Prime. WLC 2, which was initially added in Cisco Prime with an IP address 9.6.61.3, is deleted from Cisco Prime database after HA pairing is formed.
10. In order to check the redundancy state of the Active WLC from Cisco Prime, go to Device Details > Redundancy > Redundancy States.
The Standby WLC cannot be upgraded directly from the TFTP/FTP server. After executing all scripts, the Active WLC transfers the image to the Standby WLC. Once the Standby WLC receives the image from the Active WLC, it starts executing upgrade scripts. All the logs for image transfer and script execution on the Standby WLC can be seen on the Active WLC.
Note The FUS image can be upgraded while the controllers have HA enabled. The secondary controller will get upgraded just like it does when upgrading the regular code. However, when you initiate the reboot on the primary controller both controllers will be unreachable until the FUS upgrade completes on both the active and the standby in the HA pair. This process will take around 30 to 40 minutes to complete just like in a non-HA FUS upgrade.
1. After the WLCs are configured in the HA setup, the Standby WLC cannot be upgraded directly from the TFTP/FTP server.
2. Initiate upgrade on the Active WLC in the HA setup via CLI/GUI, and wait for the upgrade to finish.
3. Once the Active WLC executes all the upgrade scripts, it will transfer the entire image to the Standby WLC via the Redundant Port.
4. When the Standby WLC receives the image from the Active WLC, it will start executing the upgrade scripts. The transfer of the image to standby and the execution of the upgrade scripts on the Standby WLC can be seen on the Active WLC Console/Telnet/SSH/Http connection.
5. After a successful message of Standby Upgrade is observed on the Active WLC, it is important to issue the show boot command on the Active WLC in order to make sure the new image is set as the primary image.
6. Once verified, initiate primary image pre-download on the Active WLC in order to transfer the new image to all the APs in the network.
7. After pre-image is completed on all the APs, issue the show AP image all command in order to verify that the primary image on the WLC is set as the backup image on APs.
8. Initiate swap option to interchange the backup image as primary on the APs. With this implementation, the WLC's and AP's primary image is set to the new image.
9. Issue the schedule-reset command as per planned outage with the no swap option in order to reset the APs and WLCs so that they can boot with the new image.
10. All the APs will reboot and join the new Active WLC.
11. Issue the show boot, show sysinfo, show ap image all, and show redundancy summary commands in order to verify that both the WLCs and APs have booted with the new image.
transfer upload peer-start
command should be issued on the Active WLC in order to initiate the upload from the Standby WLC. configure redundancy interface address peer-service-port <IP Address>
. This command should be executed from the Active WLC. Also, in order to configure the route on the Standby WLC for out-of-band management on the service port, issue the configure redundancy peer-route add <Network IP Address > <IP Mask> <Gateway>
command from the Active WLC.In the HA setup, the AP's CAPWAP state is maintained on the Active WLC as well as the Standby WLC (only for APs which are in a Run state). That is, Up Time and Association Up Time is maintained on both the WLC, and when switchover is initiated, the Standby WLC takes over the network. In this example, WLC 1 is in an Active state and serving the network, and WLC 2 is in a Standby state monitoring the Active WLC. Although WLC 2 is in Standby state, it still maintains the CAPWAP state of the AP.
Failover for WLCs in HA setup can be categorized into two different sections:
In the case of Box Failover (that is, the Active WLC crashes / system hang / manual reset / force switchover), the direct command is sent from the Active WLC via the Redundant Port as well as from the Redundant Management Interface to the Standby WLC to take over the network. This may take 5-100 msec depending on the number of APs in the network. In the case of power failure on the Active WLC or some crash where the direct command for switchover cannot be sent, it may take 350-500 msec depending on the number of APs in network.
The time it takes for failover in case of power failure on an Active Box also depends on the Keepalive timer configured on the WLC (configured for 100 msec by default). The algorithm it takes to decide the failover is listed here:
In the case of a Network Failover (that is, the Active WLC cannot reach its gateway for some reason), it may take 3-4 seconds for a complete switchover depending on the number of APs in the network.
1. Complete the steps as explained in the configuration section in order to configure HA between two WLCs, and make sure before force switchover is initiated that both the WLCs are paired up as the Active WLC and the Standby WLC.
For WLC 2, go to Console connection:
2. Associate an AP to the WLC and check the status of the AP on both the WLCs. In the HA setup, a mirror copy of the AP database is maintained on both the WLCs. That is, APs CAPWAP state in maintained on Active as well as Standby WLC (only for APs which are in Run state) and when switchover is initiated, the Standby WLC takes over the network. In this example, WLC 1 is an Active WLC, WLC 2 is in a Standby state, and the AP database is maintained on both the WLCs.
3. Create an open WLAN and associate a client to it. The client database is not synced on the Standby WLC, so the client entry will not be present on the Standby WLC. Once the WLAN is created on the Active WLC, it will also be synced to the Standby WLC via the Redundant Port.
4. Issue the redundancy force-switchover command on the Active WLC. This command will trigger a manual switchover where the Active WLC will reboot and the Standby WLC will take over the network. In this case, the client on the Active WLC will be de-authenticated and join back on the new Active WLC.
Note Observe that the prompt in this example changed from 5508-Standby to 5508. This is because this WLC is now the Active WLC and the time taken for AP switchover is 1 msec.
Observe the AP CAPWAP State on WLC 2, which was the Standby WLC initially and is now the Active WLC after switchover. AP Up Time as well as Association Up Time is maintained, and the AP did not go in to the discovery state.
These matrices provide a clear picture of what condition the WLC Switchover will trigger:
configure redundancy mobilitymac <custom mac address>
command. Once configured, you should use this MAC address to form a mobility peer instead of using the system MAC address. Once HA is configured, this MAC cannot be changed.There are few scenarios where the Standby WLC may go into Maintenance Mode and not be able to communicate with the network and peer:
Note The WLC should be rebooted in order to bring it out of Maintenance Mode. Only the Console and Service Port is active in Maintenance Mode.
HA (that is, AP SSO) can be deployed with Secondary and Tertiary Controllers just like today. Both Active and Standby WLCs combined in the HA setup should be configured as primary WLC. Only on failure of both Active and Standby WLCs in the HA setup will the APs fall back to Secondary and further to Tertiary WLCs.
Each WLC has its own unique MAC address, which is used in mobility configuration with an individual controller management IP address. In HA (that is, AP SSO) setup, both the WLCs (Primary and Standby) have their own unique MAC address. In the event of failure of the Primary box and Standby takes over the network if the MAC address of the Primary box is used on another controllers in mobility setup, control path and data path will be down and user has to manually change the MAC to standby MAC address on all the controllers in mobility setup. This is a really cumbersome process as a lot of manual intervention is required.
In order to keep the mobility network stable without any manual intervention and in the event of failure or switchover, the back-and-forth concept of Mobility MAC has been introduced. When the HA pair is set up, by default, the Primary WLC's MAC address is synced as the Mobility MAC address on the Standby WLC which can be seen via the show redundancy summary command on both the controllers.
In this output, captured from a Standby controller, the Mobility MAC address can be observed, which is different from the Standbys own MAC address seen as Unit ID. This MAC address is synced from the Active WLC and should be used in mobility configuration. With this implementation, if the Active WLC goes down or even if it is replaced, the Mobility MAC address is still available and active on the Standby WLC. In case the new controller is introduced in the network because of the replacement of the previous Active WLC, it will transition its state as Standby and the same Mobility MAC address is synced again to the new Standby WLC.
You have the flexibility to configure a custom MAC address as Mobility MAC instead of using the default behavior of using the Active WLC MAC address as Mobility MAC. This can be done using the configure redundancy mobilitymac <custom mac address> command on the Active WLC. Once configured, you should use this MAC address on other controllers in order to form a mobility peer instead of using the Active WLC MAC address. This MAC address should be configured before forming the HA pair. Once the HA pair is formed, the Mobility MAC cannot be changed or edited.
In this topology, the Primary and Standby have their own MAC address. With HA pairing, the Active WLC MAC address is synced as a Mobility MAC address, which is the default behavior if a custom MAC is not configured before HA pairing. Once the Active WLC MAC address is synced as the Mobility MAC address, the same MAC is used in mobility configuration on all the controllers in the mobility setup.
A HA Pair can be established between two WLCs running in these combinations:
– If the new WLC has a higher AP count than the previous, the 90-day counter is reset.
– If the new WLC has a lower AP count than the previous, the 90-day counter is not reset.
– In order to lower AP count after switchover, the WLC offset timer will continue and nagging messages will be displayed after time expiry.
– If the new WLC has a higher AP count than the previous, the 90-day counter is reset.
– If the new WLC has a lower AP count than the previous, the 90-day counter is not reset.
– After switchover to a lower AP count, the WLC offset timer will continue and nagging messages will be displayed after time expiry.
– If the new WLC has a higher AP count than the previous, the 90-day counter is reset.
– If the new WLC has a lower AP count than the previous, the 90-day counter is not reset.
– After switchover to a lower AP count, the WLC offset timer will continue and nagging messages will be displayed after time expiry.
To support High Availability without impacting service, there needs to be support for seamless transition of clients and APs from the active controller to the standby controller. Release 7.5 supports Client Stateful Switch Over (Client SSO) in Wireless LAN controllers. Client SSO will be supported for clients which have already completed the authentication and DHCP phase and have started passing traffic. With Client SSO, a client's information is synced to the Standby WLC when the client associates to the WLC or the client’s parameters change. Fully authenticated clients, i.e. the ones in Run state, are synced to the Standby and thus, client re-association is avoided on switchover making the failover seamless for the APs as well as for the clients, resulting in zero client service downtime and no SSID outage.
1. Back-to-back Redundancy Port (RP) connectivity between the two WLCs, Redundancy Management Interface (RMI) connectivity to check peer and management gateway reachability.
2. RP connectivity with L2 adjacency between the two WLCs, RMI connectivity to check peer and management gateway reachability. This can be within the same or different data centers.
3. Two 5508, 7500 or 8500 connected to a VSS pair. Primary WLC connected to one 6500 and the Stand-by WLC to the other 6500.
Figure 1 Back-to-back RP connectivity
Note In the above equation, 3 is the Keepalive retry count, 100 is the Keepalive timer, and 60 is
3*10 + 3*10 (3 RMI pings to peer + 3 pings to gateway).
configure interface address management 9.5.56.2 255.255.255.0 9.5.56.1
configure interface address redundancy-management 9.5.56.10 peer-redundancy-management 9.5.56.11
configure redundancy unit primary
Configuration on Hot Standby WLC:
configure interface address management 9.5.56.3 255.255.255.0 9.5.56.1
configure interface address redundancy-management 9.5.56.11 peer-redundancy-management 9.5.56.10
Figure 2 RP connectivity via switches
configure interface address management 9.5.56.2 255.255.255.0 9.5.56.1
configure interface address redundancy-management 9.5.56.10 peer-redundancy-management 9.5.56.11
configure redundancy unit primary
Configuration on Hot Standby WLC
configure interface address management 9.5.56.3 255.255.255.0 9.5.56.1
configure interface address redundancy-management 9.5.56.11 peer-redundancy-management 9.5.56.10
Figure 5 WiSM2 connectivity using Redundancy VLAN over L2 network
Configuration on Cat6k for WiSM2
wism service-vlan 192 ( service port VLAN)
wism redundancy-vlan 169 ( redundancy port VLAN)
Figure 6 WiSM2 connectivity using VSS Pair
Figure 7 Active and Standby VSS Pair connected via VSL Link
Figure 8 WiSM2 connectivity using VSS Pair
To support High Availability without impacting the service, there needs to be support for seamless transition of the clients and APs from the active controller to the standby controller. Release 7.5 supports Client Stateful Switch Over (Client SSO) in Wireless LAN controllers. Client SSO will be supported for clients, which have already completed the authentication and DHCP phase and have started passing traffic. With Client SSO, the client’s information is synced to the Standby WLC when client associates or the client parameters change. Fully authenticated clients, i.e. ones in Run state, are synced to the Standby and thus, client re-association is avoided on switchover making the failover seamless for the AP as well as for the client.
1. Before configuring HA it is mandatory to have both the controllers’ management interface in same subnet.
2. HA is disabled by default. Before enabling HA it is mandatory to configure Redundant Management IP address and Peer Redundant Management IP address. Both the interfaces should be in same subnet as Management Interface.
To configure Redundant Management and Peer Redundant Management IP address click Controller tab > Redundancy > Global Configuration and enter IP address in both the fields and then click Apply.
3. Now configure one controller as Primary and another controller as Secondary from Redundant Unit drop-down. In an example below WLC 1 is configured as Primary Unit and WLC 2 is configured as Secondary Unit (will work as HA SKU UDI). While pairing, the controller that is configured as Primary will push its AP Count License to Standby WLC. To configure one controller as Primary unit and second controller as Secondary unit, click Controller tab > Redundancy > Global Configuration and select Primary/Secondary from Redundant Unit drop-down list and then click Apply.
4. After controllers are configured with Redundant Management, Peer Redundant Management IP address and Redundant Units are configured, it is very important to make sure physical connection are up between both the controllers i.e. both the WLCs are connected via Redundant Port using Ethernet cable and uplink is also connected to infrastructure switch and gateway is reachable from both the WLCs. Initiate ping to management interface gateway IP Address from both the controllers and make sure reachability to management gateway is fine.
5. To enable SSO navigate to Controller >Redundancy > Global Configuration and select the Enable option from SSO drop-down list on both the WLCs and click Apply. This step will make controllers reboot.
6. Enabling SSO will reboot controllers to negotiate HA role as per configuration and once the role is determined, configuration is synced from Active to Standby WLC via the redundant port. Initially controller configured as Secondary will report XML mismatch after downloading the configuration from Active and reboot again. In next reboot after role determination it will validate the configuration again and will report no XML mismatch and will process further to establish itself as Standby WLC. Thus, controller configured as Primary will reboot once and controller configured as Secondary will reboot twice.
WLC 2 on first reboot after enabling SSO:
WLC 2 on second reboot after downloading XML configuration from Active:
While WLC2 is booting up, no configuration change is allowed on WLC1:
7. After SSO is enabled followed by controller reboots and XML configuration is synced, WLC 1 will transition its state as Active and WLC 2 will transition its state as Standby HOT. From this point onwards GUI/Telnet/SSH for WLC 2 on management interface will not work, as all the configurations should be done from the Active controller. Standby controller i.e. WLC 2 in this case if required can only be managed via Console or Service Port.
Also once Peer WLC transitions to Standby Hot state, -Standby keyword is automatically appended to Standby WLC’s prompt name.
8. To check the redundancy status
WLC 1 -> Click Monitor > Redundancy > Summary:
WLC 1 -> show redundancy summary:
WLC 2 -> show redundancy summary:
1. At this stage both the controllers are paired up in HA setup. Any configuration done on Active will be synced to Standby controller via redundant port. Check the WLAN summary and Interface summary on standby WLC from console connection.
2. In High Availability setup, APs’ CAPWAP state in maintained on Active as well as Standby controller (only for APs which are in Run state) i.e. UP time and Associated UP time is synced from the active to the standby controller. In an example below WLC 1 is an Active state and serving the network and WLC 2 is in Standby state monitoring active controller. Although WLC 2 is in standby state it still maintains CAPWAP state of AP.
Observe the AP UP Time and Association UP Time on Active WLC
Observe the AP Uptime and Association UP Time on Standby WLC will be in sync with active WLC.
3. In case of Box Failover i.e. Active controller crashes / system hang / manual reset / force switchover direct command is sent from Active controller via Redundant Port as well as from Redundant Management Interface to Standby controller to take over the network. Failover may take ~2-360 millisecond depending on number of APs/Clients on the active controller. In case of power failure on Active WLC or some crash where direct command for switchover cannot be sent to the standby controller, it may take ~360 – 990 msec depending upon number of APs/Clients on the active controller and the Keepalive timer configured. The default Keepalive timer is 100 milliseconds. Make sure that default RTT latency is less than or equal to 80 msec.
4. With release 7.5 as part of Client SSO, the client database is also synced to standby WLC so Run state client entries will be present on Standby WLC.
WLC 1-> Console/Telnet/SSH Connection:
Client entry is present on Active WLC.
Client entry is present on Standby WLC.
5. PMK cache is also synced between the two controllers
1. Issue a command redundancy force-switchover on Active controller. This command will trigger manual switchover where Active controller will reboot and Standby controller will take over the network. In this case Run state client on Active WLC will not be de-authenticated. The command save config is initiated before redundancy force-switchover command.
Observe the change in prompt in above screen capture.
Observe the AP CAPWAP State on WLC 2 which was standby initially and is Active now after switchover. AP uptime as well as Association UP Time is maintained and AP did not go in discovery state.
2. Also notice client connectivity when switchover is initiated. Client will be not be de-authenticated.
Ping from wireless client to its gateway IP Address and management IP Address during switchover shows minimal loss.
3. To check the redundancy status
WLC 1 -> Console connection issue a command show redundancy summary:
WLC 2 -> Console connection issue a command show redundancy summary:
WLC 2-> Click on Monitor > Redundancy > Summary:
4. Initiate a force switchover again on current active WLC.
WLC, which was configured as Primary Unit, should now be active and WLC, which was configured as Secondary Unit i.e., WLC 2 should be in Hot Standby State.
WLC 1: Make sure Local state should be Active and Unit should be Primary on WLC 1 after switchover:
Observe the switchover history. WLC maintains 10 switchover histories with switchover reason.
High Availability in release 8.0 introduces enhancements and improvements to the High Availability feature-set. The following enhancements are captured in this section:
High Availability in release 8.0 also introduces new features enabling SSO such as:
Currently, the controller does not provide any indication for the completion of Bulk Sync configuration once it is initiated. The Bulk Sync can be verified only by user observation and by manually checking the number of clients synced to the standby WLC. As part of this feature, a mechanism is provided to convey the status of Bulk Sync (both AP and client sync) when standby WLC comes up.
A new field called BulkSync Status is added in the GUI under Controller > Redundancy > Summary. This field points to the status of the bulk sync to the standby WLC and the status can be Pending/In-progress/Complete.
The output of the CLI command show redundancy summary
also displays the Bulk Sync status, which can be Pending/In-progress/Complete as shown below while pairing with the standby controller.
When the standby controller is booting up, the BulkSync status shows Pending.
Figure 10 BulkSync Status—Pending
Once the standby controller completes, the boot-up process and the bulk sync starts, the status changes to In-Progress.
Figure 11 BulkSync Status—In-Progress
When the bulk sync process is complete, the BulkSync status changes to Complete.
Figure 12 BulkSync Status—Complete
As HA plays a major role in avoiding network outage, it should also be pertinent to be able to debug the state changes on the boxes at the time of SSO or at a later point in time.
The following new categories of statistics are introduced under Monitor > Redundancy > Statistics:
Figure 13 Redundancy Statistics GUI
The Infra statistics contain RF Client details and Sanity Counters as shown in Figure 14.
Figure 14 Redundancy Statistics—Infra
Figure 15 Redundancy Statistics —Transport
The Heartbeat debugs include events of reception of heartbeats, loss of heartbeats, and subsequent actions related to them.
Figure 16 Redundancy Keep-alive Statistics
The HA system monitors management gateway reachability to reduce network outage.
On the Standby controller, serviceability debugs related to the gateway reachability of the active controller and standby controller, their health states, and actions taken based on this information is reported. While on the active controller, the reachability of active WLC to the gateway alone is reported.
Figure 17 Redundancy GW-Reachability Statistics
Figure 18 Redundancy Config-Sync Statistics
The following debug/show CLI commands are introduced for this feature:
1. debug redundancy infra detail/errors/event
2. debug redundancy transport detail/errors/events/packet
3. debug redundancy keepalive detail/errors/events
4. debug redundancy gw-reachability detail/errors/events
5. debug redundancy config-sync errors/events/detail
6. debug redundancy ap-sync errors/events/detail
7. debug redundancy client-sync errors/events/detail
8. debug redundancy mobility events/errors/detail
9. show redundancy infra statistics
10. show redundancy transport statistics
11. show redundancy keepalive statistics
12. show redundancy gw-reachability statistics
13. show redundancy config-sync statistics
To address the variable network latencies in different customer deployment scenarios, keep-alive and peer-search parameters are made configurable. As part of this enhancement, the maximum number of Keepalives between active and standby controllers to trigger a failover is now configurable. Also, peer-search timer and keep-alive timer are modified to support an extended range.
The following new CLI command is added to configure the number of redundancy keep-alive retries in the range of 3 to 10.
Figure 19 Redundancy retries
CLI Command
The existing CLI command config redundancy timer keep-alive-timer of keep-alive timer is modified to support keep-alive timer from 100 to 1000 msecs.
The existing CLI command config redundancy timer peer-search-timer of peer-search timer is modified to support peer-search timer from 60 to 300 secs.
Figure 20 Redundancy timer CLI Command
The following CLI is introduced to view the redundancy keep-alive-retry value.
Figure 21 Show redundancy retries CLI Command
Use the show redundancy timers command to view the peer-search-timer and keep-alive-timer values.
Figure 22 Show redundancy timers
CLI Command
Use the show redundancy detail command to display the keep-alive and peer-search timeout values.
Figure 23 Show redundancy detail
CLI Command
The keep-alive timer, keep-alive retries, and peer-search timer can also be configured and viewed from the Controller > Redundancy > Global Configuration page in the GUI.
Figure 24 Redundancy Global Configuration GUI
Prior to release 8.0, ICMP ping is used to heart-beat with the peer WLC over the Redundancy Management Interface. As part of this feature for release 8.0, ICMP ping is replaced with a UDP message.
This will benefit due to the following factors:
It is recommended to tag the RMI and management interfaces to avoid false Switchovers. Tagging of the RMI and management interfaces is now mandatory in release 8.0 to pair WLCs in SSO mode.
Prior to release 8.0 when the standby controller looses reachability to the “Default Gateway” or “Peer RP”, the controller reboots and checks that condition while booting up and enters into the MTC mode. With this feature, the standby WLC will enter into the MTC mode “on-the-fly” without rebooting when such error scenarios occur. Once Peer-RP and the default gateway reachability is restored, the MTC mode auto-recovery mechanism introduced in release 7.6, will reboot the WLC and pair it with the active WLC. This mechanism is applicable only to the standby WLC. The active controller will still reboot before going to MTC mode.
As part of this enhancement, the gateway (GW) reachability check mechanism is modified to avoid false positives and it is also modified for the ideal time to start checking for gateway reachability once the controller boots up.
Prior to release 8.0, the “GW reachability check” is performed during Role negotiation. In release 8.0 and later, during Role negotiation, GW reachability check is not performed and is initiated only after the HA Pair-Up is complete.
Also, it is observed that certain Switch/Router configurations rate limits ICMP ping packets or drop them altogether. To avoid such conditions triggering false-positives, the new design ensures not to take switchover decisions purely based on ICMP ping losses. By the modified logic, upon 6 consecutive ping drops, an ARP request is sent to the GW IP address. A successful response to this request is considered as the GW being reachable.
Currently during the HA pairing process, once the active-standby role is determined, the configuration is synced from the active WLC to the standby WLC via the Redundancy Port. If the configuration is different, the secondary WLC reports XML mismatch and downloads the configuration from the active controller and reboots again. In the next reboot after role determination, it validates the configuration again, reports no XML mismatch, and process further in order to establish itself as the Standby WLC.
With this feature enhancement, the XMLs are sent from the to-be-Active to to-be-Standby controller at the time of initialization, just before the validation of the XMLs. This avoids the extra step of comparison and reboot since no other modules are initialized yet, resulting in faster pair up of Active and Standby WLCs.
As seen in the boot logs below, there are no comparison of XMLs and no reboot of standby WLC.
Figure 25 Standby WLC bootup log
Prior to release 8.0, configuration of “Internal DHCP Server” is not allowed on HA enabled controllers because the internal DHCP server data is not synced to the standby WLC. In release 8.0 and later, “Internal DHCP Server” is configured on HA enabled controllers and this data is synced to the standby WLC so that soon after a switchover, the “Internal DHCP Server” on the new active controller starts serving clients.
To configure the Internal DHCP server using the GUI, navigate to Controller > Internal DHCP Server
Figure 26 Internal DHCP Server GUI
The same is synced to the standby controller and is verified by executing the CLI command show dhcp summary
Figure 27 show dhcp summary on Active and Standby WLC
As part of this enhancement, Static CAC method bandwidth allocation parameters for Voice and Video and Call Statistics are synced to the Standby WLC, so that soon after a switchover, respective information is available on the new active controller that will be used for call admission control.
Release 7.5 did not provide SSO support for sleeping clients. The sleeping client database was not synced to the standby controller, which caused the sleeping clients to re-authenticate after a switchover occurred. With this release, sleeping client database is synced to the standby controller, allowing sleeping clients to avoid web re-authentication if they wake up within the sleeping client timeout interval.
The CLI command show custom-web sleep-client summary is used to verify the sleeping client database sync between the active and standby WLC.
Figure 28 Sleeping Client Database on Primary WLC
Figure 29 Sleeping Client Database on Standby WLC
Figure 30 Sleeping Client Details on Active and Standby WLC
Prior to release 8.0, when a switchover occurs on an HA pair, OEAP 600 APs restarts the CAPWAP tunnel and joins back the new active controller, and all the connected clients are de-authenticated. As a part of this feature, OEAP 600 APs ensure not to reset their CAPWAP tunnel. Also, clients continue their connection with the new active controller in a seamless manner.
As shown below, the output of show ap summary and show client summary command on the active and standby controllers displays the AP and client database sync.
Figure 31 OEAP 600 AP on Active WLC
Figure 32 OEAP 600 AP Sync to Standby WLC
Figure 33 Clients on Active WLC
Figure 34 Client Sync to Standby WLC