Introduction
This document describes how to troubleshoot when Cisco Catalyst Center does not show any Assurance data for a Catalyst 9800 Series WLC.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Use of the Catalyst Center maglev CLI
- Basic Linux foundation
- Knowledge of certificates on Catalyst Center and on the Catalyst 9800 platform
Components Used
The information in this document is based on these software and hardware versions:
- Catalyst Center appliance 1st or 2nd generation with software version 1.x or 2.x with Assurance package
- Catalyst 9800 Series Wireless LAN Controller (WLC)
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Note: While this document was initially written for Catalyst Center 1.x, most of it is valid for Catalyst Center 2.x.
Note: The Catalyst 9800 WLC must be already discovered by Catalyst Center and assigned to a site, and must run a compatible Cisco IOS® XE version. For more details about interoperability, refer to the Catalyst Center Compatibility Matrix.
Background Information
At the moment of the discovery process, Catalyst Center pushes the next configuration to the WLC.
Note: This example is from a Catalyst 9800-CL Cloud Wireless Controller. Some details can differ when you use a physical Catalyst 9800 Series appliance; X.X.X.X is the Virtual IP (VIP) address of the Catalyst Center enterprise interface and Y.Y.Y.Y is the management IP address of the WLC.
crypto pki trustpoint sdn-network-infra-iwan
enrollment pkcs12
revocation-check crl
rsakeypair sdn-network-infra-iwan
crypto pki trustpoint DNAC-CA
enrollment mode ra
enrollment terminal
usage ssl-client
revocation-check crl none
source interface GigabitEthernet1
crypto pki certificate chain sdn-network-infra-iwan
certificate 14CFB79EFB61506E
3082037D 30820265 A0030201 02020814 CFB79EFB 61506E30 0D06092A 864886F7
<snip>
quit
certificate ca 7C773F9320DC6166
30820323 3082020B A0030201 0202087C 773F9320 DC616630 0D06092A 864886F7
<snip>
quit
crypto pki certificate chain DNAC-CA
certificate ca 113070AFD2D12EA443A8858FF1272F2A
30820396 3082027E A0030201 02021011 3070AFD2 D12EA443 A8858FF1 272F2A30
<snip>
quit
telemetry ietf subscription 1011
encoding encode-tdl
filter tdl-uri /services;serviceName=ewlc/wlan_config
source-address Y.Y.Y.Y
stream native
update-policy on-change
receiver ip address X.X.X.X 25103 protocol tls-native profile sdn-network-infra-iwan
telemetry ietf subscription 1012
<snip - many different "telemetry ietf subscription" sections - which ones depends on
Cisco IOS version and Catalyst Center version>
network-assurance enable
network-assurance icap server port 32626
network-assurance url https://X.X.X.X
network-assurance na-certificate PROTOCOL_HTTP X.X.X.X /ca/ pem
Troubleshoot No Assurance Data from WLC on Catalyst Center
Step 1. Verify that the WLC is reachable and managed in the Catalyst Center inventory.
If the WLC is not in Managed status, you must fix the reachability or provision issue before you continue.
Tip: Check the inventory-manager, spf-device-manager, and spf-service-manager logs in order to identify the failure.
Step 2. Verify that Catalyst Center pushes all the necessary configurations to the WLC.
Ensure the configuration mentioned in the Background Information section was pushed to the WLC with these commands:
show run | section crypto pki trustpoint DNAC-CA
show run | section crypto pki trustpoint sdn-network-infra-iwan
show run | section network-assurance
show run | section telemetry
Known issues:
- Cisco bug ID CSCvs62939 - Cisco DNA Center does not push telemetry config to 9xxx switches after discovery.
- Cisco bug ID CSCvt83104 - eWLC Assurance config push failure if Netconf candidate datastore exists on the device.
- Cisco bug ID CSCvt97081 - eWLC DNAC-CA certificate provisioning fails for device discovered by DNS name.
Logs to verify:
- dna-wireless-service - for DNAC-CA certificate and telemetry configuration.
- network-design-service - for sdn-network-infra-iwan certificate.
Step 3. Verify that the necessary certificates get created on the WLC.
Ensure that the certificates get created correctly on WLC with these commands:
show crypto pki certificates DNAC-CA
show crypto pki certificates sdn-network-infra-iwan
Known issues and limitations:
- Cisco bug ID CSCvu03730 - eWLC is unmonitored in Cisco DNA Center because the sdn-network-infra-iwan certificate is not installed (the root cause is that the pki-broker client cert is expired).
- Cisco bug ID CSCvr44560 - ENH: Add support for CA certificates expiring after 2099 for IOS-XE
- Cisco bug ID CSCwc99759 - ENH: Add support for 8192-bit RSA Certificate Signature
Step 4. Verify the telemetry connection status.
Ensure that the telemetry connection is in the Active state on the WLC with this command:
wlc-01#show telemetry internal connection
Telemetry connection
Address Port Transport State Profile
------------------------------------------------------------------
X.X.X.X 25103 tls-native Active sdn-network-infra-iwan
Or from Cisco IOS XE Release 17.7 and later:
wlc-01#show telemetry connection all
Telemetry connections
Index Peer Address Port VRF Source Address State State Description
----- -------------------------- ----- --- -------------------------- ---------- --------------------
9825 X.X.X.X 25103 0 Y.Y.Y.Y Active Connection up
The X.X.X.X IP address must be the Catalyst Center Enterprise interface. If Catalyst Center is configured with VIPs, this must be the VIP of the Enterprise interface. If the IP address is correct and the state is Active, proceed to the next step.
If the state is Connecting, then the Hypertext Transfer Protocol Secure (HTTPS) connection from the WLC to Catalyst Center was not established successfully. There can be many different reasons for this, the most common are listed next.
4.1. The Catalyst Center VIP is not reachable from the WLC or is in DOWN status.
- On a single node with VIP, the VIP goes down when the cluster interface goes down. Verify that the cluster interface is connected.
- Verify that the WLC has connectivity to the Enterprise VIP (ICMP/ping).
- Verify that the Catalyst Center Enterprise VIP is in the UP state, with this command:
ip a | grep en
.
- Verify that the Catalyst Center Enterprise VIP is properly configured with this command:
etcdctl get /maglev/config/cluster/cluster_network
.
4.2. The WLC is in High Availability (HA); Assurance does not work after failover.
This can occur if the HA is not formed by the Catalyst Center. In that case: remove the WLC from Inventory, break the HA, discover both WLCs, and let Catalyst Center form the HA.
Note: This requirement can change in later Catalyst Center versions.
4.3. Catalyst Center did not create the DNAC-CA trustpoint and certificate.
- Check Step 2 and Step 3 to fix this problem.
4.4. Catalyst Center did not create the sdn-network-infra-iwan
trustpoint and certificate.
- Check Step 2 and Step 3 to fix this problem.
4.5. Catalyst Center did not push the Assurance configuration.
- The command
show network-assurance summary
shows Network-Assurance as Disabled
:
DC9800-WLC#show network-assurance summary
----------------------------------------------------
Network-Assurance : Disabled
Server Url :
ICap Server Port Number :
Sensor Backhaul SSID :
Authentication : Unknown
- Ensure the WLC has Device Controllability enabled, as this is required for Catalyst Center to push the configuration. Device Controllability can be enabled in the Discovery process, or once the WLC is on the Inventory and managed by Catalyst Center. Navigate to the
Inventory
page. Select Device > Actions > Inventory > Edit Device > Device Controllability > Enable.
4.6. Catalyst Center does not push the telemetry subscription configuration.
- Ensure that the WLC has the subscriptions with the
show telemetry ietf subscription all
command.
- If not, check Step 2 and Step 3 in order to fix this problem.
4.7. The TLS handshake between the WLC and Catalyst Center fails because the Catalyst Center certificate cannot be validated by the WLC.
This could be due to many reasons, the most common ones are listed here:
4.7.1. The Catalyst Center certificate is expired, or revoked, or does not have the Catalyst Center IP address in the Subject Alternate Name (SAN).
4.7.2. The revocation check fails because the Certificate Revocation List (CRL) cannot be retrieved.
- There can be many reasons for the CRL retrieval to fail; such as a DNS failure, firewall issue, connectivity issue between the WLC and the CRL Distribution Point (CDP), or one of these known issues:
- Cisco bug ID CSCvr41793 - PKI: CRL retrieval does not use HTTP Content-Length.
- Cisco bug ID CSCvo03458 - PKI, revocation check crl none, does not fallback if CRL is not reachable.
- Cisco bug ID CSCue73820 - PKI debugs not clear about CRL parse failure.
- As a workaround, configure
revocation-check none
under the DNAC-CA trustpoint.
4.7.3. Certificate error "Peer certificate chain is too long to be verified".
Note: This produces a list of the certificates in the trust chain (PEM encoded), so each certificate starts with -----BEGIN CERTIFICATE-----. Refer to the URL mentioned in the Workaround section, and execute the steps to configure the DNAC-CA certificate, but do not import the root CA certificate. Instead, import the certificate of the problematic CA.
4.7.4. WLC certificate expired.
- When the Catalyst Center version is 1.3.3.7 or earlier, the WLC certificate could have expired. When the Catalyst Center version is 1.3.3.8 or later (but not 2.1.2.6 or later) then this can still be an issue if the certificate expired before the upgrade from Version 1.3.3.7 or earlier.
- Check the validity end date in the output of the
show crypto pki certificates sdn-network-infra-iwan
command.
4.8. The collector-iosxe service on Catalyst Center does not accept the connection from the WLC because it was not notified of the new device by the inventory-manager service.
- In order to check the list of devices known by iosxe-collector, enter this command on the Catalyst Center CLI:
curl -s http://collector-iosxe-db.assurance-backend.svc.cluster.local:8077/api/internal/device/data
- In order to get just the list of hostnames and IP addresses, parse the output with jq with this command:
On Catalyst Center 1.3 and later:
curl -s 'http://collector-iosxe-db.assurance-backend.svc.cluster.local:8077/api/internal/device/data' | jq '.devices[] | .hostName, .mgmtIp'
On Catalyst Center 1.3.1 and earlier:
curl -s
http://collector-iosxe-db.assurance-backend.svc.cluster.local:8077/api/internal/device/data
' | jq '.device[] | .hostNa
me, .mgmtIp
- If this list does not contain the WLC, then restart the collector-iosxe service and confirm whether this solves the issue.
- If a restart of collector-iosxe alone does not help, a restart of the collector-manager service can help to solve this issue.
Tip: In order to restart a service, enter magctl service restart -d <service_name>
.
- If the output of the command
show telemetry internal connection
is still Connecting, tail the collector-iosxe
logs for the error:
Tip: In order to tail a log file, enter the magctl service logs -rf <service_name>
command. In this case, magctl service logs -rf collector-iosxe | lql.
.
40 | 2021-04-29 08:09:15 | ERROR | pool-15-thread-1 | 121 | com.cisco.collector.ndp.common.KeyStoreManager | java.lang.IllegalArgumentException: Exception setting server keystore
at java.util.Base64$Decoder.decode0(Base64.java:714)
- If you see this error, then open the certificate which was added to the Catalyst Center, both its .key and .pem (certificate chain) files in Notepad++. In Notepad++, navigate to
View > Show Symbol > Show All Characters.
- If you have something like this:
Then navigate to:
And save the certificates.
- Add them again to the Catalyst Center and check if the
show telemetry internal connection
command now shows Active.
4.9. Related defects:
- Cisco bug ID CSCvs78950 - eWLC to Wolverine cluster telemetry connection in Connecting state.
- Cisco bug ID CSCvr98535 - Cisco DNA Center does not configure HTTP source interface for PKI - eWLC telemetry stays Connecting.
Step 5. The telemetry state is active, but still, no data is seen in Assurance.
Verify the current status of the telemetry internal connection with this command:
dna-9800#show telemetry internal connection
Telemetry connection
Address Port Transport State Profile
------------------------------------------------------------------
X.X.X.X 25103 tls-native Active sdn-network-infra-iwan
Possible defects:
- Cisco bug ID CSCvu27838 - No wireless assurance data from 9300 with eWLC.
- Cisco bug ID CSCvu00173 - Assurance API route not registered after upgrade to 1.3.3.4 (not specific to eWLC).
Workaround
If some or all of the required configuration is not in the WLC, try to determine why the configuration is not present. Check the relevant log files if there is a match for a defect. After that, consider these options as a workaround.
Catalyst Center Version 2.x
On the Catalyst Center GUI, navigate to the Inventory
page. Choose the WLC > Actions > Telemetry > Update Telemetry Settings > Force Configuration Push > Next > Apply.
After that, wait some time until the WLC finishes the resynchronization process. Verify that Catalyst Center pushes the configuration mentioned in the Background Information section of this document and verify that the Assurance configuration is present on the WLC with the show network-assurance summary
command.
Catalyst Center Version 1.x
This can also be used for Catalyst Center 2.x if the previous GUI method still does not have the desired effect.
- The
sdn-network-infra-iwan
trustpoint and/or certificate is missed.
Contact the Cisco Technical Assistance Center (TAC) in order to manually install the Catalyst Center Assurance certificates and subscriptions.
- Network-assurance configuration is not present.
Ensure the Catalyst Center enterprise VIP address is reachable from the WLC. Then configure the section manually as shown in the next example:
conf t
network-assurance url https://X.X.X.X
network-assurance icap server port 32626
network-assurance enable
network-assurance na-certificate PROTCOL_HTTP X.X.X.X /ca/ pem
Note: On the fifth line, note the space between X.X.X.X and /ca/ and also the space between /ca/ and pem.