THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Affected Software Product | Affected Release | Affected Release Number | Comments |
---|---|---|---|
Cisco Mobility Express | 8.10 | 8.10.105.0, 8.10.112.0, 8.10.113.0, 8.10.121.0, 8.10.122.0, 8.10.130.0, 8.10.142.0, 8.10.151.0, 8.10.162.0, 8.10.171.0, 8.10.181.0 | |
Cisco Mobility Express | 8.2 | 8.2.100.0, 8.2.110.0, 8.2.111.0, 8.2.121.0, 8.2.130.0, 8.2.141.0, 8.2.151.0, 8.2.160.0, 8.2.161.0, 8.2.164.0, 8.2.166.0, 8.2.170.0 | |
Cisco Mobility Express | 8.3 | 8.3.102.0, 8.3.111.0, 8.3.112.0, 8.3.121.0, 8.3.122.0, 8.3.130.0, 8.3.131.0, 8.3.132.0, 8.3.133.0, 8.3.135.0, 8.3.140.0, 8.3.141.0, 8.3.143.0, 8.3.150.0 | |
Cisco Mobility Express | 8.4 | 8.4.100.0 | |
Cisco Mobility Express | 8.5 | 8.5.103.0, 8.5.105.0, 8.5.110.0, 8.5.120.0, 8.5.131.0, 8.5.135.0, 8.5.140.0, 8.5.151.0, 8.5.160.0, 8.5.161.0, 8.5.171.0, 8.5.182.0 | |
Cisco Mobility Express | 8.6 | 8.6.101.0 | |
Cisco Mobility Express | 8.7 | 8.7.102.0, 8.7.106.0 | |
Cisco Mobility Express | 8.8 | 8.8.100.0, 8.8.111.0, 8.8.120.0, 8.8.125.0, 8.8.130.0 | |
Wireless LAN Controller Software | 8.10 | 8.10.105.0, 8.10.112.0, 8.10.113.0, 8.10.121.0, 8.10.122.0, 8.10.130.0, 8.10.142.0, 8.10.151.0, 8.10.162.0, 8.10.171.0 | |
Wireless LAN Controller Software | 8.2 | 8.2.100.0, 8.2.110.0, 8.2.111.0, 8.2.121.0, 8.2.130.0, 8.2.141.0, 8.2.151.0, 8.2.160.0, 8.2.161.0, 8.2.164.0, 8.2.166.0, 8.2.170.0 | |
Wireless LAN Controller Software | 8.3 | 8.3.102.0, 8.3.108.0, 8.3.111.0, 8.3.112.0, 8.3.121.0, 8.3.122.0, 8.3.130.0, 8.3.131.0, 8.3.132.0, 8.3.133.0, 8.3.135.0, 8.3.140.0, 8.3.141.0, 8.3.143.0, 8.3.150.0 | |
Wireless LAN Controller Software | 8.4 | 8.4.100.0 | |
Wireless LAN Controller Software | 8.5 | 8.5.103.0, 8.5.105.0, 8.5.110.0, 8.5.120.0, 8.5.131.0, 8.5.135.0, 8.5.140.0, 8.5.151.0, 8.5.160.0, 8.5.161.0, 8.5.171.0, 8.5.182.0, 8.5IRCM | 8.5.103.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.105.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.110.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.120.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.131.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.135.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.140.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.151.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.160.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.161.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.171.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5.182.0: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 8.5IRCM: 8.5IRCM software versions affected include: 8.5.164.0; 8.5.164.216; 8.5.176.0; 8.5.176.1; 8.5.176.2; 8.5.182.104 |
Wireless LAN Controller Software | 8.6 | 8.6.101.0 | |
Wireless LAN Controller Software | 8.7 | 8.7.102.0, 8.7.106.0 | |
Wireless LAN Controller Software | 8.8 | 8.8.100.0, 8.8.111.0, 8.8.120.0, 8.8.125.0, 8.8.130.0 | |
Wireless LAN Controller Software | 8.9 | 8.9.100.0, 8.9.111.0 |
Defect ID | Headline |
CSCwb16632 | AireOS SMART Licensing registration/renewal due to SSL certificate problem |
CSCwa55717 | CBW access points / Mobility Express cannot download software from Cisco.com |
CSCwd11225 | Cisco.com upgrade fails in Mobility Express with "Error parsing response from server" |
For affected versions of the AireOS software, some Secure Sockets Layer (SSL) certificates issued from the QuoVadis root certificate authority (CA) trust chain before March 31, 2021 cannot be renewed from this CA. Once those certificates expire on devices or are removed from the Cisco cloud servers, functions such as Smart Licensing and Smart Call Home will fail to establish secure connections to Cisco and might not operate properly.
Smart Licensing and Smart Call Home have replaced the QuoVadis root certificate with one from IdenTrust. Normally, the new root certificate will automatically be downloaded from tools.cisco.com by the Wireless LAN Controller (WLC). However, if the network path between the WLC and tools.cisco.com blocks access to TCP ports 80 and 443 on tools.cisco.com, the certificate download will fail and the WLC will be unable to contact Smart Licensing or Smart Call Home.
Additionally, for affected versions of Mobility Express software, direct software download from Cisco.com does not work.
The QuoVadis Root CA 2 Public Key Infrastructure (PKI) used by AireOS software to issue SSL certificates is subject to an industry-wide issue that affects revocation abilities. Due to this issue, no new QuoVadis Root CA 2 certificates will be issued or renewed by Cisco after March 31, 2021. This affects certificate renewals on devices, Cisco cloud servers, and third-party services.
Certificates issued before the QuoVadis Root CA 2 was decommissioned will continue to be valid. However, the certificates will not renew when they expire on either the device or the Cisco cloud server. This will cause functions such as Smart Licensing and Smart Call Home to fail to establish secure connections to Cisco cloud servers.
This table shows a summary of the QuoVadis Root CA 2 certificate expiration dates for affected Cisco services.
Cisco Cloud Server | QuoVadis Certificate Expiration Date | Affected Services |
---|---|---|
tools.cisco.com | February 5, 2022 |
|
api.cisco.com cloudsso.cisco.com dl.cisco.com |
February 5, 2022 | Mobility Express software download |
Additionally, the Mobility Express software download changed the API and is non-backward compatible, so connecting to Cisco.com servers gives a connection failure.
Expiration of the QuoVadis Root CA 2 certificates affects these services with the associated symptoms.
Affected Services | Symptoms for Affected Services |
---|---|
Smart Licensing | Failure to connect to the server (Details are provided in this section) |
Smart Call Home | Failure to connect to the server and the Call-Home HTTP request fails |
Mobility Express | Failure to download software from Cisco.com |
AireOS Symptoms
For affected versions of AireOS, devices will be unable to connect to the Smart Licensing services hosted by Cisco. Smart licenses might fail with these messages:
"Fail to send out Call Home HTTP message."
"Unknown CA."
Note: For additional information, refer to the Cisco Smart Licensing Guide for your specific version of AireOS software.
In order to determine whether or not you are affected, first find out if you use Smart Licensing and/or Smart Call Home. Enter these commands:
show call-home config-local
show license summary
If either of these services are enabled, you are susceptible. Follow the instructions in the Workaround/Solution section.
In order to verify that you are impacted by this issue, enable these debugs and show outputs on the WLC:
debug license core all enable
debug license events enable
debug license errors enable
debug license info enable
show license summary
show license tech-support
show license all
These error logs might be observed on the affected device:
*Fri Mar 18 02:06:11.597 UTC: CH-LIB-TRACE: ch_pf_curl_head_init[111], init msg header
* SSL certificate problem: self signed certificate in certificate chain
*Fri Mar 18 02:06:13.432 UTC: CH-LIB-ERROR: ch_pf_curl_send_msg[483], failed to perform, err code 60, err string "Error"
*Fri Mar 18 02:06:13.432 UTC: CH-LIB-TRACE: ch_pf_http_unlock[215], unlock http mutex.
*Fri Mar 18 02:06:13.432 UTC: CH-LIB-TRACE: ch_pf_send_http[263], send http msg, result 35
In order to check the show license status, enter the show license status
command.
(Cisco Controller) >show license status
Smart Licensing is ENABLED
Utility:
Status: DISABLED
Data Privacy:
Sending Hostname: yes
Callhome hostname privacy: DISABLED
Smart Licensing hostname privacy: DISABLED
Version privacy: DISABLED
Transport:
Type: Callhome
Registration:
Status: REGISTERING - REGISTRATION IN PROGRESS
Export-Controlled Functionality: NOT ALLOWED
Initial Registration: FAILED on Mar 02 2022 13:13:48 CET
Failure reason: Fail to send out Call Home HTTP message.
Next Registration Attempt: Mar 02 2022 13:30:15 CET
License Authorization:
Status: EVAL MODE
Evaluation Period Remaining: 89 days, 23 hours, 58 minutes, 32 seconds
Export Authorization Key:
Features Authorized:
<none>
Cisco Mobility Express Symptoms
Software download from Cisco.com to Mobility Express does not work. If you run Mobility Express earlier than 8.10.181.0, the download icon in the home page (a down-arrow inside a circle) is red, and hovering over the icon shows this error:
Connection failure: 60. Peer certificate cannot be authenticated with given CA certificates.
If you run Mobility Express 8.10.181.0 or later, the msglog (ctrl/msg.txt in the Support Bundle) shows this error:
*emWeb: Sep 23 18:38:57.420: check latest version failed: Error parsing response from server
Cisco has migrated from the QuoVadis Root CA 2 to the IdenTrust Commercial Root CA 1 for SSL certificates. Cisco recommends one of these two options to add the new IdenTrust Commercial Root CA 1 certificate to the AireOS.
Solution for AireOS WLCs (Preferred)
For AireOS devices, upgrade to one of the software versions shown in the table in order to resolve the root CA certificate issue for affected platforms.
WLC Model | Fixed Version |
---|---|
|
|
Note: WLCs not listed here are not affected by this issue.
Workaround for AireOS WLCs
Configure your network to allow your WLC to access tools.cisco.com via TCP ports 80 and 443. Enter the show network summary
command to see the WLC's DNS server. Query the DNS server that is configured on your WLC to find out the IP address(es) to which tools.cisco.com resolves.
Once the WLC has access to tools.cisco.com, it will automatically download the IdenTrust Commercial Root CA 1 certificate.
When the certificate update happens, there is no reboot required.
In order to confirm that the IdenTrust Commercial Root CA 1 certificate is installed on the WLC, enter the grep include IdenTrust "show certificate all"
command.
Workaround for Mobility Express
Cisco no longer supports the Cisco.com download method for Mobility Express. Download new Mobility Express software from software.cisco.com via a browser, then load that software into Mobility Express via TFTP, SFTP, or HTTP. See Cisco Mobility Express User Guide, Cisco Wireless Release 8.10 for more details.
These products are affected:
Cisco has created a web page to provide customers and partners with additional information on this issue. Consult the QuoVadis Root CA 2 Decommission page for a full list of products affected, associated Field Notices, and frequently asked questions.
Version | Description | Section | Date |
1.3 | Updated the Problem Symptom and Workaround/Solution Sections | — | 2023-FEB-06 |
1.2 | Updated the Title, Description, Problem Symptoms, Workaround/Solution, and Defect ID Sections | — | 2022-OCT-11 |
1.0 | Initial Release | — | 2022-JUL-26 |
For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:
To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications
Unleash the Power of TAC's Virtual Assistance