THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Revision | Publish Date | Comments |
---|---|---|
1.1 |
21-Mar-22 |
Updated the Workaround/Solution Section and Added the Additional Information Section |
1.0 |
24-Feb-22 |
Initial Release |
Affected OS Type | Affected Software Product | Affected Release | Affected Release Number | Comments |
---|---|---|---|---|
NON-IOS |
IOS XR Software |
7 |
7.0.0, 7.0.1, 7.0.2, 7.1.1, 7.1.2, 7.1.3, 7.2.0, 7.2.1, 7.2.2, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.4.1, 7.5.1 |
|
NON-IOS |
IOS XR Software |
6 |
6.0.0, 6.0.1, 6.0.2, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.2.1, 6.2.2, 6.2.3, 6.3.1, 6.3.2, 6.3.3, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.5.1, 6.5.2, 6.5.3, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.7.1, 6.7.2, 6.7.3, 6.7.4, 6.8.1 |
Defect ID | Headline |
---|---|
CSCvx00476 | QuoVadis root CA decommission on all |
For affected versions of the Cisco IOS® XR products, some Secure Socket Layer (SSL) certificates issued from the QuoVadis root certificate authority (CA) trust chain before March 31, 2021 cannot be renewed from this CA. Once those certificates expire on devices or are removed from the Cisco cloud servers, functions such as Smart Licensing and Smart Call Home will fail to establish secure connections to Cisco and might not operate properly.
The QuoVadis Root CA 2 Public Key Infrastructure (PKI) used by Cisco IOS XR products to issue SSL certificates is subject to an industry-wide issue that affects revocation abilities. Due to this issue, no new QuoVadis Root CA 2 certificates will be issued or renewed by Cisco after March 31, 2021. This affects certificate renewals on devices, Cisco cloud servers, and third-party services.
Certificates issued before the QuoVadis Root CA 2 was decommissioned will continue to be valid. However, the certificates will not renew when they expire on either the device or the Cisco cloud server. This will cause functions such as Smart Licensing and Smart Call Home to fail to establish secure connections to Cisco cloud servers.
This table shows a summary of the QuoVadis Root CA 2 certificate expiration dates for affected Cisco services.
Cisco Cloud Server | QuoVadis Certificate Expiration Date | Affected Services |
---|---|---|
tools.cisco.com | February 5, 2022 |
|
smartreceiver.cisco.com | January 26, 2023 |
|
Expiration of the QuoVadis Root CA 2 certificates affects these services with the associated symptoms.
Affected Services | Symptoms for Affected Services |
---|---|
Smart Licensing | Failure to connect to the server (Details are provided in this section) |
Smart Call Home | Failure to connect to the server and the Call-Home HTTP request fails |
For Cisco IOS XR software, affected devices will be unable to connect to the Smart Licensing and Smart Call Home services hosted by Cisco. Smart licenses might fail entitlement and reflect an Out of Compliance status.
The features that use Smart Licensing will continue to function for up to one year after the last successful secure connection.
Some Smart Licensing symptoms are:
Smart Licensing Probable Symptoms
If Smart License registration fails, these syslog messages will be seen:
RP/0/RP0/CPU0:Feb 22 10:00:07.962 UTC: smartlicserver[127]: %LICENSE-SMART_LIC-3-COMM_FAILED : Communications failure with the Cisco Smart Software Manager (CSSM) : Fail to send out Call Home HTTP message
RP/0/RP0/CPU0:Feb 22 10:00:07.962 UTC: smartlicserver[127]: %LICENSE-SMART_LIC-3-AGENT_REG_FAILED : Smart Agent for Licensing Registration with the Cisco Smart Software Manager (CSSM) failed: Fail to send out Call Home HTTP message
Smart Call Home Symptoms
This error log might be observed in the affected device:
RP/0/RP0/CPU0:Feb 22 09:57:36.246 UTC: http_client[255]: %SECURITY-XR_SSL-3-CERT_VERIFY_ERR_2_PARAM : SSL certificate verify error: Peer certificate verification failed - no trusted cert 'Crypto Engine' detected the 'warning' condition 'Invalid trustpoint or trustpoint not exist'
If the download fails, this message might be seen:
RP/0/RP0/CPU0:Feb 22 09:57:24.025 UTC: cepki[279]: %SECURITY-PKI-6-ERR_1_PARAM : Download failed, HTTP returned an error
Note: Offline licensing, such as Permanent License Reservation (PLR) and Specific License Reservation (SLR), is not affected by the certificate change on the Smart Licensing server.
For additional information, refer to the Cisco Smart Licensing Guide and the Cisco IOS XR Platform Guide for your specific version of Cisco IOS XR software.
This log shows the license authorization failure:
RP/0/RP0/CPU0:ios#show license status Wed Feb 23 06:08:07.562 UTC Smart Licensing is ENABLED Utility: Status: DISABLED Data Privacy: Sending Hostname: yes Callhome hostname privacy: DISABLED Smart Licensing hostname privacy: DISABLED Version privacy: DISABLED Transport: Type: Callhome Registration: Status: REGISTERED Smart Account: BU Production Test 1 Virtual Account: NCS550 Testing Export-Controlled Functionality: ALLOWED Initial Registration: SUCCEEDED on Feb 23 2022 05:56:41 UTC Last Renewal Attempt: None Next Renewal Attempt: Aug 22 2022 05:56:40 UTC Registration Expires: Feb 23 2023 05:51:36 UTC License Authorization: Status: OUT OF COMPLIANCE on Feb 23 2022 05:56:50 UTC Last Communication Attempt: FAILED on Feb 23 2022 06:08:07 UTC Failure reason: Fail to send out Call Home HTTP message Next Communication Attempt: Feb 23 2022 06:08:36 UTC Communication Deadline: May 24 2022 05:56:21 UTC Export Authorization Key: Features Authorized: <none> Miscellaneous: Custom Id: <empty> RP/0/RP0/CPU0:ios#
Cisco has migrated from the QuoVadis Root CA 2 to the IdenTrust Commercial Root CA 1 for SSL certificates. Cisco recommends a manual certificate update to add the new IdenTrust Commercial Root CA 1 certificate to the Cisco IOS XR products.
Manual Certificate Update
In order to check the built-in and downloaded certificates installed on your device, enter the show crypto ca trustpool
command.
Example:
RP/0/RP0/CPU0:ios#show crypto ca trustpool Wed Apr 1 10:25:52.995 UTC Trustpool: Built-In ================================================== CA certificate Serial Number : 5F:F8:7B:28:2B:54:DC:8D:42:A3:15:B5:68:C9:AD:FF Subject: CN=Cisco Root CA 2048,O=Cisco Systems Issued By : CN=Cisco Root CA 2048,O=Cisco Systems Validity Start : 20:17:12 UTC Fri May 14 2004 Validity End : 20:25:42 UTC Mon May 14 2029 SHA1 Fingerprint: DE990CED99E0431F60EDC3937E7CD5BF0ED9E5FA Trustpool: Built-In ================================================== CA certificate Serial Number : 2E:D2:0E:73:47:D3:33:83:4B:4F:DD:0D:D7:B6:96:7E Subject: CN=Cisco Root CA M1,O=Cisco Issued By : CN=Cisco Root CA M1,O=Cisco Validity Start : 21:50:24 UTC Tue Nov 18 2008 Validity End : 21:59:46 UTC Fri Nov 18 2033 SHA1 Fingerprint: 45AD6BB499011BB4E84E84316A81C27D89EE5CE7 Trustpool: Built-In ================================================== CA certificate Serial Number : 3C:91:31:CB:1F:F6:D0:1B:0E:9A:B8:D0:44:BF:12:BE Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Issued By : OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US Validity Start : 00:00:00 UTC Mon Jan 29 1996 Validity End : 23:59:59 UTC Wed Aug 02 2028 SHA1 Fingerprint: A1DB6393916F17E4185509400415C70240B0AE6B Trustpool: Built-In ================================================== CA certificate Serial Number : 05:09 Subject: CN=QuoVadis Root CA 2,O=QuoVadis Limited,C=BM Issued By : CN=QuoVadis Root CA 2,O=QuoVadis Limited,C=BM Validity Start : 18:27:00 UTC Fri Nov 24 2006 Validity End : 18:23:33 UTC Mon Nov 24 2031 SHA1 Fingerprint: CA3AFBCF1240364B44B216208880483919937CF7
Manual Certificate Update Options
The three options for a manual certificate update are described in this section.
Option 1. Download a Bundle of Certificates From the Default Location
The bundle of certificates can be downloaded from http://www.cisco.com/security/pki/trs/ios.p7b by default. If a device is not able to resolve the domain name, it will not download the pool of certificates. This bundle of certificates has the Smart Licensing root certificates.
Initiate the download with this command:
crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios.p7b
Option 2. Download the Bundle of Certificates by Hosting it Locally
Cisco IOS XR devices support trustpool policy configuration which allows you to change the default CA Bundle URL. This configuration command can be used to change the CA bundle policy.
crypto ca trustpool policy cabundle url <local customer defined url>
Download the CA bundle file from the default location in order to host it locally in an HTTP server. This can be leveraged in case of a connectivity issue.
Enter this command in order to display the trustpool policy:
show crypto ca trustpool policy
Option 3. File-Based Download
Customers who are not able to authenticate the certificate with options 1 or 2 can use the file-based approach. Download the file from the default URL (http://www.cisco.com/security/pki/trs/ios.p7b) and copy it to the "tmp" directory. You can then enter this command to authenticate the CA certificates to the device.
crypto ca trustpool import url /tmp/pki_bundle_0.p7b
Note: Cisco IOS XR does not support customer-signed CA bundle of certificates. It must be signed by the Cisco M1 root certificate, and it is managed by the Cisco Information Security team.
Enter this command for authorization:
RP/0/RP0/CPU0:ios#license smart renew auth
Check the license status:
RP/0/RP0/CPU0:ios#show license status Wed Feb 23 06:02:34.825 UTC Smart Licensing is ENABLED Utility: Status: DISABLED Data Privacy: Sending Hostname: yes Callhome hostname privacy: DISABLED Smart Licensing hostname privacy: DISABLED Version privacy: DISABLED Transport: Type: Callhome Registration: Status: REGISTERED Smart Account: BU Production Test 1 Virtual Account: NCS550 Testing Export-Controlled Functionality: ALLOWED Initial Registration: SUCCEEDED on Feb 23 2022 05:56:41 UTC Last Renewal Attempt: None Next Renewal Attempt: Aug 22 2022 05:56:40 UTC Registration Expires: Feb 23 2023 05:51:36 UTC License Authorization: Status: OUT OF COMPLIANCE on Feb 23 2022 05:56:50 UTC Last Communication Attempt: SUCCEEDED on Feb 23 2022 06:01:22 UTC Next Communication Attempt: Feb 23 2022 18:01:21 UTC Communication Deadline: May 24 2022 05:56:20 UTC Export Authorization Key: Features Authorized: <none> Miscellaneous: Custom Id: <empty> RP/0/RP0/CPU0:ios#
Cisco has created a web page to provide customers and partners with additional information on this issue. Consult the QuoVadis Root CA 2 Decommission page for a full list of products affected, associated Field Notices, and frequently asked questions.
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.
Unleash the Power of TAC's Virtual Assistance