Field Notice: FN - 72115 - Nexus Product Line: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing and Smart Call Home Functionality - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.5	02-Aug-22	Updated the Product Hierarchy - No Content Changes
1.4	08-Jul-22	Updated the Products Affected, Problem Symptom, and Workaround/Solution Sections
1.3	27-May-22	Updated the Defect Information and Background Sections
1.2	25-Feb-22	Updated the Products Affected, Problem Description, Background, Problem Symptom, and Workaround/Solution Sections
1.1	14-Jan-22	Updated the Products Affected and Workaround/Solution Sections
1.0	15-Oct-21	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	NX-OS System Software	7	7.3(0)D1(1), 7.3(0)DX(1), 7.3(0)DY(1), 7.3(0)N1(1), 7.3(1)D1(1), 7.3(1)DY(1), 7.3(1)N1(1), 7.3(10)N1(1), 7.3(2)D1(1), 7.3(2)D1(2), 7.3(2)D1(3), 7.3(2)D1(3a), 7.3(2)N1(1), 7.3(3)D1(1), 7.3(3)N1(1), 7.3(4)D1(1), 7.3(4)N1(1), 7.3(5)D1(1), 7.3(5)N1(1), 7.3(6)D1(1), 7.3(6)N1(1), 7.3(7)D1(1), 7.3(7)N1(1), 7.3(7)N1(1a), 7.3(7)N1(1b), 7.3(8)N1(1), 7.3(9)N1(1)
NON-IOS	NX-OS System Software	8	8.0(1), 8.1(1), 8.1(1a), 8.1(1b), 8.1(2), 8.1(2a), 8.2(1), 8.2(2), 8.2(3), 8.2(4), 8.2(5), 8.2(6), 8.3(1), 8.3(2), 8.4(1), 8.4(1a), 8.4(2), 8.4(2a), 8.4(2b), 8.4(3), 8.4(4), 8.5(1)
NON-IOS	NX-OS System Software	9	9.2(1), 9.2(2), 9.2(3), 9.2(4), 9.3(1), 9.3(2), 9.3(3), 9.3(4), 9.3(5), 9.3(6), 9.3(7), 9.3(7a)	This does not include MDS 9.X Releases
NON-IOS	NX-OS System Software	10	10.1(1)
NON-IOS	Data Center Network Manager	11	11.0(1), 11.1(1), 11.2(1), 11.3(1), 11.4(1), 11.5(1), 11.5(2), 11.5(3)
NON-IOS	APIC Software	3.0	3.0(1k), 3.0(2h), 3.0(2k), 3.0(2n)
NON-IOS	APIC Software	3.1	3.1(1i), 3.1(2m), 3.1(2o), 3.1(2p), 3.1(2q), 3.1(2s), 3.1(2t), 3.1(2u), 3.1(2v)
NON-IOS	APIC Software	3.2	3.2(1l), 3.2(1m), 3.2(2l), 3.2(2o), 3.2(3i), 3.2(3n), 3.2(3o), 3.2(3r), 3.2(3s), 3.2(4d), 3.2(4e), 3.2(5d), 3.2(5e), 3.2(5f), 3.2(6i), 3.2(7f), 3.2(7k), 3.2(9b), 3.2(9f), 3.2(9h), 3.2(10e), 3.2(10f)
NON-IOS	APIC Software	4.0	4.0(1h), 4.0(2c), 4.0(3d)
NON-IOS	APIC Software	4.1	4.1(1i), 4.1(1j), 4.1(1k), 4.1(1l), 4.1(2g), 4.1(2m), 4.1(2o), 4.1(2s), 4.1(2u), 4.1(2w), 4.1(2x)
NON-IOS	APIC Software	4.2	4.2(1i), 4.2(1j), 4.2(1l), 4.2(2e), 4.2(2f), 4.2(2g), 4.2(3j), 4.2(3l), 4.2(3n), 4.2(3q), 4.2(4i), 4.2(4k), 4.2(4o), 4.2(4p), 4.2(5k), 4.2(5l), 4.2(5n), 4.2(6d), 4.2(6h), 4.2(6l), 4.2(6o), 4.2(7f), 4.2(7l), 4.2(7q), 4.2(7r), 4.2(7s)
NON-IOS	APIC Software	5.0	5.0(1k), 5.0(1l), 5.0(2e), 5.0(2h)
NON-IOS	APIC Software	5.1	5.1(1h), 5.1(2e), 5.1(3e), 5.1(4c), 5.1.80.EFT
NON-IOS	APIC Software	5.2	5.2(1g), 5.2(2e), 5.2(2f), 5.2(2g), 5.2(2h), 5.2(3e), 5.2(3f), 5.2(3g), 5.2(4d), 5.2(4e)

Defect Information

Defect ID	Headline
CSCvx28432	QuoVadis root CA decommission on all MDS platforms
CSCvx00444	QuoVadis root CA decommission on Nexus3k/Nexus9K
CSCvx00450	QuoVadis root CA decommission on Nexus5K/Nexus6K/Nexus7K
CSCvx29318	QuoVadis Cert Impact for Smart Licensing on DCNM
CSCwa97230	Smart license registration fails - F3058 Fail to send out Call Home HTTP message

Problem Description

For affected versions of the Nexus and MDS switches, some Secure Sockets Layer (SSL) certificates issued from the QuoVadis root certificate authority (CA) trust chain before March 31, 2021 cannot be renewed from this CA. Once those certificates expire on devices or are removed from the Cisco cloud servers, functions such as Smart Licensing and Smart Call Home will fail to establish secure connections to Cisco and might not operate properly.

Background

The QuoVadis Root CA 2 Public Key Infrastructure (PKI) used by Nexus and MDS switches software to issue SSL certificates is subject to an industry-wide issue that affects revocation abilities. Due to this issue, no new QuoVadis Root CA 2 certificates will be issued or renewed by Cisco after March 31, 2021. This affects certificate renewals on devices, Cisco cloud servers, and third-party services.

Certificates issued before the QuoVadis Root CA 2 was decommissioned will continue to be valid. However, the certificates will not renew when they expire on either the device or the Cisco cloud server. This will cause functions such as Smart Licensing and Smart Call Home to fail to establish secure connections to Cisco cloud servers.

This table shows a summary of the QuoVadis Root CA 2 certificate expiration dates for affected Cisco services.

Cisco Cloud Server	QuoVadis Certificate Expiration Date	Affected Services
tools.cisco.com	February 5, 2022	Smart Licensing Smart Call Home
smartreceiver.cisco.com	January 26, 2023	Smart Licensing

Problem Symptom

Expiration of the QuoVadis Root CA 2 certificates affects these services with the associated symptoms.

Affected Services	Symptoms for Affected Services
Smart Licensing	Failure to connect to the server (Details are provided in this section)
Smart Call Home	Failure to connect to the server and the Call-Home HTTP request fails

 

For Nexus and MDS switches, affected devices will be unable to connect to the Smart Licensing and Smart Call Home services hosted by Cisco. Smart licenses might fail entitlement and reflect an Out of Compliance status.

The features that use Smart Licensing will continue to function for one year after the last successful secure connection. Some Smart Licensing symptoms are:

• The device might indicate a failure to communicate with the Smart Licensing server within 30 days from the last successful connection.
• The device will show the "Authorization Expired" state if there is no communication with the Smart Licensing server within 90 days.
• The device will show the "Unregistered" state if there is no communication with the Smart Licensing server after one year and the licensed features usage become suspended.

For Nexus and MDS switches, enter the show license status command in order to view the licensing status. Affected platforms will show "Initial Registration: FAILED " in the output.

switch# show license status
Smart Licensing is ENABLED

Registration:
  Status: REGISTERING - REGISTRATION IN PROGRESS
  Export-Controlled Functionality: Not Allowed
  Initial Registration: FAILED on Aug 27 09:09:25 2021 UTC
  Failure reason: Fail to send out Call Home HTTP message
  Next Registration Attempt: Aug 27 09:27:06 2021 UTC

License Authorization:
  Status: No Licenses in Use

Smart License Conversion:
  Automatic Conversion Enabled: False
  Status: Successful on Jan  1 00:00:00 1970 UTC

Old certificates are shown in this example:

switch# show crypto ca trustpool
Trustpool download status :
==================================================
CA certificate
Serial Number        :01
Subject              :Cisco Licensing Root CA
Issued By            :Cisco Licensing Root CA
Validity Start       :May 30 19:48:47 2013 GMT
Validity End         :May 30 19:48:47 2038 GMT
==================================================
CA certificate
Serial Number        :01A65AF15EE994EBE1
Subject              :Cisco Basic Assurance Root CA 2099
Issued By            :Cisco Basic Assurance Root CA 2099
Validity Start       :May 26 19:19:29 2017 GMT
Validity End         :May 26 19:19:29 2099 GMT
==================================================
CA certificate
Serial Number        :03
Subject              :Cisco ECC Root CA
Issued By            :Cisco ECC Root CA
Validity Start       :Apr  4 08:15:44 2013 GMT
Validity End         :Sep  7 16:24:07 2099 GMT
==================================================
CA certificate
Serial Number        :5FF87B282B54DC8D42A315B568C9ADFF
Subject              :Cisco Root CA 2048
Issued By            :Cisco Root CA 2048
Validity Start       :May 14 20:17:12 2004 GMT
Validity End         :May 14 20:25:42 2029 GMT
==================================================
CA certificate
Serial Number        :019A335878CE16C1C1
Subject              :Cisco Root CA 2099
Issued By            :Cisco Root CA 2099
Validity Start       :Aug  9 20:58:28 2016 GMT
Validity End         :Aug  9 20:58:28 2099 GMT
==================================================
CA certificate
Serial Number        :2ED20E7347D333834B4FDD0DD7B6967E
Subject              :Cisco Root CA M1
Issued By            :Cisco Root CA M1
Validity Start       :Nov 18 21:50:24 2008 GMT
Validity End         :Nov 18 21:59:46 2033 GMT
==================================================
CA certificate
Serial Number        :01
Subject              :Cisco Root CA M2
Issued By            :Cisco Root CA M2
Validity Start       :Nov 12 13:00:18 2012 GMT
Validity End         :Nov 12 13:00:18 2037 GMT
==================================================
CA certificate
Serial Number        :01
Subject              :Cisco RXC-R2
Issued By            :Cisco RXC-R2
Validity Start       :Jul  9 21:46:56 2014 GMT
Validity End         :Jul  9 21:46:56 2034 GMT
==================================================
CA certificate
Serial Number        :066C9FCF99BF8C0A39E2F0788A43E696365BCA
Subject              :Amazon Root CA 1
Issued By            :Amazon Root CA 1
Validity Start       :May 26 00:00:00 2015 GMT
Validity End         :Jan 17 00:00:00 2038 GMT
==================================================
CA certificate
Serial Number        :066C9FD29635869F0A0FE58678F85B26BB8A37
Subject              :Amazon Root CA 2
Issued By            :Amazon Root CA 2
Validity Start       :May 26 00:00:00 2015 GMT
Validity End         :May 26 00:00:00 2040 GMT
==================================================
CA certificate
Serial Number        :066C9FD5749736663F3B0B9AD9E89E7603F24A
Subject              :Amazon Root CA 3
Issued By            :Amazon Root CA 3
Validity Start       :May 26 00:00:00 2015 GMT
Validity End         :May 26 00:00:00 2040 GMT
==================================================
CA certificate
Serial Number        :066C9FD7C1BB104C2943E5717B7B2CC81AC10E
Subject              :Amazon Root CA 4
Issued By            :Amazon Root CA 4
Validity Start       :May 26 00:00:00 2015 GMT
Validity End         :May 26 00:00:00 2040 GMT
==================================================
CA certificate
Serial Number        :083BE056904246B1A1756AC95991C74A
Subject              :DigiCert Global Root CA
Issued By            :DigiCert Global Root CA
Validity Start       :Nov 10 00:00:00 2006 GMT
Validity End         :Nov 10 00:00:00 2031 GMT
==================================================

Old certificates are not removed.

This is an example of a new certificate:

CA certificate
Serial Number        :0A0142800000014523C844B500000002
Subject              :IdenTrust Commercial Root CA 1
Issued By            :IdenTrust Commercial Root CA 1
Validity Start       :Jan 16 18:12:23 2014 GMT
Validity End         :Jan 16 18:12:23 2034 GMT
==================================================

This is an example of a typical old certificate:

CA certificate
Serial Number        :0509
Subject              :QuoVadis Root CA 2
Issued By            :QuoVadis Root CA 2
Validity Start       :Nov 24 18:27:00 2006 GMT
Validity End         :Nov 24 18:23:33 2031 GMT

Note: Cisco provides a 60-day grace period before affected Smart Licenses are placed in an Authorization Expired status that would impact feature functionality. Smart license registration for new products might be affected and requires the workaround/solution as shown in the next section.

Note: Offline licensing, such as Permanent License Reservation (PLR) and Specific License Reservation (SLR), is not affected by the certificate change on the Smart Licensing server.

For additional information, refer to the Cisco Smart Licensing Guide and the Nexus and MDS guide for your specific version of Nexus software.

For ACI APICs, these logs can be used to identify exposure:

The Cisco APIC is not able to register to the Smart Account in the CSSM and the F3058 "Fail to send out Call Home HTTP" message is generated.

The Call Home logs in the /var/log/dme/log/ch_dbg.log file show this failure reason:

*Wed Feb 16 10:15:20.083 UTC: CH-TRANS-ERROR: ch_pf_curl_send_msg[539], failed to perform, err code 60, err string "Peer certificate cannot be authenticated with given CA certificates"

Workaround/Solution

Cisco has migrated from the QuoVadis Root CA 2 to the IdenTrust Commercial Root CA 1 for SSL certificates. Cisco recommends these two options to add the new IdenTrust Commercial Root CA 1 certificate to the Nexus and MDS switches.

• Manual Certificate Update
• Software Upgrade

The updated IdenTrust Root CA 1 is shown here and complies with sha1WithRSAEncryption signature algorithm requirements.

-----BEGIN CERTIFICATE-----
MIIFYDCCA0igAwIBAgIQCgFCgAAAAUUjyES1AAAAAjANBgkqhkiG9w0BAQsFADBK
MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVu
VHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTQwMTE2MTgxMjIzWhcNMzQw
MTE2MTgxMjIzWjBKMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScw
JQYDVQQDEx5JZGVuVHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwggIiMA0GCSqG
SIb3DQEBAQUAA4ICDwAwggIKAoICAQCnUBneP5k91DNG8W9RYYKyqU+PZ4ldhNlT
3Qwo2dfw/66VQ3KZ+bVdfIrBQuExUHTRgQ18zZshq0PirK1ehm7zCYofWjK9ouuU
+ehcCuz/mNKvcbO0U59Oh++SvL3sTzIwiEsXXlfEU8L2ApeN2WIrvyQfYo3fw7gp
S0l4PJNgiCL8mdo2yMKi1CxUAGc1bnO/AljwpN3lsKImesrgNqUZFvX9t++uP0D1
bVoE/c40yiTcdCMbXTMTEl3EASX2MN0CXZ/g1Ue9tOsbobtJSdifWwLziuQkkORi
T0/Br4sOdBeo0XKIanoBScy0RnnGF7HamB4HWfp1IYVl3ZBWzvurpWCdxJ35UrCL
vYf5jysjCiN2O/cz4ckA82n5S6LgTrx+kzmEB/dEcH7+B1rlsazRGMzyNeVJSQjK
Vsk9+w8YfYs7wRPCTY/JTw436R+hDmrfYi7LNQZReSzIJTj0+kuniVyc0uMNOYZK
dHzVWYfCP04MXFL0PfdSgvHqo6z9STQaKPNBiDoT7uje/5kdX7rL6B7yuVBgwDHT
c+XvvqDtMwt0viAgxGds8AgDelWAf0ZOlqf0Hj7h9tgJ4TNkK2PXMl6f+cB7D3hv
l7yTmvmcEpB4eoCHFddydJxVdHixuuFucAS6T6C6aMN7/zHwcz09lCqxC0EOoP5N
iGVreTO01wIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB
/zAdBgNVHQ4EFgQU7UQZwNPwBovupHu+QucmVMiONnYwDQYJKoZIhvcNAQELBQAD
ggIBAA2ukDL2pkt8RHYZYR4nKM1eVO8lvOMIkPkp165oCOGUAFjvLi5+U1KMtlwH
6oi6mYtQlNeCgN9hCQCTrQ0U5s7B8jeUeLBfnLOic7iPBZM4zY0+sLj7wM+x8uwt
LRvM7Kqas6pgghstO8OEPVeKlh6cdbjTMM1gCIOQ045U8U1mwF10A0Cj7oV+wh93
nAbowacYXVKV7cndJZ5t+qntozo00Fl72u1Q8zW/7esUTTHHYPTa8Yec4kjixsU3
+wYQ+nVZZjFHKdp2mhzpgq7vmrlR94gjmmmVYjzlVYA211QC//G5Xc7UI2/YRYRK
W2XviQzdFKcgyxilJbQN+QHwotL0AMh0jqEqSI5l2xPE4iUXfeu+h1sXIFRRk0pT
AwvsXcoz7WL9RccvW9xYoIA55vrX/hMUpu09lEpCdNTDd1lzzY9GvlU47/rokTLq
l1gEIt44w8y8bckzOmoKaT+gyOpyj4xjhiO9bTyWnpXgSUyqorkqG5w2gXjtw+hG
4iZZRHUe2XWJUc0QhJ1hYMtd+ZciTY6Y5uN/9lu7rs3KSoFrXgvzUeF0K+l+J6fZ
mUlO+KWA2yUPHGNiiskzZ2s8EIPGrd6ozRaOjfAHN3Gf8qv8QfXBi+wAN10J5U6A
7/qxXDgGpRtK4dw4LTzcqx+QGtVKnO7RcGzM7vRX+Bi6hG6H
-----END CERTIFICATE-----

 

Product	Defect	Fixed Release	Workaround
Nexus 3000/Nexus 9000 Series Switches	CSCvx00444	10.2(1)F / 10.1(2) / 9.3(8)	Shown in Example 1
Nexus 7000 Series Switches	CSCvx00450	8.4(5) / 8.2(7a) / 7.3(8)D1(1)	Shown in Example 1
Nexus 5000/Nexus 6000 Series Switches	CSCvx00450	Fixed in a release in CY22	Shown in Example 1
MDS 9000	CSCvx28432	8.4(2c)	Shown in Example 1
DCNM Controller	CSCvx29318	11.5.3 and later releases	Shown in Example 2
ACI	CSCwa97230	4.2(7t), 5.2(5c)	Shown in Example 3

 

Note: Older versions of Data Center Network Manager (DCNM) that have connectivity (either externally or to OnPrem) will attempt to update certificates automatically via Smart Agent. If for some reason the automated update process fails, the customer can use these examples to manually update the certificates.

For DCNM that is not connected to the Internet, use Release 11.5(3) or later or use Example 2 if not connected.

Note: Affected Nexus 3000/Nexus 9000 releases have overlap with MDS software. Release 9.2(1) for Nexus 3000/Nexus 9000 is affected, however 9.X releases for MDS are not affected.

 

Example 1. Manual Certificate Update

In order to resolve the issue without a software upgrade, enter these CLI commands to manually import the IdenTrust Commercial Root CA 1 into the Nexus and MDS switches trust store.

switch# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b

Or if it is already copied to bootflash, enter this command:

switch# crypto ca trustpool import url bootflash:ios_core.p7b

Note: Existing certificates issued from the HydrantID SSL ICA G3 do not need to be replaced. They are normal certificates issued from the current SSL certificate service and can be used until expiration.

 

Example 2. Software Upgrade for Non-Internet Connected Devices

# disable smart license in the DCNM UI

# on the CLI, as root, stop dcnm

appmgr stop dcnm  / service FMServer stop 

# go to the agent directory

cd /usr/local/cisco/dcm/fm/conf/agent

rm -f ios_core.p7b

yum install -y wget

wget http://www.cisco.com/security/pki/trs/ios_core.p7b

chown fmserver.fmserver ios_core.p7b

 

Example 3. ACI Certificate Update

Note: These commands are run on APIC.

If the APIC version is 5.2.4, complete these steps:

1. Enter this command to update the certificate:

   icurl -X POST -k -d   ' <licenseLicPolicy dn= "uni/fabric/licensepol" licenseAction= "import-private-certificate"  cert="A"/>'

   'http://localhost:7777/api/mo/uni/fabric/licensepol.xml'

   Put the updated IdenTrust Root CA 1, mentioned in the Workaround/Solution section of Field Notice 72115, instead of {{A}} of the cert attribute.

2. Create a new token from your Smart Account and copy the token.
3. In the Cisco APIC GUI, choose System > Smart Licensing. Click Actions in the upper-right corner, and then click Configure Network Settings.
4. Paste the token in "Product Instance ID Token".
5. Ensure the Smart License Mode is set to "Direct connect to CSSM", and click OK.

If the APIC version is 5.2.3 or earlier, complete these steps:

1. Copy the root CA from Field Notice 72115  The field notice also includes the root CA in text format in its Workaround/Solution section.
2. In the Cisco APIC GUI, choose System > Smart Licensing. Click the toolbox, and choose Import/Remove private Certificate.
3. Paste the root certificate in the box. Ensure the action is set to "import", and click Submit.

For More Information

Cisco has created a web page to provide customers and partners with additional information on this issue. Consult the QuoVadis Root CA 2 Decommission page for a full list of products affected, associated Field Notices, and frequently asked questions.

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.