Field Notice: FN - 72406 - Cisco DNA Center: Failure in PKI Certificate Refresh Process Causes System Function Failures - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	24-May-22	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	DNA Center Software	1	1.3, 1.3.0.7, 1.3.1.0, 1.3.1.2, 1.3.1.3, 1.3.1.4, 1.3.1.5, 1.3.1.6, 1.3.1.7, 1.3.3.0, 1.3.3.1, 1.3.3.3, 1.3.3.4, 1.3.3.5, 1.3.3.6, 1.3.3.7, 1.3.3.8
NON-IOS	DNA Center Software	2	2.1.2.0, 2.1.2.3, 2.1.2.4, 2.1.2.5, 2.1.2.6, 2.1.2.7
NON-IOS	DNA Center Software	2	2.2.2.0, 2.2.2.1, 2.2.2.3, 2.2.2.4, 2.2.2.5, 2.2.2.6, 2.2.2.7
NON-IOS	DNA Center Software	2	2.2.3.0, 2.2.3.3, 2.2.3.4

Defect Information

Defect ID	Headline
CSCwa89160	Provide a consolidated fix for multiple cert refresh related issues
CSCvw23564	Certificate refresh script fails after etcd certificate expires
CSCvv95329	[maglev] Root CA has expired and is not refreshed automatically
CSCvy55791	Upgrade failure due to an expired Docker CA certificate
CSCvy64706	Kubelet.conf client certificate fails to renew
CSCvx56103	Longevity: kubelet.conf certs expired and kubelet is down on the 3-node longevity cluster

Problem Description

Affected versions of Cisco Digital Network Architecture (DNA) Center can fail to renew one or more Public Key Infrastructure (PKI) X.509 certificates. After the PKI certificates expire, key system functions that rely on them fail and cause significant loss of function. The failures can include loss of access to the user interface.

Background

In affected versions of Cisco DNA Center, several software defects have been discovered that affect the ability of the software to renew PKI certificates properly before those certificates expire. More than one certificate is affected, and the failure characteristics and symptoms vary. For details on individual symptoms and failure modes, consult the defect description text on the Cisco Bug Search Tool for each software defect that is referenced in this field notice. Any one of the software defects, if encountered, can result in severe loss of function in Cisco DNA Center.

We have already provided individual software solutions for each defect that is referenced in this field notice. However, we have determined that additional software improvements are required in order to create a holistic solution and mitigate the possibility for future PKI certificate renewal failures.

Problem Symptom

There are multiple possible symptoms. This list provides a sample:

• You experience a complete loss of access to the Cisco DNA Center graphical user interface.
• The Cisco DNA Center user interface reports "invalid certificate" errors.
• An orange banner appears on the Cisco DNA Center user interface with the message, "Assurance services have been temporarily disrupted. The system is working to restore this functionality."

For more details on individual symptoms, consult the defect description text on the Cisco Bug Search Tool for each software defect that is referenced in this field notice.

Workaround/Solution

We have created a software fix that implements design improvements to the PKI certificate renewal infrastructure in Cisco DNA Center. This fix mitigates the possibility of future certificate renewal failures and improves upon the software point fixes that were provided previously. We strongly recommend that you upgrade to a software version that includes this new design improvement.

There are two solution options. Choose the option that matches your software version.

Option 1: Cisco DNA Center version 2.x

Upgrade to Cisco DNA Center software version 2.2.3.5. This software version includes the PKI certificate renewal improvements (defect CSCwa89160) as well as the fixes for the other software defects listed in this field notice.

Note: As of March 30, 2022, Cisco also installs the PKI certificate renewal enhancement fix (defect CSCwa89160) as part of the upgrade process for software version 2.1.2.8 (or later) and version 2.2.2.8 (or later). If you have 2.1.2.x version software or 2.2.2.x version software installed and do not want to migrate to version 2.2.3.5, you can obtain the fix through an upgrade to the latest patch release for your particular version.

Option 2: Cisco DNA Center version 1.3.3.9 and earlier

The PKI certificate renewal improvements fix is not available for Cisco DNA Center software versions 1.3.3.9 and earlier. However, a software fix is available in version 1.3.3.9 for the PKI certificate defect CSCvw23564 that is found in versions 1.3.3.8 and earlier.

If you cannot upgrade to version 1.3.3.9, a workaround script is available to help mitigate defect CSCvw23564. The script is available for download from Cisco.com.

Complete these steps to download, install, and run the script:

1. Download the script file CSCvw23564-Workaround.zip from Cisco.com. (URL: https://software.cisco.com/download/specialrelease/e14dc2f107b00f9be163e94365394a51)

    

2. Use a Secure Copy (SCP) or SFTP tool to copy the downloaded file CSCvw23564-Workaround.zip to the Cisco DNA Center server /data/tmp directory. For a three-node Cisco DNA Center cluster, copy the file to each node. A sample CLI command is shown here:

   scp -P 2222 CSCvw23564-Workaround.zip  maglev@<Node_IP_Address>:/data/tmp

   Note: Replace <Node_IP_Address> with the IP address of the Cisco DNA Center cluster node.

    

3. Log in to Cisco DNA Center as the maglev user.

    

4. Extract the contents of the CSCvw23564-Workaround.zip file with the unzip CLI command:

   unzip /data/tmp/CSCvw23564-Workaround.zip  -d /data/tmp/

    

5. Run the script files that were extracted in Step 4 with the sudo CLI command as shown in this example:

   sudo /data/tmp/CSCvw23564-Workaround.sh

    

6. For a three-node DNA Center cluster, repeat Step 3, Step 4, and Step 5 for each node in the cluster, one node at a time.

 

Note: If this message appears when you run the script, your Cisco DNA Center device is not susceptible to the defect. The script terminates and no further action is necessary.

Can't open /etc/maglev/.pki/etcd/server.crt for reading, No such file or directory
140249370886592:error:02001002:system library:fopen:No such file or directory:../crypto
/bio/bss_file.c:72:fopen('/etc/maglev/.pki/etcd/server.crt','r')
140249370886592:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
unable to load certificate
[INFO] | ETCD certs files are not present. Skipping hook run for CSCvw23564

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.