THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Affected Software Product | Affected Release | Affected Release Number | Comments |
---|---|---|---|
Cisco DNA Center Software | 2 |
2.3.2.0, 2.3.2.1, 2.3.2.1-AIRGAP, 2.3.2.3, 2.3.3.0, 2.3.3.0-AIRGAP, 2.3.3.1, 2.3.3.1-AIRGAP, 2.3.3.3, 2.3.3.3-AIRGAP, 2.3.3.4, 2.3.3.4-AIRGAP, 2.3.3.5, 2.3.3.5-AIRGAP, 2.3.3.6, 2.3.3.6-AIRGAP, 2.3.3.7, 2.3.3.7-AIRGAP, 2.3.4.0, 2.3.4.0-AIRGAP, 2.3.4.3, 2.3.4.3-AIRGAP, 2.3.5.0, 2.3.5.0-AIRGAP, 2.3.5.3, 2.3.5.3-AIRGAP, 2.3.5.4, 2.3.5.4-AIRGAP, 2.3.7.0, 2.3.7.0-AIRGAP |
Defect ID | Headline |
CSCwe15923 | Internally auto-generated Etcd certificate is not activated after renewal |
CSCwh81546 | Internally auto-generated Etcd certificate is not activated after upgrade |
On affected releases of Cisco DNA Center, the Public Key Infrastructure (PKI) certificates for the etcd container are not activated after they automatically renew. This condition leaves the etcd container pointing to stale certificates instead of the renewed certificates. Once the stale etcd certificates expire, the web UI becomes unavailable, and other features may stop functioning.
Cisco DNA Center introduced digital certificates for etcd in Release 2.3.2.0 to ensure secure data communication with Kubernetes, both within a node and between nodes in a cluster. These certificates are valid for one year and are automatically renewed before they expire. The renewed certificates are processed by a helper container and then made available to the etcd container. In affected Cisco DNA Center releases, the etcd container does not recognize and activate those renewed certificates dynamically and continues to point to the expired certificates until etcd is restarted.
Because Kubernetes function is impacted by this problem, there are multiple symptoms, many of which result in the Cisco DNA Center node appearing unresponsive. For example, the main web user interface may stop functioning or become inaccessible. Also, SSL: CERTIFICATE_VERIFY_FAILED errors may be seen on the CLI console, as shown in the following example:
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'),)': /v2/keys/maglev/config/node-x.x.x.x?sorted=true&recursive=true
If the etcd certificates on a Cisco DNA Center node have already expired and the system is already experiencing symptoms, contact the Cisco Technical Assistance Center (TAC) for help. The workaround procedure to restore function requires intervention from authorized Cisco personnel. Once affected nodes have been fully returned to service, install one of the fixes that is described below to prevent the problem from recurring.
Cisco has developed a software fix for this problem.
In the following table, the left column shows the affected Cisco DNA Center releases and the right column shows the recommended solutions.
Note: This table contains new affected releases and relevant solutions that have been added since the original publication of this Field Notice on September 29, 2023.
Select the recommended solution that matches your current affected release:
Cisco DNA Center Release | Recommended Solution |
---|---|
2.3.2.0 2.3.2.1 2.3.2.3 2.3.3.0 2.3.3.1 2.3.3.3 2.3.3.4 2.3.3.5 2.3.3.6 |
Upgrade to Release 2.3.3.7 or download and run the CSCwe15923 Fix script per the instructions that follow this table. Note: For those who choose to upgrade to Release 2.3.3.7, the upgrade process automatically installs the fix for this problem. There is no need to apply the Hotfix manually or run an additional script. |
2.3.2.1-AIRGAP 2.3.3.0-AIRGAP 2.3.3.1-AIRGAP 2.3.3.3-AIRGAP 2.3.3.4-AIRGAP 2.3.3.5-AIRGAP 2.3.3.6-AIRGAP |
Download and run the CSCwe15923 Fix script per the instructions that follow this table. |
2.3.3.7 |
For Release 2.3.3.7 deployments, the recommended solution depends upon when you upgraded to Release 2.3.3.7:
|
2.3.3.7-AIRGAP | Download and run the CSCwe15923 Fix script per the instructions that follow this table. |
2.3.4.0 and 2.3.4.0-AIRGAP 2.3.4.1 and 2.3.4.1-AIRGAP 2.3.5.0 and 2.3.5.0-AIRGAP |
Upgrade to Release 2.3.5.4 or 2.3.5.4-AIRGAP, as appropriate. |
2.3.5.3 |
Upgrade to Release 2.3.5.4 or apply the 2.3.5.3-HF5 Hotfix per the instructions that follow this table. |
2.3.5.3-AIRGAP | Upgrade to Release 2.3.5.4-AIRGAP. |
2.3.5.4 |
For Release 2.3.5.4 deployments, the recommended solution depends upon when you upgraded to Release 2.3.5.4:
|
2.3.5.4-AIRGAP |
For Release 2.3.5.4-AIRGAP deployments, the recommended solution depends upon how you upgraded to Release 2.3.5.4-AIRGAP:
|
2.3.7.0 and 2.3.7.0-AIRGAP | Upgrade to Release 2.3.7.3 or Release 2.3.7.3-AIRGAP, as appropriate, when available. |
CSCwe15923 Fix Script Procedure
Cisco recommends that customers who are running Cisco DNA Center releases 2.3.3.6 and earlier (except AIRGAP) upgrade to Release 2.3.3.7.
Customers who are unable to upgrade to Release 2.3.3.7 or who are running an AIRGAP release can instead download and run the CSCwe15923 Fix script for this problem. The script takes no more than three minutes to run, and the entire procedure should take no longer than 30 minutes for a single node. To download, install, and run the script, complete the following steps:
scp -P 2222 CSCwe15923-recovery-artifacts-2.3.3.x-RA.tar.gz maglev@<Node_IP_Address>:/
mv /home/maglev/.magshell/CSCwe15923-recovery-artifacts-2.3.3.x-RA.tar.gz /data/tmp/
tar zxvf CSCwe15923-recovery-artifacts-2.3.3.x-RA.tar.gz
sudo ./CSCwe15923.sh
Note: For a three-node cluster, repeat steps 4 - 11 for each additional node in the cluster.
Hotfix Installation Procedure
Before installing the Hotfix on your system, consult the following release notes:
To download and install the Hotfix, complete the following steps:
Note: For DR deployments only, pause DR first, then follow the steps on both the Main and Recovery sites. For instructions on how to pause DR, see the Pause Your Disaster Recovery System section of the Cisco DNA Center Administrator Guide.
Version | Description | Section | Date |
2.0 | Added a new bug and additional affected releases. Updated problem symptom description. Updated solution table to include additional instructions for updating Release 2.3.3.7. Also updated solution table to include solutions for newly added affected releases. | Products Affected, Defect Information, Problem Symptom, Workaround/Solution | 2023-NOV-01 |
1.0 | Initial Release | — | 2023-SEP-29 |
For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:
To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications
Unleash the Power of TAC's Virtual Assistance