THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Revision | Publish Date | Comments |
---|---|---|
1.0 |
25-Jul-22 |
Initial Release |
Affected OS Type | Affected Software Product | Affected Release | Affected Release Number | Comments |
---|---|---|---|---|
NON-IOS |
DNA Center Software |
2 |
2.1.2.7, 2.1.2.8, 2.2.1.0, 2.2.2.0, 2.2.2.3, 2.2.2.4, 2.2.2.5, 2.2.2.6, 2.2.2.7, 2.2.2.8, 2.2.3.0, 2.2.3.3 |
Note: versions prior to 2.1.2.7 are omitted because the problem described in this field notice is superseded by https://www.cisco.com/c/en/us/support/docs/field-notices/723/fn72359.html in those earlier versions. |
Defect ID | Headline |
---|---|
CSCwa78814 | Cisco AI Cloud agent client certificate renewal |
Affected versions of Cisco Digital Network Architecture (DNA) Center include a PKI (Public Key Infrastructure) certificate for Cisco AI Network Analytics that does not automatically renew and cannot be manually renewed by the user. Once this certificate expires, Cisco DNA Center will fail to establish a secure connection with the Cisco AI Network Analytics cloud. As a result, all Cisco DNA Center AI Analytics and AI Endpoint Analytics features will no longer function. This problem is not applicable to Airgap versions of Cisco DNA Center.
In Cisco DNA Center, the AI Network Analytics agent communicates with the Cisco AI Network Analytics cloud via Transport Layer Security (TLS) sessions authenticated by PKI X.509 certificates. The agent client certificate is stored on Cisco DNA Center and has a three year expiration. Affected versions of Cisco DNA Center do not have the capability to renew the agent client certificate without manual intervention from authorized Cisco personnel. Once the certificate expires, the agent can no longer communicate with the cloud and all AI Network Analytics features cease to function.
Note: Cisco AI Network Analytics agent client certificates signed after June 2021 have a one year expiration.
The features shown here are impacted by this problem:
There are two ways you can encounter an expired agent client certificate:
Affected systems with an unexpired Cisco AI Analytics agent client certificate will not exhibit any failure symptoms. There is no proactive notification on the Cisco DNA Center user interface to indicate an agent client certificate that is about to expire. Follow the instructions shown here in order to determine the agent client certificate expiration date on your system.
How to Find the Cisco AI Network Analytics Agent Client Certificate Expiration Date
The Cisco AI Analytics agent’s client certificate is stored along with the AI Network Analytics configuration. Use the Cisco DNA Center CLI commands shown here in order to display the expiration date:
export MAGLEV_TOKEN=$(cat ~/.maglevconf | grep token | awk -F '= ' '{print $2}')
curl -s -X GET -H "X-Auth-Token:$MAGLEV_TOKEN" -k http://kairos-agent.ai-network-analytics.svc.cluster.local:8089/api/v1/config | jq -r '."client-cert"' | openssl x509 -text | grep "Not After"
The example shown here shows the result returned. The certificate expiration date is highlighted for clarity.
$ curl -s -X GET -H "X-Auth-Token:$MAGLEV_TOKEN" -k http://kairos-agent.ai-network-analytics.svc.cluster.local:8089/api/v1/config | jq -r '."client-cert"' | openssl x509 -text | grep "Not After"
Not After : Feb 26 13:03:00 2023 GMT
Symptoms of an Expired Agent Client Certificate
Systems with an expired Cisco AI Network Analytics agent client certificate produce symptoms in common with other certificate and connectivity problems. Like with all of those problem types, the secure connection from Cisco DNA Center to the Cisco AI Analytics cloud will fail. The connection failure can be observed from the Cisco DNA Center user interface and in logs. If any of the symptoms shown here occur on your system, use the certificate expiration date instructions in the previous section in order to determine if the failure is caused by an expired agent client certificate.
Example: Heatmap
Choose Cisco DNA Center UI > Assurance > Trends and Insights > Network Heatmap
.
Oops! There is an error fetching data.
The agent logs can be collected through these steps.
Run this command on the Cisco DNA Center Appliance CLI.
$ magctl service logs -a ai-network-analytics kairos-agent
Open the Cisco DNA Center Root Cause Analysis (RCA) bundle. The logs will show errors similar to this example:
{
"component": "apiproxy",
...
"level": "warning",
"method": "POST",
"msg": "api proxy: request failed",
"object": "ProxyContext",
"package": "proxy",
"resBody": "<html>\r\n<head><title>400 The SSL certificate error</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>The SSL certificate error</center>\r\n</body>\r\n</html>\r\n",
"status": 400,
...
}
There are two solution options. The choice depends on whether the agent client certificate has expired.
Solution for Systems with an Unexpired Certificate
A software fix is available for Cisco DNA Center. The fix implements an automatic X.509 certificate renewal process for the agent client certificate in order to prevent expiration. Cisco recommends customers upgrade their affected Cisco DNA Center appliances to one of these versions with the fix:
Note: The software fix will prevent the expiration of a Cisco AI Network Analytics agent client certificate, but it cannot renew a certificate that has already expired. If your agent client certificate has already expired, you must choose the solution for an expired certificate as shown in the next section.
Solution for Systems with an Expired Certificate
If the agent client certificate has expired on your system, the certificate must be renewed in order to restore Cisco AI Network Analytics features. The certificate can only be renewed via a manual process performed on your system and must be performed by authorized Cisco personnel. Contact the Cisco Technical Assistance Center (TAC) for assistance with the certificate renewal.
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.
Unleash the Power of TAC's Virtual Assistance