Field Notice: FN - 72265 - Expired PKI Certificate on vEdge, ISR, and ASR Routers Causes SD-WAN Umbrella DNS Connections to Fail - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	08-Nov-21	Initial Release
1.1	22-Nov-21	Updated the Workaround/Solution Section

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	IOSXE	17	17.3.1, 17.3.1a, 17.3.1w, 17.3.1x, 17.3.2, 17.3.2a, 17.3.3, 17.3.3a, 17.3.4, 17.3.4a, 17.4.1, 17.4.1a, 17.4.1b, 17.4.2, 17.5.1, 17.5.1a, 17.6.1, 17.6.1a	Cisco SD-WAN "controller mode" only
NON-IOS	vEdge Software	20	20.3.1, 20.3.2, 20.3.3, 20.3.4, 20.4.1.1, 20.4.1.2, 20.4.2, 20.5.1, 20.6.1
NON-IOS	IOSXE	16	16.10.1, 16.10.1a, 16.10.1b, 16.10.1c, 16.10.1d, 16.10.1e, 16.10.1f, 16.10.1g, 16.10.1i, 16.10.1s, 16.10.2, 16.10.3, 16.11.1, 16.11.1a, 16.11.1b, 16.11.1c, 16.11.1s, 16.11.2, 16.12.1, 16.12.1a, 16.12.1c, 16.12.1s, 16.12.1t, 16.12.1w, 16.12.1x, 16.12.1y, 16.12.1z, 16.12.1z1, 16.12.1z2, 16.12.2, 16.12.2a, 16.12.2s, 16.12.2t, 16.12.3, 16.12.3a, 16.12.3s, 16.12.4, 16.12.4a, 16.12.5, 16.12.5a, 16.12.5b, 16.12.6	Cisco SD-WAN images only

Defect Information

Defect ID	Headline
CSCvz86967	DST Root CA X3 Expiration causing umbrella integration to fail
CSCvz86972	cEdge DST Root CA X3 Expiration causing umbrella integration to fail

Problem Description

The public key infrastructure (PKI) certificate used by Cisco SD-WAN routers to register with Cisco Umbrella Domain Name System (DNS) expired on 2021-09-30. Cisco SD-WAN routers with the expired PKI certificate fail to register with the Umbrella DNS service. The result of this failure is that all subsequent client DNS requests are dropped.

Background

This problem affects Cisco vEdge router products as well as Cisco IOS^® XE routers that run in SD-WAN Controller mode. Affected router series include Cisco Integrated Services Router (ISR), Cisco Aggregation Services Router (ASR), the Cisco CSR 1000v Cloud Services Router, the Cisco Integrated Services Virtual Router (ISRV), and Cisco Catalyst 8000v Edge Software. Affected products include a PKI certificate based on the DST Root CA X3 trust anchor. This certificate is used to establish a secure connection between a Cisco SD-WAN router and Cisco Umbrella DNS on routers that are configured to use Cisco Umbrella as their DNS service. The DST Root CA X3 certificate expired on 2021-09-30 and does not auto-renew. Once the certificate expires, all subsequent secure DNS registration requests between the SD-WAN router and Cisco Umbrella fail.

For additional background information, see DST Root CA X3 Expiration (September 2021).

Problem Symptom

Affected devices fail to establish secure connections with the Umbrella DNS service and DNS registration fails. Once registration fails, no DNS capability is available on Cisco IOS XE SD-WAN routers and all DNS requests from clients fail. On Cisco vEdge routers, DNS requests will be forwarded instead of redirected to the secure Umbrella DNS service. Without an available DNS service, client devices will experience a variety of network reachability failures such as web sites unavailable, cloud services unavailable, and so on.

Affected devices that are already in operation and part of an overlay will not immediately experience DNS related failures. The expired certificate is only used during device registration with the Cisco Umbrella DNS service, not for individual DNS requests. Device registration occurs when the Cisco Umbrella DNS service is initially configured or when the device is rebooted with an existing Cisco Umbrella DNS configuration present.

This problem only affects Cisco SD-WAN routers configured for Umbrella DNS. Cisco routers that run in Cisco IOS XE Autonomous mode are not affected. Cisco IOS XE devices that run in Autonomous mode use a different PKI certificate for Umbrella DNS. That certificate is not affected. Also, this problem does not affect devices configured for Cisco Umbrella Secure Internet Gateway (SIG) tunnel.

Workaround/Solution

There is no workaround. Affected devices must have the expired DST Root CA X3 certificate replaced with a new unexpired certificate rooted in ISRG Root X1. Customers who do not currently use Cisco Umbrella DNS, but expect to deploy it in the future, can replace the expired certificate by upgrading the SD-WAN router software to a version that contains the new certificate. The new certificate is installed automatically during the upgrade. Software releases that contain the new certificate are expected to become available in December 2021.

Customers with affected routers already configured for Cisco Umbrella DNS can replace the expired certificates by copying a new ISRG Root X1 rooted certificate to each affected router. Follow these instructions in order to replace the certificate.

Cisco IOS XE Routers That Run in Controller Mode

1. Download the new unexpired certificate from this web site and place it on a device that has access to the affected routers in the SD-WAN overlay.

   https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem

2. Enter the Linux scp command or similar mechanism in order to perform a secure file copy from the download device onto each affected router. For example:

   scp ./isrg-root-x1-cross-signed.pem admin@<EdgeIP>:bootflash:trustidrootx3_ca.ca

   Substitute <EdgeIP> with the IP address of the affected router.

3. Once the file copy completes, reload the router in order to complete the installation process.

Alternately, the new ISRG Root X1 rooted certificate can be downloaded to Cisco vMange and copied to each affected router. It is not possible to copy the new certificate directly into the router's bootflash with this method. Instead, the new certificate must be copied into a temporary directory first and then copied into the final bootflash location while logged into the router.

1. Log into vManage and access vshell.

   vManage# vshell
   vManage:~$ pwd
   /home/admin

2. Download the new unexpired certificate from the letsencrypt.org web site.

   wget https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem --no-check-certificate

3. Enter the Linux scp command in order to perform a secure file copy from vManage into a temporary location on each affected router. For example:

   scp -P 830 isrg-root-x1-cross-signed.pem admin@<EdgeIP>:/bootflash/sdwan/trustidrootx3_ca.ca

   Substitute <EdgeIP> with the IP address of the affected router.

4. Log into the affected router.

5. Enter the copy CLI command in order to copy the new certificate from the temporary location into bootflash.

   router# copy bootflash:/sdwan/trustidrootx3_ca.ca bootflash:
   Destination filename [trustidrootx3_ca.ca]?

6. Enter the delete CLI command in order to remove the certificate file from the temporary location.

   router# delete bootflash:/sdwan/trustidrootx3_ca.ca

7. Reload the router in order to complete the certificate installation process.

Cisco vEdge Routers

Cisco vEdge routers require authorized root access in order to replace the certificate. The access procedure requires Cisco Technical Assistance Center (TAC) intervention. Customers with affected Cisco vEdge routers should contact the Cisco TAC for assistance with the expired certificate.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.