Field Notice: FN63942 - Wireless Lightweight Access Points and WLAN Controllers Fail to Create CAPWAP Connections Due to Certificate Expiration - Software Upgrade Recommended

Due to the expiration of Manufacturer Installed Certificates (MICs) in Wireless LAN Controllers (WLCs) and/or Access Points (APs), connections of these types might fail to establish:  

• AP-to-WLC Control and Provisioning of Wireless Access Points (CAPWAP) connections
• WLC-to-WLC encrypted mobility connections (CAPWAP)
• WLC-to-Mobility Services Engine (MSE) or Connected Mobile Experience (CMX) Network Mobility Services Protocol (NMSP) connections

When an AP attempts to establish a new connection, the AP fails to join. When you configure mobility tunnels between controllers, they fail to establish a connection.

The likelihood that this issue will be encountered is 100% for wireless APs and WLCs that are more than 10 years old and were manufactured prior to 2017.

In order to help determine the potential future impact, a list of APs and WLCs and their release dates can be found in the Lightweight AP - Fail to create CAPWAP/LWAPP connection due to certificate expiration Cisco Community article.

Prior to July 18, 2005, Cisco APs were not manufactured with MICs.

All Cisco wireless products that were manufactured after July 18, 2005 have SHA-1 MICs that expire after 10 years.

Starting in 2017, Cisco manufactured all wireless devices with additional SHA-2 MICs that expire in 2037 or later.

Starting in 2019, all 9800 WLCs and 9100 APs were manufactured with SHA-2 ("SUDI99") certificates that expire in 2099.

Here are some examples of the types of error message you might see when you experience this issue.

The wireless APs fail to connect to the AireOS WLCs. At the time of the join failure, the AireOS WLC msglog might show messages similar to this example:

Jul 10 16:13:52.443 spam_lrad.c:6164 LWAPP-3-PAYLOAD_ERR: Join request does not contain valid certificate in certificate payload - AP 00:11:22:33:44:55

An example of a WLC's MIC expiration case is shown here, as seen on a Cisco IOS^® AP:

*Aug 1 05:16:27.127: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain 
 validation has failed.
The certificate (SN: 423F49A800000005FEAE) has expired. Validity period ended
 on 14:39:18 UTC Sep 3 2023 Peer certificate verification failed 001A
*Aug 1 05:16:27.127: %CAPWAP-3-ERRORLOG: Certificate verification failed!

AP-COS fails to join an AireOS WLC whose MIC is expired, as seen on AP-COS:

[*12/12/2035 12:34:50.0099] Cert Verification FAILED with error 10 (certificate has expired) at 0 depth...
[*12/12/2035 12:34:50.0099]
[*12/12/2035 12:34:50.0099] /C=US/ST=California/L=San Jose/O=Cisco Systems/CN=AIR-CT5508-K9-00211bfe58c0/emailAddress=support@cisco.com

Cisco IOS AP fails to join an AireOS WLC whose MIC is expired, as seen on a Cisco IOS AP:

*Aug 1 05:16:27.127: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed. The certificate (SN: 423F49A800000005FEAE) has expired. Validity period ended on 14:39:18 UTC Sep 3 2023 Peer certificate verification failed 001A
*Aug 1 05:16:27.127: %CAPWAP-3-ERRORLOG: Certificate verification failed!

Cisco IOS AP fails to join a C9800 WLC, as seen on C9800:

Syslog messages that would appear when a problematic AP joins 9800 WLCs:

.Jan  1 00:01:05.630: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed.  The certificate (SN: 0159BC17) has expired.    Validity period ended on 2027-02-23T17:02:59Z
.Jan  1 00:01:05.633: %DTLS_TRACE_MSG-3-WLC_DTLS_ERR: Chassis 1 R0/0: wncd: DTLS Error, session:x.x.x.x[5272] MAC: aaaa.bbbb.cccc.dddd Certificate validation failed
.Jan  1 00:01:05.633: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: sangudla-3802I Session-IP:x.x.x.x[5272] CAPWAP DTLS session closed for AP, cause: DTLS handshake error

C9800 fails to connect to an AireOS WLC, as seen on C9800:

.Jan  1 00:01:20.620: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed.  The certificate (SN: 019A376F) has expired.    Validity period ended on 2027-06-08T18:09:45Z

AireOS WLC fails to establish an NMSP connection to a CMX/MSE, as seen on the MSE:

6/10/20 14:21:48 ERROR[com.aes] [100] [ConnectHandler:handle-09] THROW javax.net.ssl.SSLHandshakeException: Handshake failed
...
Caused by: java.security.cert.CertificateExpiredException: Certificate expired at Thu Apr 23 22:03:27 PDT 2020 (compared to Wed Jun 10 14:21:48 PDT 2020) at com.cisco.ciscossl.provider.ciscojce.ssl.OpenSSLX509Certificate.checkValidity(OpenSSLX509Certificate.java:458)

Workaround for APs That Fail to Join the WLC Due to an Expired Certificate

Note: This workaround should only be used in order to allow the APs to join the WLC just long enough to upgrade the software and implement the solution provided in this section.

If the AP's and/or WLC's certificates have expired:

1. Disable Network Time Protocol (NTP).
2. Change the WLC clock time to a recent earlier time when the certificates were still valid. If you set the clock back too far, newer APs might not be able to join.

Note: When you temporarily disable NTP and change the WLC time settings, it can adversely affect other time-dependent WLC features such as MFP, Simple Network Management Protocol Version 3 (SNMPv3), and Location.

Solution for Expired WLC Certificates

Situation: The WLC does not run a fixed software version and some APs cannot join.

1. Upgrade to a fixed version of the software.
2. Enter the config ap cert-expiry-ignore {mic|ssc} enable command.
3. If any of the APs that cannot join have not downloaded the fixed software:
   1. Disable NTP.
   2. Set the clock back to a time before the WLC certificate expired (might keep newer APs from joining).
   3. Have all APs join the WLC, download new software, and rejoin.
   4. Set the clock to the correct time and re-enable NTP.

Situation: The WLC runs fixed software, but some APs cannot join.

1. Enter the config ap cert-expiry-ignore {mic|ssc} enable command.
2. If any of the APs that cannot join have not downloaded the fixed software:
   1. Disable NTP.
   2. Set the clock back to a time before the WLC certificate expired (might keep newer APs from joining).
   3. Have all APs join the WLC, download new software, and rejoin.
   4. Set the clock to the correct time and re-enable NTP.

How to Identify Which Software Versions Have the Fix

If you run AireOS software, the fixes required depend on the APs that you have in your network:

• Cisco IOS APs that were manufactured prior to August 2014 can be fixed via Cisco bug ID CSCuq19142 for AireOS Versions 7.0.252.0, 7.4.140.0, 8.0.120.0, 8.1.102.0, and later.
• AP-COS APs can be fixed via Cisco bug ID CSCvb93909 in AireOS 8.5 and later.
• Cisco IOS APs that were manufactured with SHA-2 certificates in August 2014 and later can be fixed via Cisco bug ID CSCvs22835 in Version 8.5.160.0 and later.

If you run Cisco IOS XE Software Release 16.10 or later, then no software upgrade is required. Complete the configuration steps in the next section in order to address issues with expired AP certificates and/or encrypted mobility tunnels that fail to form.

Solution for Expired AP Certificates and/or for Scenario of Encrypted Mobility Tunnels That Fail to Form

C9800 Command to Accept Expired Certificates

configure terminal
crypto pki certificate map map1 1
 issuer-name co cisco manufacturing ca
crypto pki certificate map map1 2
 issuer-name co act2 sudi ca

crypto pki trustpool policy
 match certificate map1 allow expired-certificate
    
exit

Create a Certificate Map and Add the Rules

configure terminal
crypto pki certificate map map1 1
issuer-name co Cisco Manufacturing CA

Use the Certificate Map Under the Trustpool Policy

configure terminal
crypto pki trustpool policy
match certificate map1 allow expired-certificate

Note: You can add more rules and filters under the same map. The rule mentioned in the previous configuration specifies that any certificate whose issuer-name contains Cisco Manufacturing CA (case insensitive) is selected under this map.

Situation: NMSP does not come up on MSE.

On MSE, enter the config unauthenticated-nmsp command.

See these documents for recommended versions for your platforms:

• Recommended AireOS Wireless LAN Controller Releases
• Recommended Cisco IOS XE Releases for Catalyst 9800 Wireless LAN Controllers
• Guidelines for Cisco Wireless Software Release Product Bulletin

For APs that use a MIC, the approximate expiration date can be derived from the device serial number (SN).

A Python script that automates this process (works with MICs, SSCs, and LSCs) is available from the Access Point Certificate check tool - apCertCheck Cisco Community article.

MIC

You can use the SN to determine the approximate date that the MIC will expire.

The AP MIC will expire, on average, ten years past the date of manufacture. Note that some APs might have earlier or more recently created MICs under some conditions. For example, consider a scenario where the AP motherboard was manufactured and stored, but not assembled until some time later, or when the AP was subject to a Return Material Authorization (RMA) and a refurbishing process.

In order to determine when the AP was manufactured, enter this command on the WLC to find the AP SN:

(Cisco Controller) >show ap inventory all
Inventory for lap1130-sw3-9
NAME: "Cisco AP" , DESCR: "Cisco Wireless Access Point"
PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE
NAME: "Dot11Radio0" , DESCR: "802.11G Radio"
PID: UNKNOWN, VID: , SN: GAM112706LC?
NAME: "Dot11Radio1" , DESCR: "802.11A Radio"
PID: UNKNOWN, VID: , SN: ALP112706LC

The AP chassis SN is in the first section of the output. For example: PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE.

Note: Refer to the Derive Manufactured Date from the Product SN section of this Field Notice for additional information.

Alternatively, the exact date that the MIC expires can be found with the output of the show crypto pki certificates (Cisco IOS) command or show crypto (AP-COS) command from the AP CLI. Look for the Certificate entry, and ignore the CA Certificate entries. The end date that is associated with the Validity Date section is the expiration date for the MIC certificate:

AP_CLI#show crypto pki certificates
CA Certificate
Status: Available...
...
Certificate
Status: Available
Certificate Serial Number: 728AF4350000001E4C89
Certificate Usage: General Purpose
Issuer:
cn=Cisco Manufacturing CA
o=Cisco Systems
Subject:
Name: C1130-001c58b5b3a4
ea=support@cisco.com
cn=C1130-001c58b5b3a4
o=Cisco Systems
l=San Jose
st=California
c=US
CRL Distribution Points:
http://www.cisco.com/security/crl/cmca.crl
Validity Date:
start date: 04:22:10 UTC Jul 11 2007
end date: 04:32:10 UTC Jul 11 2017
Associated Trustpoints: Cisco_IOS_MIC_cert

AireOS WLC

If you run AireOS Version 8.0 or later, then in order to determine when the WLC certificate expires, enter this command and look for the Cisco SHA1 device cert entry:

WLC_CLI: show certificate all
Certificate Name: Cisco SHA1 device cert

Subject Name :
C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT2504-K9-d0c282d65a20,
 MAILTO=support@cisco.com
Issuer Name :
O=Cisco Systems, CN=Cisco Manufacturing CA
Serial Number :
454384735992863371807890
Validity :
Start : 2011 Jul 26th, 20:17:17 GMT
End : 2021 Jul 26th, 20:27:17 GMT
Signature Algorithm :
rsa-pkcs1-sha1
Hash key :
SHA1 Fingerprint : 98:89:eb:12:2a:98:bc:fe:ad:5b:8f:23:63:0f:47:d1:36:ce:f5:be
MD5 Fingerprint : ba:f3:98:9a:cd:f8:01:08:84:b8:66:3c:6a:6c:d3:05

Note: This command is not available in AireOS releases earlier than Version 8.0. There is no similarly straightforward command that can be used in order to derive this date in earlier AireOS releases. As an alternate method, use the WLC SNs in order to determine the earliest possible MIC expiration date.

Derive Manufactured Date from the Product SN

The SN format is LLLYYWWSSSS. The YY is the year of manufacture, and the WW is the week of manufacture. The date code can be found in the four middle digits of the SN.

Note: Since the SN format for the AIR-CT7510 is different, you cannot use it to derive the manufacturing date. Instead, for the AIR-CT7510, enter the show certificate all command as noted previously.

The Manufacturing Year Codes are:

01 = 1997
02 = 1998
03 = 1999
04 = 2000
05 = 2001
06 = 2002
07 = 2003
08 = 2004
09 = 2005
10 = 2006
11 = 2007
12 = 2008
13 = 2009
14 = 2010
15 = 2011
16 = 2012
17 = 2013
18 = 2014

The Manufacturing Week Codes are:

01-05 = January
06-09 = February
10-14 = March
15-18 = April
19-22 = May
23-27 = June
28-31 = July
32-35 = August
36-40 = September
41-44 = October
45-48 = November
49-52 = December

For example, the SN FCZ1128Q0PE has a Manufacturing Year Code of 11, which means that it was manufactured in the year 2007. The week code is 28, which means that it was manufactured in July of that year.