Cisco WLC or AP device Certificate Expired !!!

If you are using older Cisco WLAN Controllers (“WLC”) or access points in your network you might find your access points simply disappearing from your WLC one day and not being able to rejoin it. This could be due to a problem related to the digital certificates of the devices in your network.

As an administrator, you very rarely have to deal with the certificates used for authentication between the WLC and the AP because most of the time it just… works.

Some quick facts about device certificates in terms of Cisco WLCs and APs:

• During manufacture, a device certificate is installed in all WLCs and APs leaving the factory.
• This certificate is used to perform authentication between the WLC and an AP wanting to join the WLC.
• Without this mutual authentication, the WLC and AP won’t be able to establish a secure DTLS-tunnel between them for encrypting CAPWAP control traffic, which means your APs won’t be able to join the WLC.
• Device certificates for both WLCs and APs have a valid time of 10 years from the manufacturing date.
• Time is an important factor for the certification validity means that the time/date of your WLC and connecting APs is important.
• APs get their time from the WLC as soon as they try to connect. Even if the CAPWAP/DTLS-connection is not successfully established, the AP will still get the time from the WLC.

The Problem

I myself have run into two problems related to device certificates:

• Older access point does not want to join any WLC
• Newer access point does not want to join an older WLC

The first issue shows up when you have an old access point that has hit that 10-year mark and its device certificate has finally hit the expiration date. At the time of writing this article, these access points are usually of the models AP-1131 and AP-1142. Your AP will not be disconnected from the WLC immediately on the certificate’s expiration date but in case of a restart of either the AP or the WLC where a new CAPWAP-tunnel must to established between the two, the connection will not be completed.

Connecting to the console port of an AP is the easiest method to see this problem in action and it looks something like this. 10.10.66.250 is the IP address of the WLC.

*Sep 13 18:26:24.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.10.66.250 peer_port: 5246

*Sep 13 18:26:24.000: %CAPWAP-5-CHANGED: CAPWAP changed state to

*Sep 13 18:26:24.099: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed.  

The certificate (SN: 3C1E27950000000CAAAA) has expired.    Validity period ended on 19:56:24 UTC Sep 12 2019

*Sep 13 18:26:24.099: %LWAPP-3-CLIENTERRORLOG: Peer certificate verification failed

*Sep 13 18:26:24.099: %CAPWAP-3-ERRORLOG: Certificate verification failed!

The error message is pretty clear, the AP itself is alerting us that its certificate has expired.

The second issue presents itself when you have a newer access point that is trying to connect to an older WLC, whose own device certificate has reached 10 years of age and therefore expired. At the time of writing this article, the earliest manufactured models of the WLC 5508 controller have been starting to run into this issue as of May 2018.

Connecting to the console port of an AP also shows this problem, the error message could look a bit different depending on if you using an AP running the older IOS software (pre-1800/2800/3800 series APs) or newer AP-COS software (1800/2800/3800 and forward).

For old IOS-based access points the error could look like this:

[*09/09/2019 12:34:50.0099] Cert Verification FAILED with error 10 (certificate has expired) at 0 depth...

[*09/09/2019 12:34:50.0099] [*09/09/2019 12:34:50.0099] /C=US/ST=California/L=San Jose/O=Cisco Systems/CN=AIR-CT5508-K9-00211bfeAAAA/[email protected]

For new AP-COS-based access points could look like this:

[*09/09/2019 04:55:26.3299] CAPWAP State: DTLS Teardown

[*09/09/2019 04:55:30.9385] CAPWAP State: Discovery

[*09/09/2019 04:55:30.9385] Did not get log server settings from DHCP.

[*09/09/2019 04:55:41.0000] CAPWAP State: DTLS Setup

[*09/09/2019 04:55:41.3399] Bad certificate alert received from peer.

[*09/09/2019 04:55:41.3399] DTLS: Received packet caused DTLS to close connection

The Solution

If the certificate of one or more of your APs has expired you got two options. You can either:

1) Turn off NTP and manually set the clock of your WLC to a time and date where the certificates are still valid. The access point will also get their clock updated when they try to join the WLC so there is no need to set the time directly on the access point.

If you are currently using NTP to set the time of the WLC, use the command show time to see which NTP-servers you are using at the moment. All of your NTP servers are assigned an index number (like 1, 2, 3, and so on) and you need to refer to that index number to delete the NTP-server from being used. In my case, I only have one NTP-server configured so all I need to delete is the one with index 1. After that, we will set the time manually.

   (Cisco Controller)> config time ntp delete 1

   (Cisco Controller)> config time manual 09/30/18 11:30:00

NTP-servers can also be deleted in the web-GUI under Controller > NTP > Server and then hover your mouse over the little blue triangle next to each NTP-server to see an option to remove them. Time can also be configured manually in the web-GUI under Commands > Set Time

2) Disable the device certificate authentication completely and let the AP join the WLC anyway using:

   (Cisco Controller)> config ap cert-expiry-ignore mic enable

If the certificate of your WLC has expired you may need to use both workarounds to get newer access points to join the WLC at all. Depending on your WLC version, only using one of the workarounds might not work as there were some changes to these workarounds in version 8.5 of the AireOS software, which is the operating system of the WLCs.

How do I check when my WLC’s device certificate expires?

Maybe you haven’t been hit by the issues above yet but you want to know how much time you have left until you either need to apply the workarounds or invest in new hardware. SSH into your WLC and run the following command to list all certificates installed in your WLC. Only one of the certificates installed in the WLC is used for device authentication towards the access points, so make sure to look for this one (“Cisco device cert”):

(Cisco Controller)> show certificate all

Scroll down until you find this particular certificate and check the Validity End Date:

----------------------------
Certificate Name: Cisco SHA1 device cert

     Subject Name :
         C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT5508-K9-ccd8e14058a0, [email protected]
     Issuer Name :
         O=Cisco Systems, CN=Cisco Manufacturing CA
     Serial Number (Hex):
         792EA1F60000002140AA
     Validity :
         Start : Sep 10 14:29:13 2009 GMT  
         End   : Sep 10 14:39:13 2019 GMT <<<<<<<<<<<<<<<<<<<<<
     Signature Algorithm :
         sha1WithRSAEncryption
     Hash key :
         SHA1 Fingerprint  : 3d:54:c0:b3:8b:ab:8f:51:dd:28:04:ed:54:77:16:1c:ae:c6:aa:bb
         SHA256 Fingerprint  : b8:94:86:6b:9f:04:bd:8c:0c:43:c6:46:c4:30:f7:9f:59:0c:f9:97:da:c9:07:e6:2f:18:c8:0e:aa:ae:bb:2d