I ran into an issue where some older Cisco Aironet APs (Access Points) stopped connecting to a Cisco WLC (Wireless LAN Controller). No config changes had been made and some of the Cisco Aironet APs would connect and some wouldn’t. All of them were the same model, the Cisco Aironet APs were able to ping the Cisco WLC and vice versa.
What happened is the Cisco MIC (Manufacture Installed Certificate) expired and the default setup of a Cisco WLC is to reject any Cisco Aironet AP with an expired MIC.
It looks like this could impacts every Cisco WLC when used with older Cisco Aironet APs that have an expired Cisco MIC. Cisco has a Field Notice about this issue, you can read it here FN63942.
Any Cisco Aironet AP that was manufactured from July 18, 2005 until 2017 will have a Cisco MIC that expires 10 years after the manufacture date. There seems to be no way to replace or renew that Cisco MIC, this will keep being an issue that could randomly show up until 2027 when all of them should be broken.
The reason some of my Cisco Aironet APs worked and some didn’t is because they were manufactured at different times even though they have the same model number.
The fix is super quick we just need to tell the Cisco WLC to ignore expired Cisco MICs.
Before we just ignore the expired Cisco MICs we should confirm that it is the actual issue. Here are a few ways to do that.
Confirming with Logs
You can run the command show msglog on the Cisco WLC to see if you have entries like this.
%SSHPM-3-GENERIC_CERT_ERROR: [SA]sshpmPkiApi.c:2237 Certificate validation failed! Reason Cisco user certificate not verified by cisco root., Certificate type : MIC, Certificate issuer :Cisco Certificate
Confirming with Serial Numbers
You can also use the serial number of the Cisco Aironet AP to confirm your manufacture date to see when the Cisco MIC might expire.
To get the serial number you can do so from the Cisco WLC or from the Cisco Aironet AP.
On the Cisco WLC run the command show ap inventory all
The output should look something like this.
Inventory for AP01
NAME: "AP3600" , DESCR: "Cisco Aironet 3600 Series (IEEE 802.11n) Access Point"
PID: AIR-CAP3602I-A-K9, VID: V01, SN: FGL1712xxxx
On a Cisco Aironet AP run the command show version
The Processor board ID is the serial number or you can use the Top Assembly Serial Number
The output should look something like this
cisco AIR-CAP3602I-A-K9 (PowerPC) processor (revision A0) with 188398K/60928K bytes of memory.
Processor board ID FGL1712xxxx
PowerPC CPU at 800Mhz, revision number 0x2151
Last reset from power-on
1 Gigabit Ethernet interface
2 802.11 Radios
32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address:
Part Number : 73-14521-02
PCB Serial Number :
Top Assembly Part Number : 800-35852-02
Top Assembly Serial Number : FGL1712xxxx
Top Revision Number : C0
Product/Model Number : AIR-CAP3602I-A-K9
With the serial number we can decode it to figure out when it was manufactured to get an idea of when the Cisco MIC might expire.
Use the Cisco Field Notice FN63942 to gather the decoding info.
In my example my serial number is FGL1712xxxx the first 3 characters mean nothing to us so we can ignore them. We are left with 1712xxxx.
The last 4 characters mean nothing to us so we can ignore them. We are now left with 1712 if we decode it then we know 17 means 2013 and 12 means March.
Now we know our Cisco Aironet AP was manufactured in March 2013 and because it was before 2017 that means the Cisco MIC will expire in 10 years leaving us with an expiry of March 2023.
Confirming with certificates
Another way to confirm the issue is to look at the config on the Cisco Aironet AP itself and decode the certificates on it.
On the Cisco Aironet AP run the command show running-config
The part we care about will look something like this
crypto pki certificate chain chain Cisco_IOS_MIC_cert
certificate ca 02
6F20526F 6F742043 41204D32 301E170D 31323131 31323133 35303538 5A170D33
37313131 32313330 3031375A 3036310E 300C0603 55040A13 05436973 636F3124
30220603 55040313 1B436973 636F204D 616E7566 61637475 72696E67 20434120
Cisco stores the certificates in hex. To decode the certificate we need to convert from hex to base64 then add the normal -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- parts to it and then we can read it. I used CyberChef to do all of this. Here’s the recipe I used.
It should look something like this
Looking at the output we can see that the Cisco MIC for that Cisco Aironet AP expired on 7 March 2023.
The Fix
- SSH into the Cisco WLC
- run the command
config ap cert-expiry-ignore mic enable
We can also tell it to ignore expired Self Signed Certificates too.
- run the command
config ap cert-expiry-ignore ssc enable
- Save your changes by running the command
save config
After that everything should just start working again.
