Introduction
This document describes the recovery procedure of Catalyst 9115/91120 access points that boot in the U-boot prompt instead of normal operating system.
Background Information
Between Oct. 20 and Oct. 27, 2021, Cisco manufactured a number of C9115 and C9120 access points with a bad image. These units log "uncorrectable ECC error" messages to the console during bootup, and are unable to boot successfully. To recover them, you must console into each AP, and load in a special image via TFTP.
This issue is documented as Field Notice: FN - 72278 - Some C9120 and C9115 APs Might Get Stuck in Bootup with 'Uncorrectable ECC Error' Displayed - Workaround Provided and as the Cisco bug ID CSCwa12652.
Recovery Procedure - Overview
- Download the special image bundle-axel-SS-8_10_130_0.img.
- Put the image on a TFTP server.
- Console into the affected AP.
- Break into u-boot mode.
- Use u-boot mode to copy the image into the AP via TFTP.
Detailed Steps
Download Special Image Bundle-Axel-SS-8_10_130.0.img
- Browse to the hidden posting https://software.cisco.com/download/specialrelease/aa90a6d87be2275f1f0111c4973295c2
- Download the Catalyst 9115AX/9120AX Access Point Manufacture Image bundle-axel-SS-8_10_130_0.img
Note: These images are only to be used to recover an unbootable AP.
Copy Special Image to TFTP Server
Set up a TFTP server in a network that is cabled to the AP's LAN port (labeled "2.5G").
See Appendix A for an example of how to set up the Tftpd64 server in Windows.
Get Into U-boot on the Console
- Connect a serial cable to the AP's console port (this has a pale blue label that is not labeled "2.5G"). The serial port can be configured for 9600 bps, 8 databits, 1 stopbit, no parity, no flow control.
See Appendix B for an example that uses MobaXterm to connect to the serial port.
- Power up the AP.
As soon as the AP starts powering up, enter the ESC key repetitively in the terminal window, until you see the u-boot prompt.
Verify that the firmware has been loaded with good CRC: OK
Firmware loading completed successfully
bcm4908_eth-0
MAC: 70:69:5a:76:40:3c
Hit ESC key to stop autoboot:
u-boot>
Download Special Image into AP Flash
Configure u-boot to access the TFTP server via IP. Text manually entered via the terminal is in bold.
- Configure the AP with an unused IP address in the TFTP server's subnet.
u-boot> setenv ipaddr 10.1.1.101
- Configure the AP's netmask.
u-boot> setenv netmask 255.255.255.0
- Configure the IP address of the TFTP server.
u-boot> setenv serverip 10.1.1.1
- Configure the IP address of the default gateway. If the TFTP server is on the same subnet as the AP, you can enter the TFTP server's address here.
u-boot> setenv gatewayip 10.1.1.1
- Clear the target TFTP directory, and save the settings.
Note: To verify the current settings, you can use the printenv command.
u-boot> setenv tftpdir
u-boot> saveenv
Saving Environment to SPI Flash...
Erasing at 0x0 -- 100% complete.
SF: Detected mx25l6405d with page size 256 Bytes, erase size 4 KiB, total 8 MiB
Erasing SPI flash...Writing to SPI flash...done
- Ping the TFTP server to make sure it is reachable.
u-boot> ping 10.1.1.1
Use bcm4908_eth-0 device
host 10.1.1.1 is alive
- Download the image into AP flash and provide the answers to the prompts like in this example (the interactive prompts can apear after the image is loaded).
Note: Do not power cycle the AP during this step.
u-boot> boardinit bundle-axel-SS-8_10_130_0.img
// Some output here was removed in order to emphasize the interactive prompts. Full outputs are shown after the final step.
Program PHY firmware? [y/N]: y
Program UBIFS image? [y/N]: y
Program bootloaders? [y/N]: n //pay attention: the last option must be “n”
- Reboot the AP and use the reset command. Once it is fully rebooted, it must respond to Enter with a "Username:" prompt.
u-boot> reset
Example output during flashing:
u-boot> boardinit bundle-axel-SS-8_10_130_0.img
Download image bundle-axel-SS-8_10_130_0.img...
Use bcm4908_eth-0 device
TFTP from server 10.1.1.1; our IP address is 10.1.1.104
Filename 'bundle-axel-SS-8_10_130_0.img'.
Load address: 0x501144b0
Loading: #################################################################
// Note: The # symbols can continue to print until the image is fully transfered, this can take a few minutes //
done
Bytes transferred = 147999664 (8d24bb0 hex)
Program PHY firmware? [y/N]: y
Writing PHY firmware to NOR flash...
>>> Backup current PHY firmware image @300000:45000...[Success]
>>> Erase flash blocks ..........[Success]
>>> Write PHY firmware image ...[Success]
[Success]
Program UBIFS image? [y/N]: y
Writing UBIFS to NAND...
NAND erase.part: device 0 offset 0x6c0000, size 0x3f940000
Skipping bad block at 0x00e00000 ……
Erasing at 0x3ffc0000 -- 100% complete.
OK
NAND write: device 0 offset 0x6c0000, size 0x8c80000
Skip bad block 0x00e00000
147324928 bytes written: OK
[Success]
Program bootloaders? [y/N]: n //pay attention: the last option must be “n”
u-boot>
u-boot> reset
Appendix A - Set Up Tftpd64 in Windows
In this example, a Windows PC is manually configured with the address 10.1.1.1, and Ttftpd64 is set up to serve the special image.
Configure PC's Address as 10.1.1.1
Download Tftpd64
TFTPD64 Software
Copy Special Image to TFTP Base Folder
Copy the bundle-axel-SS-8_10_130_0.img file into the TFTP base directory (for example, C:\tftpboot).
Appendix B - Connect to AP Console via MobaXterm
Appendix C - Example Topologies
Option 1 - Multiple APs, PoE Switch and Terminal Server
Option 2 - Multiple APs, PoE Switch and USB Hub with Multiple USB-to-RS232 Adapters
Option 3 - Single AP, PoE Injector