Introduction
This document describes the process to diagnose and resolve the issue when a Cisco IOS® XE SD-WAN upgrade fails due to insufficient storage capacity.
Prerequisites
WAN Edge Router
Requirements
Cisco recommends that you have knowledge of these topics:
- Cisco Software-Defined Wide Area Network (SD-WAN)
- Cisco IOS® XE Command Line Interface (CLI)
For Cisco IOS XE SD-WAN upgrade information Upgrade Cisco IOS XE SD-WAN Router with the Use of CLI or vManage
Components Used
This document is based on these software and hardware versions:
- Router c1111-8P
- Software Version 17.3.2
- Software Version 17.3.4
- Router in controller mode
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Conditions
1. Unable to upgrade the router model c1111-8p to 17.3 and other versions before 17.5.
2. Consider these errors:
Not enough space under bootflash for image download and package expansion on image installation. Required space: 1207202432 Available space: 659275776. Image installation failed.
Not enough space in SD-WAN installer filesystem. Please remove one of the installed images and try again. SD-WAN installer filesystem size - 396672 Kilobytes, Available space - 196986 Kilobytes, Required space - 243602 Kilobytes.
3. Upgrade fails as shown in these images:
Image1
Image2
4. There is only one version installed. Use the command show sdwan software
for verification.
Example
edge#show sdwan software
VERSION ACTIVE DEFAULT PREVIOUS CONFIRMED TIMESTAMP
-----------------------------------------------------------------------------------
17.03.04a.0.5574 true true false user 2021-10-22T23:00:30-00:00
Total Space:387M Used Space:191M Available Space:192M
Troubleshoot
1. Delete unnecessary files with this command:
delete /force /recursive bootflash:/<directory>/<file ext>
You can delete the next list of files. They are internal logs created for troubleshooting purposes.
Caution: Delete them if you are sure there is no other investigation in progress.
bootflash:/vmanage-admin/*.tar.gz
bootflash:/core/*.core.gz
bootflash:/core/*.tar.gz
bootflash:/syslog/*
bootflash:/tracelogs/*
bootflash:/crashinfo_*
Example
delete /force /recursive bootflash:/tracelog/*
delete /force /recursive bootflash:/ vmanage-admin/*.tar.gz
Warning: Do not delete these files. They are essential files for router functionality.
bootflash:/.sdwaninstaller
bootflash:/.sdwaninstallerfs
bootflash:/.cdb_backup folder
bootflash:/sdwan folder
bootflash:/*pkg
bootflash:/*.conf.
2. Reboot helps to release bootflash memory as shown in the example.
Caution: Control plane and Data plane flaps during reboot.
*** Before Reboot ***
edge#show sdwan software
VERSION ACTIVE DEFAULT PREVIOUS CONFIRMED TIMESTAMP
-----------------------------------------------------------------------------------
17.03.04a.0.5574 true true false user 2021-10-22T23:00:30-00:00
Total Space:387M Used Space:191M Available Space:192M >>> Here
edge#
*** Rebooting edge
edge#reload
Proceed with reload? [confirm]Connection to 192.168.1.6 closed by remote host.
*** After Reboot ***
edge#show sdwan software
VERSION ACTIVE DEFAULT PREVIOUS CONFIRMED TIMESTAMP
-----------------------------------------------------------------------------------
17.03.04a.0.5574 true true false user 2021-10-22T23:00:30-00:00
Total Space:387M Used Space:190M Available Space:193M >>>> Here
Work Around
Apply the next steps if previous actions did not work to upgrade the router c1111-8p.
Issue 1
If the upgrade issue is related to this error Not enough space under bootflash, reference to the next instructions.
Step 1. Enable IOx.
Config-transaction
iox
commit
end
Step 2. Run the next command to move the IOx log from bootflash to flash.
app-hosting move system techsupport to flash:
Step 3. Disable IOx.
Config-transaction
no iox
commit
end
Example
Before IOx enabling
edge#dir
Directory of bootflash:/
<snipped>
2908606464 bytes total (646643712 bytes free)
edge#
After IOx disabling
edge#dir
Directory of bootflash:/
<snipped>
2908606464 bytes total (1671614464 bytes free)
edge#
Step 4. Proceed to upgrade.
request platform software sdwan software install bootflash:<version.bin>
request platform software sdwan software activate <IOS version>
request platform software sdwan software upgrade-confirm
request platform software sdwan software set-default <IOS version>
Tip: For Upgrade via vManage Upgrade SD-WAN cEdge Router with the Use of CLI or vManage
Issue 2
If the upgrade issue is related to this error 'Not enough space in SD-WAN installer filesystem', then reference these instructions.
Step 1. Log in to the shell.
For more information, reference to .
Step 2. Go to the .sdwaninstaller/images directory. Use Linux commands to find and remove old files.
cd /bootflash/.sdwaninstaller/images
ls -l
rm -rf <old file>
Step 3. Exit from the shell with Linux command.
exit
step 4. Proceed to upgrade.
request platform software sdwan software install bootflash:<version.bin>
request platform software sdwan software activate <IOS version>
request platform software sdwan software upgrade-confirm
request platform software sdwan software set-default <IOS version>
Tip: For Upgrade via vManage Upgrade SD-WAN cEdge Router with the Use of CLI or vManage
Example
**** Before to Clean Up ****
[23-Sep-2022 18:08:05 UTC] Installing software image
[23-Sep-2022 18:08:09 UTC] Current active partition: 17.03.04a.0.5574
[23-Sep-2022 18:08:10 UTC] Configuring upgrade confirm timer to 15 minutes
[23-Sep-2022 18:08:11 UTC] Software image: c1100-universalk9.17.03.05.SPA.bin
[23-Sep-2022 18:08:11 UTC] Software image download may take upto 180 minutes
[23-Sep-2022 18:08:17 UTC] Not enough space in SDWAN installer filesystem. Please remove one of the installed images and try again. SDWAN installer filesystem size - 396672 Kilobytes, Available space - 196986 Kilobytes, Required space - 243602 Kilobytes.
[23-Sep-2022 18:08:17 UTC] Failed to process device request -
Error type : application
Error tag : operation-failed
Error info : <error-info>
<bad-element>install</bad-element>
</error-info>
edge#show sdwan software
VERSION ACTIVE DEFAULT PREVIOUS CONFIRMED TIMESTAMP
-----------------------------------------------------------------------------------
17.03.04a.0.5574 true true false user 2021-10-22T23:00:30-00:00
Total Space:387M Used Space:190M Available Space:193M >>> Here
edge#
**** Clean old files from sdwaninstaller ****
[edge:/]$ cd /bootflash/.sdwaninstaller/images
[edge:/bootflash/.sdwaninstaller/images]$ ls -l
total 2
drwxr-xr-x. 3 root root 1024 Oct 3 2019 SR_BOOT=usb0:c1100-ucmk9.16.9.3.SPA.bin
drwxr-xr-x. 3 root root 1024 Sep 23 13:33 c1100-universalk9.17.03.04a.SPA.bin
[edge:/bootflash/.sdwaninstaller/images]$ rm -rf SR_BOOT=usb0:c1100-ucmk9.16.9.3.SPA.bin
[edge:/bootflash/.sdwaninstaller/images]$ ls -l
total 1
drwxr-xr-x. 3 root root 1024 Sep 23 13:33 c1100-universalk9.17.03.04a.SPA.bin
[edge:/bootflash/.sdwaninstaller/images]$ exit
exit
Session log harddisk:tracelogs/system_shell_R0-0.18547_0.20220923134926.bin closed.
edge#
edge#show sdwan software
VERSION ACTIVE DEFAULT PREVIOUS CONFIRMED TIMESTAMP
-----------------------------------------------------------------------------------
17.03.04a.0.5574 true true false user 2021-10-22T23:00:30-00:00
Total Space:387M Used Space:120M Available Space:262M >>>> Space increased
**** Upgrade Successful:
[23-Sep-2022 18:53:38 UTC] Installing software image
[23-Sep-2022 18:53:43 UTC] Current active partition: 17.03.04a.0.5574
[23-Sep-2022 18:53:44 UTC] Configuring upgrade confirm timer to 15 minutes
[23-Sep-2022 18:53:44 UTC] Software image: c1100-universalk9.17.03.05.SPA.bin
[23-Sep-2022 18:53:44 UTC] Software image download may take upto 180 minutes
[23-Sep-2022 18:53:54 UTC] Connection Instance: 2, Color: private1
[23-Sep-2022 18:53:54 UTC] Downloading http://192.168.16.17:8080/software/package/c1100-universalk9.17.03.05.SPA.bin?deviceId=10.10.10.10
[23-Sep-2022 19:06:04 UTC] Downloaded http://192.168.16.17:8080/software/package/c1100-universalk9.17.03.05.SPA.bin?deviceId=10.10.10.10
[23-Sep-2022 19:09:31 UTC] install_add: START Fri Sep 23 14:06:10 COT 2022
Sep 23 14:06:12.207 %INSTALL-5-INSTALL_START_INFO: R0/0: install_engine: Started install add bootflash:c1100-universalk9.17.03.05.SPA.bin
[23-Sep-2022 19:09:31 UTC] Installed 17.03.05.0.6600
Caveat
Possible Problems during the upgrade via vManage GUI:
Fix: verify the bootflash and delete all the images you are not using. You can delete all the packages via the CLI file with the command # delete bootflash:*17.03.01*.
This command deletes all the files that have the name 17.03.01.
Edge#delete bootflash:*17.03.01*
Delete filename [*17.03.01*]?
Delete bootflash:/isr4300-firmware_dreamliner.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_dsp_analogbri.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_dsp_sp2700.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_dsp_tilegx.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_ngwic_t1e1.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_async.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_bri_st_fw.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_cwan.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_ge.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_shdsl.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_ssd.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_xdsl.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_prince.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-rpboot.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_sm_10g.17.03.01a.SPA.pkg? [confirm]
You can also boot from a USB drive, delete all images from bootflash, copy the bin file into the bootflash, and reboot.
Note: If the router is in controller mode all the commands from IOS-XE related to packages do not work.
Note: Change the boot variable via CLI is also disabled in controller mode. You can move the device to autonomous mode, upgrade, and then move it back to controller mode.
-
The image is already present in the router
You can verify if the version is already present via the vManage GUI or CLI with the command show sdwan software
.
Fix: Remove the unused images via the vManage GUI or with the command request platform software sdwan software remove <Name>
.
-
We have more than three images in the router
You can verify if the version is already present via the vManage GUI or CLI with the command show sdwan software
.
Fix: Remove the unused images via the vManage GUI or with the command request platform software sdwan software remove <Name>
.
Related Information