Introduction
This document describes the steps to resolve Full Disk Space Error in Secure Web Appliance (SWA).
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Administrative access to the SWA
- FTP access to SWA
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Errors Related to Full Disk
There are different errors and warnings in SWA that indicate the disk is full or the disk space is nearly full. Here is the list of errors and warnings. These logs are different in each software version and due to the delivery methods, such as alerts, system logs, or the output of the displayalerts
command from the CLI.
Processing of collected reporting data has been disabled due to lack of logging disk space. Disk usage is above 97 percent.
User admin Disk space for /data has exceeded threshold value 90% with current capacity of 99 %
The reporting/logging disk is full on a WSA
This appliance has disk usage that is higher than expected.
WARNING: Data partition utilization on appliance is high and can cause issues
Monitor Disk Usage
You can monitor and view disk usage from both the GUI and CLI.
View Disk Usage in GUI
After you log in to the SWA GUI, in My-Dashboard Page, you can see Reporting / logging Disk
usage from the System Overview
section.
Note: In SWA reports and logs are stored in a single partition, known as the DATA Partition.
Also in the GUI, under the Reporting
menu, navigate to System Status
. Alternately, you can view disk usage from the Overview
section, under the Reporting
menu.
View Disk Usage in CLI
- From the output of
status
or status detail
, you can see the Reporting / logging Disk
usage
SWA.CLI> status
Enter "status detail" for more information.
Status as of: Sun Feb 19 19:55:13 2023 CET
Up since: Sat Feb 11 14:00:56 2023 CET (8d 5h 54m 17s)
System Resource Utilization:
CPU 25.9%
RAM 13.6%
Reporting/Logging Disk 58.1%
- From the output of
ipcheck
, you can see the allocated disk space to each partition and the percentage of used space per partition.
SWA.CLI> ipcheck
...
Disk 0 200GB VMware Virtual disk 1.0 at mpt0 bus 0 scbus2 target 0 lun 0
Disk Total 200GB
=== Skiped ===
Root 4GB 65%
Nextroot 4GB 1%
Var 400MB 29%
Log 130GB 8%
DB 2GB 0%
Swap 8GB
Proxy Cache 50GB
=== Skiped ===
- In SHD Logs, the
Reporting / logging Disk
utilization for every minute is shown as DskUtil
. To access SHD Logs, use these steps:
- Type
grep
ortail
in the CLI.
- Find
shd_logs
. Type: SHD Logs Retrieval: FTP Poll
, from the list and type the associated number.
- In
Enter the regular expression to grep
, you can type a regular expression to search inside the logs. For example, you can type date and time.
Do you want this search to be case insensitive? [Y]>
, you can leave this as the default, unless you need to search for case sensitivity, which in SHD logs, you do not need this option.
Do you want to search for non-matching lines? [N]>
, you can set this line as default, unless you need to search for everything except your grep regular expression.
Do you want to tail the logs? [N]>
. This option is only available in the output of the grep, if you let this as default (N), it shows the SHD Logs from the first line of current file.
Do you want to paginate the output? [N]>
. If you select Y
, the output is the same as the output of the less command. You can navigate between lines and pages. Also, you can search inside the logs (Type /then the keyword and press Enter
). To exit the log, type q
.
In this example 52.2% of Reporting / logging Disk
is consumed.
Mon Feb 20 23:46:14 2023 Info: Status: CPULd 66.4 DskUtil 52.2 RAMUtil 11.3 Reqs 0 Band 0 Latency 0 CacheHit 0 CliConn 0 SrvConn 0 MemBuf 0 SwpPgOut 0 ProxLd 0 Wbrs_WucLd 0.0 LogLd 0.0 RptLd 0.0 WebrootLd 0.0 SophosLd 0.0 McafeeLd 0.0 WTTLd 0.0 AMPLd 0.0
I
Disk Structure and Troubleshoot Full Partition
As mentioned before from the output of ipcheck
, there are seven partitions in the SWA:
Partition Name |
Description |
Root |
Keeps internal Operation system files |
Nextroot |
This Partition is used for Upgrade |
Var |
Keeps internal Operation system files |
Log |
Holds logs and Reporting file |
DB |
Configuration and internal Databases |
Swap |
SWAP memory |
Proxy Cache |
Keeps Cached data |
Root Partition is Full
If the root partition (known as rootfs or /) is full or more than 100%, which is sometimes expected, and the SWA removes unnecessary files.
If you see some system performance drop, first try to reboot the appliance, then again check the disk capacity of the root partition. If the issue still persists, please contact Cisco Customer Service, to open a TAC case.
Next Root Partition is Full
If your upgrade fails, make sure, your Next Root Partition is free or has enough free space for the upgrade,
Initially, virtual SWA, Email Security Appliance (ESA) and virtual Security Management Appliance) SMA images were built with a Nextroot partition size of less than 500MB. Over the years, and with newer AsyncOS releases that include additional features, upgrades have had to use more and more of this partition throughout the upgrade process. Some times when you try to upgrade from older versions, upgrades fail because of this partition size.
From upgrade logs in the CLI, you can see these errors:
Finding partitions... done.
Setting next boot partition to current partition as a precaution... done.
Erasing new boot partition... done.
Extracting eapp done.
Extracting scanerroot done.
Extracting splunkroot done.
Extracting savroot done.
Extracting ipasroot done.
Extracting ecroot done.
Removing unwanted files in nextroot done.
Extracting distroot
/nextroot: write failed, filesystem is full
./usr/share/misc/termcap: Write failed
./usr/share/misc/pci_vendors: Write to restore size failed
./usr/libexec/getty: Write to restore size failed
./usr/libexec/ld-elf.so.1: Write to restore size failed
./usr/lib/libBlocksRuntime.so: Write to restore size failed
./usr/lib/libBlocksRuntime.so.0: Write to restore size failed
./usr/lib/libalias.so: Write to restore size failed
./usr/lib/libarchive.so: Write to restore size failed
For a virtual SWA, download a new image file per this document : Cisco Secure Email and Web Virtual Appliance Installation Guide
Then try to import the configuration backup from the older version to the newly installed SWA. If you see the Configuration Import Error
, please open a Service Request Case.
For SMA and ESA, you can find the workaround for this issue from this link: How to Apply the Workaround for Cisco vESA/vSMA Upgrade Fail Due to Small Partition Size - Cisco
Var Partition is Full
If the Var
partition is full, you get these errors when you log in to CLI or from the Displayalerts
command in CLI:
/var: write failed, filesystem is full
The temporary data partition is at 99% capacity
To resolve this issue, first restart the appliance. If the capacity of /var partition is still more than 100%, contact Cisco TAC support.
Reporting/logging Partition is Full
If the Reporting / Logging partition is full, the errors can be:
Processing of collected reporting data has been disabled due to lack of logging disk space. Disk usage is above 97 percent.
User admin Disk space for /data has exceeded threshold value 90% with current capacity of 99 %
The reporting/logging disk is full on a WSA
WARNING: Data partition utilization on appliance is high and can cause issues
The root cause of these errors can be categorized as:
- Log files occupy too much disk space.
- There are some core files generated on the device that leads to full disk usage.
- Reporting occupies too much disk space.
- Web Tracking occupies too much disk space.
- Some internal logs occupy too much disk space.
Log Files Occupy too much Disk Space
To view log files, you can connect to the SWA by FTP to the management interface.
Note: FTP by default is disabled.
To enable FTP from the GUI, use these steps:
Step 1. Log in to the GUI.
Step 2. Click Interfaces
under the Network
menu.
Step 3. Click Edit Settings
.
Step 4. Select FTP
from the Appliance Management Services
section.
Step 5. (Optional) you can change the default FTP Port.
Step 6. Click Submit
.
Step 7. Commit Changes.
After the FTP connection, you can view the logs, the creation date, and size of each log file. If you need to archive the logs, you can download them from FTP. Or to free up disk space, you can remove old logs.
Use these steps to resolve this issue:
Tip: If you see the log files did not occupy much disk space, most likely the issue is related to reports or core files.
Core Files on the Device
To view if SWA has core files, from CLI use these steps:
Step 1. Log in to CLI.
Step 2. Execute the command: diagnostic
(it is a hidden command and cannot be auto filled with TAB).
Step 3. Type PROXY
.
Step 4. Type LIST
.
The output shows if there are any core files. To remove the core files, contact Cisco Support Service, a TAC Engineer needs to investigate the cause of core files, then they can remove the files.
Reporting Occupies too much Disk Space
There are two types of reports in SWA: Reporting and WebTracking. WebTracking occupies most of the disk space.
To check the history of WebTracking, navigate to WebTracking
from the GUI. Under the Reporting
menu, from the Time Range
section, select Custom Range
, The highlighted dates show the WebTracking report history.
To take a back-up from WebTracking, you can export the report to CSV from the Printable Download
link in the report.
Tip: Avoid the generation of WebTracking reports for long time periods, which depends on the normal daily web traffic. The reports for longer durations, can cause the SWA to became unresponsive.
At the time this article is written, there is no feature to manually delete older reports. (Cisco bug ID CSCun82094)
In order to delete some of your reports, you need to contact TAC Support, or you can delete all of the reports from the CLI with these steps:
Step 1. Log in to CLI.
Step 2. Execute the diagnostic
command. (It is a hidden command and cannot be auto completed with TAB.)
Step 3. Type REPORTING
and press Enter
.
Step 4. Type DELETEDB
and press Enter
.
Caution: This command deletes all reports data. It cannot be aborted.
Internal Logs Occupy Disk
If your device has the conditions of the defect: Cisco bug ID CSCvy69039, you need to open a TAC case to check the internal logs from the back end and remove the large log files manually.
This is a temporary workaround, but on the affected version, the log file is auto created after deletion and file size repeatedly grows from 0 again.
Related Information