Address ACI faults F1527, F1528, F1529 - fltEqptStorageFull

Introduction

This document describes Application Centric Infrastructure (ACI) fault codes F1527, F1528, and F1529 along with remediation steps.

Background Information

These three faults occur when the utilization of the storage capacity of a controller exceeds its threshold.
F1527 is a warning fault that occurs when the usage is greater than 75%.

F1528 is a major fault that occurs when the usage is between 85% and 90%.

F1529 is a critical fault that occurs when the usage is greater than 90%.

code : F1529
cause : equipment-full
descr : Storage unit /techsupport on node 1 with hostname rtp-aci08-apic1 mounted at /techsupport is 100% full
dn : topology/pod-1/node-1/sys/ch/p-[/techsupport]-f-[/dev/mapper/vg_ifc0-techsupport]/fault-F1529
rule : eqpt-storage-full-critical
severity : critical

Quick Start to Address Fault

1. Identify the directory that is at capacity.
2. Verify that the capacity has been reached.
3. Clean up files in the directory.

Tip: There is also offer an automated tool that can assist you in resolving excessive disk consumption issues. This tool identifies the mount point mentioned in the Fault and provides you with a list of files that can be deleted to free up disk space. You can access the tool at: https://github.com/datacenter/aci-tac-scripts/tree/main/aci-exceeded-storage-allocation

Detailed Steps to Address Fault

Identify the Directory

The directory that the fault is raised against is identified in the fault description.

In the two examples here, you can see that the F1527 fault is raised against the /firmware directory and the F1529 fault is tied to /techsupport.

You can also see in the description that the faults are raised on node 1.

code : F1527
descr : Storage unit /firmware on Node 1 mounted at /firmware is 76% full
dn : topology/pod-1/node-1/sys/ch/p-[/firmware]-f-[/dev/mapper/vg_ifc0-firmware]/fault-F1527

code : F1529
descr : Storage unit /techsupport on node 1 with hostname rtp-aci08-apic1 mounted at /techsupport is 100% full
dn : topology/pod-1/node-1/sys/ch/p-[/techsupport]-f-[/dev/mapper/vg_ifc0-techsupport]/fault-F1529

Verifying Capacity

Once you know which directory the fault is raised on, you can use the CLI to verify that you are using that much drive space.

Using the command  df -h,  you can see the available disk space for each mount.
In the table here, you see that the /firmware is using 76% of its available space and  /data/techsupport  is using 100%.

rtp-aci08-apic1# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/vg_ifc0/boot                 40G   13G   25G  35% /bin
/dev/mapper/vg_ifc0_ssd-data     176G  4.2G  162G   3% /var/log/dme
devtmpfs                          32G     0   32G   0% /dev
tmpfs                            4.0G  182M  3.9G   5% /dev/shm
/dev/mapper/vg_ifc0-firmware      40G   28G  9.3G  76% /firmware
/dev/mapper/vg_ifc0-scratch       40G   49M   38G   1% /home
tmpfs                             32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/vg_ifc0-techsupport   40G   38G     0 100% /data/techsupport
tmpfs                             16G  592K   16G   1% /tmp
/dev/sdc1                         55M  1.2M   49M   3% /tmp/bootflash
tmpfs                            2.0G  721M  1.3G  36% /var/log/dme/log
/dev/mapper/vg_ifc0-logs          40G  5.0G   33G  14% /var/log/dme/oldlog
/dev/mapper/vg_ifc0-data2        156G   11G  137G   8% /data2
/dev/mapper/vg_ifc0-dmecores      50G   53M   47G   1% /var/log/dme/core
tmpfs                             32G  9.0G   23G  29% /var/run/utmp

Clean Up Files

After you have verified the fault condition is present, you can clean up files in the directory.
To do this, navigate to that directory; then you can list the files by size (ls -lahS) and remove any large files (rm <fileName>) that are no longer needed.

You can then verify again with the df -h  command that the space has been cleaned up.

rtp-aci08-apic1# cd /data/techsupport
rtp-aci08-apic1# ls -lahS
total 38G
-rw-r--r--  1 admin   admin    10G Aug 10 18:12 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-49UTC_logs_3of3.tgz
-rw-r--r--  1 admin   admin   9.4G Aug 10 18:13 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-79UTC_logs_3of3.tgz
-r--r-----  1 ifc     admin   3.9G Jul 24 02:05 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T01-49UTC_logs_3of3.tgz
-r--r-----  1 ifc     admin   3.7G Jul 24 01:55 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T01-49UTC_1of3.tgz
-r--r-----  1 ifc     admin   2.5G May 15 19:33 dbgexp_tsod-upgrde427sto524d_rtp-aci08-apic1_sysid-1_2023-05-15T19-24UTC_logs_3of3.tgz
-r--r-----  1 ifc     admin   2.1G May  4 19:17 dbgexp_tsod-failed_upgrade_repro_rtp-aci08-apic1_sysid-1_2023-05-04T19-08UTC_logs_3of3.tgz
-r--r-----  1 ifc     admin   1.1G Aug 10 18:04 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-08-10T17-59UTC_logs_3of3.tgz
-rw-r--r--  1 admin   admin   1.1G Aug 10 18:11 1g.img
-r--r-----  1 ifc     admin   952M May  4 19:17 dbgexp_tsod-failed_upgrade_repro_pod8-spine1_sysid-201_2023-05-04T19-08UTC_logs_3of3.tgz
-r--r-----  1 ifc     admin   946M May  3 19:44 dbgexp_tsod-failed_upgrade_repro_pod8-spine1_sysid-201_2023-05-03T19-35UTC_logs_3of3.tgz
-r--r-----  1 ifc     admin   894M May 15 19:27 dbgexp_tsod-upgrde427sto524d_rtp-aci08-apic1_sysid-1_2023-05-15T19-24UTC_1of3.tgz
-r--r-----  1 ifc     admin   892M May  4 19:12 dbgexp_tsod-failed_upgrade_repro_rtp-aci08-apic1_sysid-1_2023-05-04T19-08UTC_1of3.tgz
-r--r-----  1 ifc     admin   253M Mar 31 20:33 dbgexp_tsod-12345_12345_sysid-105_2023-03-31T20-25UTC_logs_3of3.tgz
-r--r-----  1 ifc     admin   205M Jul 18 14:40 dbgexp_coreexp-default_pod8-spine3_sysid-203_2023-07-18T14-21UTC_1689690102_0x1b01_kernel_log.0.tar.gz
-r--r-----  1 ifc     admin   141M Aug 10 18:02 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-08-10T17-59UTC_db_2of3.tgz
-r--r-----  1 ifc     admin   134M Jul 24 02:00 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T01-49UTC_db_2of3.tgz
-r--r-----  1 ifc     admin   130M May 15 19:29 dbgexp_tsod-upgrde427sto524d_rtp-aci08-apic1_sysid-1_2023-05-15T19-24UTC_db_2of3.tgz
<snip>

rtp-aci08-apic1# rm dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-49UTC_logs_3of3.tgz
rtp-aci08-apic1# rm dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-79UTC_logs_3of3.tgz
rtp-aci08-apic1# df -h | grep techsupport
/dev/mapper/vg_ifc0-techsupport   40G   18G   20G  49% /data/techsupport

Fault Raised on /

If the directory that is full is the / directory, you will not be able to clean up the affected files without being root.

code : F1528
descr : Storage unit / on Node 1 with hostname rtp-aci08-apic1 mounted at / is 89% full
dn : topology/pod-1/node-1/sys/ch/p-[/]-f-[/dev/vg_ifc0/boot]/fault-F1528

When you use the  df -h  command here, you do not see anything mounted on /.
You use that /bin that is 100% full. However, when you at the files there, you only see 606M is used not 40G.

rtp-aci08-apic1# df -h                                             
Filesystem                       Size  Used Avail Use% Mounted on
/dev/vg_ifc0/boot                 40G   40G     0 100% /bin
/dev/mapper/vg_ifc0_ssd-data     176G  4.2G  162G   3% /var/log/dme
devtmpfs                          32G     0   32G   0% /dev
tmpfs                            4.0G  182M  3.9G   5% /dev/shm
/dev/mapper/vg_ifc0-firmware      40G   28G  9.3G  76% /firmware
/dev/mapper/vg_ifc0-scratch       40G   49M   38G   1% /home
tmpfs                             32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/vg_ifc0-techsupport   40G   18G   20G  49% /data/techsupport
tmpfs                             16G  592K   16G   1% /tmp
/dev/sdc1                         55M  1.2M   49M   3% /tmp/bootflash
tmpfs                            2.0G  726M  1.3G  36% /var/log/dme/log
/dev/mapper/vg_ifc0-logs          40G  5.1G   33G  14% /var/log/dme/oldlog
/dev/mapper/vg_ifc0-data2        156G   11G  137G   8% /data2
/dev/mapper/vg_ifc0-dmecores      50G   53M   47G   1% /var/log/dme/core
tmpfs                             32G  7.1G   25G  23% /var/run/utmp
rtp-aci08-apic1# cd /bin 
rtp-aci08-apic1# ls -lahS | head
total 606M
-rwxr-xr-x  1 root root  103M Jul 26 20:44 nomad
-rwxr-xr-x  1 root root   60M Mar  1  2021 podman
-rwxr-xr-x  1 root root   51M Sep  9  2020 containerd
-rwxr-xr-x  1 root root   47M Aug  4  2021 consul
-rwxr-xr-x  1 root root   32M Apr 27  2021 atomix
-rwxr-xr-x  1 root root   30M Apr 27  2021 atomix-downgrade-grub
-rwxr-xr-x  1 root root   26M Sep  9  2020 ctr
-rwxr-xr-x  1 root root   25M Feb 13  2019 etcd
-rwxr-xr-x  1 root root   21M Feb 13  2019 etcdctl

In order to see the actual files taking up the space on /, you need to access the APIC CLI with the root login.
To do this, you need to contact Cisco TAC for assistance.

Defect Validation

CSCwj42027 - DME log folder crossing 75% usage in APIC

Affected software versions may raise a fault with description "Storage unit /data/log on node <#> with hostname <HOSTNAME> mounted at /data/log is <##>% full". Various log files in the /data/log directory fail to rotate at 20M size.

Contact TAC to truncate any large log files and apply a non-persistent workaround to prevent the fault from reoccurring.

Reference: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwj42027