Introduction
This document describes how to troubleshoot and clear the "START_BACKING_THREAD:BIND: no SPACE LEFT" error in NCS1K.
Problem Description
Observing start_backing_thread:bind: No space left on the device unable to monitor alarms in the device (rootfs is filled 100%). PM logs collection in rootfs directory and filling it up. Version: XR 7.3.2
Observations and Actions Performed
- Command
<show alarms b s a>
output is not displayed and observed error "start_backing_thread:bind: No space left on device" and rootfs is fully occupied.
- Rootfs is filled 100%.
Show Media Output
- Pm log files are filled in rootfs directory.PM logs stored in rootfs
Procedure to Delete PM Files
This command displays all the files in the root directory.
RP/0/RP0/CPU0:MUMNGD-MCD-O-NK4-01#run
[xr-vm_node0_RP0_CPU0:~]$cd /root
[xr-vm_node0_RP0_CPU0:/root]$ls -ltrah
This command deletes the file from the directory.
rm “filename” ----à
Verify the size of rootfs using show media
command.
RP/0/RP0/CPU0:MUMNGD-MCD-O-NK4-01#show media
- It is important to know why auto deletion did not occur in the system.
- It is identified there is a Python script enabled for automatic log collection every 1 hour in rootfs.
- The pm log generation is not done by the device by default, It is configured manually. Hence, this auto-deletion also needs to be done manually.
- Need to verify the source from where the pm log files are being stored in rootfs to kill this process.
- It is found that some Python scripts are running in the background to collect pm logs in rootfs in 2 nodes out of 4 ncs1k4 nodes in their network.
- These were added during deployment time.
Hard Disk Files
To kill the Python process, use the reload
command in XR.
Resolution:
To terminate these Python processes, you need to XR RELOAD
on the nodes.
XR RELOAD
- Command reloads the VM.
RP/0/RP0/CPU0:MUMNGD-MCD-O-NK4-01#reload (Reloads the VM.)
Reloading does not impact service but you must lose management visibility for around 5-10 mins. You can also clean up the pm-related files using rm
command:
<rm -rf "/misc/disk1/pm_data">
<rm -rf "/root/*pm.txt">
<rm -rf "/root/PM-data*">
<rm -rf *2022.tgz (deletes all the files ending with 2022.tgz)>
Similarly, you can delete files that are not needed.