This document describes a problem encountered with Nexus 5000 Series switches and also provides a solution and temporary workaround for the problem.
Cisco recommends that you have knowledge of Cisco Nexus CLI.
The information in this document is based on Nexus 5000 Series switches that run any version earlier than 5.0(3)N2(1).
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
The Nexus 5000 Series switch reports this error every three minutes:
N5k %SYSMGR-2-VOLATILE_DB_FULL: System volatile database usage is
unexpectedly high at 80%.
N5k %SYSMGR-2-VOLATILE_DB_FULL: System volatile database usage is
unexpectedly high at 80%.
N5k %SYSMGR-2-VOLATILE_DB_FULL: System volatile database usage is
unexpectedly high at 80%.
N5k %SYSMGR-2-VOLATILE_DB_FULL: System volatile database usage is
unexpectedly high at 80%.
When you enter the show run command (and produce more than 190 lines of output) or the show run switch-profile command (regardless of the output size), a copy of the CSM_ACFG file in /dev/shm file might generate. These files are not automatically cleaned out. Eventually they fill up the volatile memory, which causes the device to reload. Additionally, the issue is exacerbated if some sort of script is used which periodically collects or alters the configuration.
In order to troubleshoot this issue, first collect the output from the show system internal flash command. It should show high usage in the /dev/shm folder:
N5k# show system internal flash
Mount-on 1K-blocks Used Available Use% Filesystem
/ 204800 115408 89392 57 /dev/root
/proc 0 0 0 0 proc
/post 2048 4 2044 1 none
/sys 0 0 0 0 none
/isan 1536000 452496 1083504 30 none
/var/tmp 307200 704 306496 1 none
/var/sysmgr 1024000 6320 1017680 1 none
/var/sysmgr/ftp 409600 48604 360996 12 none
/var/sysmgr/ftp/cores 102400 0 102400 0 none
/callhome 61440 0 61440 0 none
/dev/shm 524288 427248 97040 80 none <<<<<<<<<
/volatile 153600 0 153600 0 none
/debug 20480 4 20476 1 none
/dev/mqueue 0 0 0 0 none
/mnt/cfg/0 114909 4904 104072 5 /dev/sda5
/mnt/cfg/1 112920 4904 102186 5 /dev/sda6
/var/sysmgr/startup-cfg 307200 9940 297260 4 none
/dev/pts 0 0 0 0 devpts
/mnt/plog 56192 1644 54548 3 /dev/mtdblock2
/mnt/pss 114917 5348 103635 5 /dev/sda4
/bootflash 1609984 410536 1117664 27 /dev/sda3
In order to confirm that the contents in the /dev/shm folder are the csm_acfg files, collect the output from these commands:
If the Nexus 5000 Series switch crashes, it reports this message as a reload reason in the show system reset-reason command output:
Reason: Reset triggered due to HA policy of Reset
System version: 5.0(2)N2(1)
Service: syslogd hap reset
When the show logging nvram command is entered, the output shows errors such as:
N5k %$ VDC-1 %$ %PSS-0-PSS_WRITE_LOG_FAILURE: snmpd: failed to write log: No space left on device
N5k %$ VDC-1 %$ last message repeated 4 times
Upgrade to version 5.0(3)N2(1) in order to resolve this issue. For more informaton about this issue, reference Cisco bug ID CSCtn71292.
Additionally, as a temporary workaround:
Revision | Publish Date | Comments |
---|---|---|
1.0 |
02-Jul-2013 |
Initial Release |