Introduction
This document describes how to troubleshoot and fix an issue on the supervisor in the Cisco WAN Automation Engine (WAE).
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Problem
On WAE 7.3, this is the issue:
[wae@ds1waenod01 etc]$ sudo systemctl status supervisord
● supervisord.service - Process Monitoring and Control Daemon
Loaded: loaded (/usr/lib/systemd/system/supervisord.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2021-12-07 16:52:30 PST; 1 months 14 days ago
Process: 23468 ExecStart=/usr/bin/supervisord -c /etc/supervisord.conf (code=exited, status=1/FAILURE)
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: d.main()
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]:
File "/usr/lib/python2.7/site-packages/supervisor/supervisord.py", line 70, in main
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: rlimit_messages = self.options.set_rlimits()
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]:
File "/usr/lib/python2.7/site-packages/supervisor/options.py",
line 1316, in set_rlimits
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: self.usage(msg % locals())
Dec 07 16:52:30 ds1waenod01.tnt.lab supervisord[23468]: KeyError: 'min'
Dec 07 16:52:30 ds1waenod01.tnt.lab systemd[1]: supervisord.service: control process exited,
code=exited status=1
Dec 07 16:52:30 ds1waenod01.tnt.lab systemd[1]: Failed to start Process Monitoring and Control Daemon.
Dec 07 16:52:30 ds1waenod01.tnt.lab systemd[1]: Unit supervisord.service entered failed state.
Dec 07 16:52:30 ds1waenod01.tnt.lab systemd[1]: supervisord.service failed.
Due to this, supervisor is rendered inactive. Therefore, it is not up:
[wae@ds1waenod01 root]$ supervisorctl status
unix:///opt/supervisor/run/supervisor.sock refused connection
The effort to restart the WAE process and the server reboot are not successful.
Troubleshoot
In order to triage this further, capture these details:
- WAE Version
- OS version: cat /etc/redhat-release
- Supervisor version
: rpm -qa supervisor
- File /etc/security/limits.conf
Root Cause
This is a known issue observed in supervisor version 3.2 or lower. The WAE version 7.3 can have a lower supervisor version. For details, see Supervisor Issues.
Solution
The solution is to upgrade the WAE version and if that is not possible, then upgrade the supervisor. However, if an upgrade is not an option, use this workaround:
Comment these parameters in /etc/supervisord.conf and then restart supervisor/wae.
;minfds=1000000 ; (min. avail startup file descriptors;default 1024)
;minprocs=257805 ; (min. avail process descriptors;default 200)
systemctl restart supervisord.service
supervisorctl restart wae:*
Expected Outcome
WAE service starts as expected:
supervisorctl status
wae:kafka RUNNING pid 11726, uptime 0:01:06
wae:logrotate RUNNING pid 11648, uptime 0:01:07
wae:wae-monitor RUNNING pid 11667, uptime 0:01:07
wae:waectl RUNNING pid 11634, uptime 0:01:07
wae:zookeeper RUNNING pid 11633, uptime 0:01:07
Related Information