Hello,
We have upgraded our vCenter appliance (VCSA) to 6.7U3 a few days ago and we noticed a gap of logs in our syslog server (kiwi) since then.
I did a bit of troubleshooting but Rsyslog (the syslog client running on VCSA) is completely new to me.
I use this command to restart Rsyslog:
systemctl restart rsyslog
Right after starting up Rsyslog, logs are being sent to our syslog server.
~10min later, no more logs are sent.
The vCenter log file in our syslog server stops getting updated.
I did a tcpdump in our vCenter and I see that the vCenter stops sending logs.
Using UDP or TCP doesn't fix the issue.
I looked for errors in various log files in the vCenter but can't find anything.
This is what /var/log/vmware/rsyslogd/rsyslogd-syslog.log looks like after restarting Rsyslog:
2019-09-11T11:53:12.812087+02:00 info rsyslogd [origin software="rsyslogd" swVersion="8.37.0" x-pid="21203" x-info="http://www.rsyslog.com"] exiting on signal 15.
2019-09-11T11:54:42.617065+02:00 warning rsyslogd environment variable TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try http://www.rsyslog.com/e/2442 ]
2019-09-11T11:54:42.617568+02:00 info rsyslogd imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
2019-09-11T11:54:42.618409+02:00 info rsyslogd [origin software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="http://www.rsyslog.com"] start
Rsyslog is still running based on this command
systemctl status rsyslog.service
● rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-09-11 11:54:42 CEST; 39min ago
Docs: man:rsyslogd(8)
Main PID: 22235 (rsyslogd)
Tasks: 12
Memory: 5.7M
CPU: 191ms
CGroup: /system.slice/rsyslog.service
└─22235 /usr/sbin/rsyslogd -n
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Main process exited, code=killed, status=9/KILL
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Stopped System Logging Service.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Unit entered failed state.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: rsyslog.service: Failed with result 'signal'.
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Starting System Logging Service...
Sep 11 11:54:42 vcenter.domain.local systemd[1]: Started System Logging Service.
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: environment variable TZ is not set, auto correcting this to TZ=/etc/localtime [v8.37.0 try http://www.rsyslog.com/e/2442 ]
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.37.0]
Sep 11 11:54:42 vcenter.domain.local rsyslogd[22235]: [origin software="rsyslogd" swVersion="8.37.0" x-pid="22235" x-info="http://www.rsyslog.com"] start
(real hostname has been replaced by vcenter.domain.local)
I created a ticket at VMware support, but the agent wasn't able to find any errors as well and she suggested to take a backup of our vCenter and reinstall with a restore to get a fresh install of Photon OS since Rsyslog is integrated in Photon OS. I'm not going to do that now, maybe as a last troubleshooting step.
In the meantime, do you guys have an idea? Wrong Rsyslog config?
Thx for your help.