Icinga2 service refuse to start after reboot

Icinga 2
1

Fresh install of icinga2 on RHEL7 runs fine but after reboot it will not start or log. No debug log as well. May be something I missed but it just won’t do anything after the reboot. Thank you in advance.

icinga2 daemon -C (no errors all information)

icinga2 feature list >>
Disabled features: command compatlog elasticsearch gelf graphite icingadb influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker debuglog ido-mysql mainlog notification
[root@txslirmapa1v icinga2]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.12.3)

2

Have you tried to start icinga in debug mode?

3

Yes. I have. But all Icinga2 logging stopped the day I rebooted. See below. Maybe someone could show me how to start Icinga2 manually.

-rw-r----- 1 icinga icinga 0 Apr 2 03:46 debug.log
-rw-r----- 1 icinga icinga 139098 Apr 2 11:33 icinga2.log.1
-rw-r----- 1 icinga icinga 38324604 Apr 2 11:35 debug.log.1
[root@icinga2]# date
Sat Apr 3 16:32:37 CDT 2021
[root@icinga2]#

4

Found this error missing argument :

Process: 16945 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=exited, status=1/FAILURE)

Executing this command manually:

/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG}

[2021-04-03 16:46:02 -0500] critical/icinga-app: Error while parsing command-line options: the required argument for option ‘–errorlog’ is missing

5

No, debug mode means:

/usr/sbin/icinga2 daemon -x notice

6

Thanks. I ran the command but it stopped at the end.

[2021-04-04 23:23:09 -0500] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/include.conf
[2021-04-04 23:23:09 -0500] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/81df536d-55b7-4920-af3a-ca9d0009dc05/include.conf
[2021-04-04 23:23:09 -0500] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/81df536d-55b7-4920-af3a-ca9d0009dc05/…/active.conf
[2021-04-04 23:23:09 -0500] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/81df536d-55b7-4920-af3a-ca9d0009dc05/conf.d/downtimes/txsl.nss.removed.com!load!c2b74896-80a6-43ff-a2ad-46dfae401d8d.conf
[2021-04-04 23:23:09 -0500] information/ConfigItem: Committing config item(s).
[2021-04-04 23:23:09 -0500] notice/WorkQueue: Spawning WorkQueue threads for ‘DaemonUtility::LoadConfigFiles’
[2021-04-04 23:23:09 -0500] notice/Library: Loading library ‘libmysql_shim.so.2.12.3’
[2021-04-04 23:23:09 -0500] notice/ApiListener: Updating cache: Config package ‘_api’ has active stage ‘81df536d-55b7-4920-af3a-ca9d0009dc05’.
[2021-04-04 23:23:09 -0500] information/ApiListener: My API identity: txsl.nss.removed.com
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 Host.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 Downtime.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 2 NotificationCommands.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 2 FileLoggers.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 12 Notifications.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 2 HostGroups.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 3 Zones.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 Endpoint.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 ApiUser.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 ApiListener.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 235 CheckCommands.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 3 TimePeriods.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 UserGroup.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 User.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 11 Services.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 3 ServiceGroups.
[2021-04-04 23:23:09 -0500] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2021-04-04 23:23:09 -0500] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2021-04-04 23:23:09 -0500] notice/WorkQueue: Stopped WorkQueue threads for ‘DaemonUtility::LoadConfigFiles’
[2021-04-04 23:23:09 -0500] notice/cli: Notifying umbrella process (PID 6605) about the config loading success
[2021-04-04 23:23:09 -0500] notice/cli: Waiting for the umbrella process to let us doing the actual work
[2021-04-04 23:23:09 -0500] notice/cli: Worker process successfully loaded its config
[2021-04-04 23:23:09 -0500] notice/cli: The umbrella process let us continuing
[2021-04-04 23:23:09 -0500] information/ConfigObject: Restoring program state from file ‘/var/lib/icinga2/icinga2.state’
[2021-04-04 23:23:09 -0500] notice/WorkQueue: Spawning WorkQueue threads for ‘ConfigObject::RestoreObjects’
[2021-04-04 23:23:09 -0500] information/ConfigObject: Restored 286 objects. Loaded 0 new objects without state.
[2021-04-04 23:23:09 -0500] notice/WorkQueue: Stopped WorkQueue threads for ‘ConfigObject::RestoreObjects’
[2021-04-04 23:23:09 -0500] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/modified-attributes.conf
[2021-04-04 23:23:09 -0500] information/ConfigItem: Triggering Start signal for config items
[2021-04-04 23:23:09 -0500] notice/WorkQueue: Stopped WorkQueue threads for ‘DaemonCommand::Run’
[2021-04-04 23:23:10 -0500] notice/cli: Seemless worker (PID 6661) stopped, stopping as well

7

I’d delete /var/lib/icinga2/icinga2.state and give it another try.

8

Thank you. Deleted file. Daemon did not start as well. All icinga logs still nothing since Apr 2nd the day I rebooted.

[2021-04-05 09:41:49 -0500] information/ConfigItem: Instantiated 11 Services.
[2021-04-05 09:41:49 -0500] information/ConfigItem: Instantiated 3 ServiceGroups.
[2021-04-05 09:41:49 -0500] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2021-04-05 09:41:49 -0500] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2021-04-05 09:41:49 -0500] notice/WorkQueue: Stopped WorkQueue threads for ‘DaemonUtility::LoadConfigFiles’
[2021-04-05 09:41:49 -0500] notice/cli: Notifying umbrella process (PID 24088) about the config loading success
[2021-04-05 09:41:49 -0500] notice/cli: Waiting for the umbrella process to let us doing the actual work
[2021-04-05 09:41:49 -0500] notice/cli: Worker process successfully loaded its config
[2021-04-05 09:41:49 -0500] notice/cli: The umbrella process let us continuing
[2021-04-05 09:41:49 -0500] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/modified-attributes.conf
[2021-04-05 09:41:49 -0500] information/ConfigItem: Triggering Start signal for config items
[2021-04-05 09:41:49 -0500] notice/WorkQueue: Stopped WorkQueue threads for ‘DaemonCommand::Run’
[2021-04-05 09:41:49 -0500] notice/cli: Seemless worker (PID 24144) stopped, stopping as well

9

I’m not familiar with RHEL7 and its AppArmor. Your issue might be related, can you check the logs.

10

Thanks for your help. We don’t have AppArmor or SELinux enabled. Will continue on with a new VM to installed as this one has Perl installed for another application.