Icinga daemon does not start on macOS Big Sur

MichalMMac · December 18, 2020, 4:27pm

I know this is an odd one.
I am using Icinga 2 on some of my macOS computers in the role of Icinga client.
Icinga 2 is built from source with dependencies provided by MacPorts.

So far I was able to make it work. I have it running on macOS 10.15.7 Catalina.

Today I compiled Icinga 2 on macOS 11.1 Big Sur. Compilation went fine.
I am able to run the binary:

/opt/local/sbin/icinga2 --version                     17:08:31
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.3-1)

Copyright (c) 2012-2020 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: macOS
  Platform version: 11.1
  Kernel: Darwin
  Kernel version: 20.2.0
  Architecture: x86_64

Build information:
  Compiler: Clang 11.0.3.11030032
  Build host: our.company.server
  OpenSSL version: OpenSSL 1.1.1i  8 Dec 2020

Application information:

General paths:
  Config directory: /opt/local/etc/icinga2
  Data directory: /opt/local/var/lib/icinga2
  Log directory: /opt/local/var/log/icinga2
  Cache directory: /opt/local/var/cache/icinga2
  Spool directory: /opt/local/var/spool/icinga2
  Run directory: /opt/local/var/run/icinga2

Old paths (deprecated):
  Installation root: /opt/local
  Sysconf directory: /opt/local/etc
  Run directory (base): /opt/local/var/run
  Local state directory: /opt/local/var

Internal paths:
  Package data directory: /opt/local/share/icinga2
  State path: /opt/local/var/lib/icinga2/icinga2.state
  Modified attributes path: /opt/local/var/lib/icinga2/modified-attributes.conf
  Objects path: /opt/local/var/cache/icinga2/icinga2.debug
  Vars path: /opt/local/var/cache/icinga2/icinga2.vars
  PID path: /opt/local/var/run/icinga2/icinga2.pid

However when I try to run it as daemon it exits with code 1.

/opt/local/sbin/icinga2 daemon --log-level debug
[2020-12-18 17:08:13 +0100] information/cli: Icinga application loader (version: r2.12.3-1)
[2020-12-18 17:08:13 +0100] notice/cli: Spawning seemless worker process doing the actual work
[2020-12-18 17:08:13 +0100] notice/cli: Spawned worker process (PID 784), waiting for it to load its config
[2020-12-18 17:08:13 +0100] information/cli: Loading configuration file(s).
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/etc/icinga2/icinga2.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/etc/icinga2/constants.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/etc/icinga2/zones.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/itl
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/command-icinga.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/command-plugins.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/big-data.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/databases.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/hardware.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/icingacli.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/ipmi.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/logmanagement.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/metrics.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/network-components.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/network-services.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/operating-system.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/raid-controller.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/smart-attributes.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/storage.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/virtualization.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/vmware.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/plugins-contrib.d/web.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/manubulon
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/command-plugins-manubulon.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/windows-plugins
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/command-plugins-windows.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/nscp
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/share/icinga2/include/command-nscp-local.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/etc/icinga2/features-enabled/api.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/etc/icinga2/features-enabled/mainlog.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/var/lib/icinga2/api/packages/_api/include.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/var/lib/icinga2/api/packages/_api/ef750578-3108-4107-8a5f-80bc6ec20779/include.conf
[2020-12-18 17:08:13 +0100] notice/ConfigCompiler: Compiling config file: /opt/local/var/lib/icinga2/api/packages/_api/ef750578-3108-4107-8a5f-80bc6ec20779/../active.conf
[2020-12-18 17:08:13 +0100] information/ConfigItem: Committing config item(s).
[2020-12-18 17:08:13 +0100] debug/Timer: TimerThreadProc started.
[2020-12-18 17:08:13 +0100] notice/WorkQueue: Spawning WorkQueue threads for 'DaemonUtility::LoadConfigFiles'
[2020-12-18 17:08:13 +0100] notice/ApiListener: Updating cache: Config package '_api' has active stage 'ef750578-3108-4107-8a5f-80bc6ec20779'.
[2020-12-18 17:08:13 +0100] information/ApiListener: My API identity: our.company.server
[2020-12-18 17:08:13 +0100] notice/TlsUtility: Available TLS cipher list: TLS_AES_256_GCM_SHA384 TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-CHACHA20-POLY1305 ECDHE-RSA-CHACHA20-POLY1305 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES256-SHA384 ECDHE-RSA-AES256-SHA384 ECDHE-ECDSA-AES128-SHA256 ECDHE-RSA-AES128-SHA256 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 AES256-GCM-SHA384 AES128-GCM-SHA256
[2020-12-18 17:08:13 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2020-12-18 17:08:13 +0100] information/ConfigItem: Instantiated 235 CheckCommands.
[2020-12-18 17:08:13 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2020-12-18 17:08:13 +0100] information/ConfigItem: Instantiated 2 Endpoints.
[2020-12-18 17:08:13 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-12-18 17:08:13 +0100] information/ConfigItem: Instantiated 6 Zones.
[2020-12-18 17:08:13 +0100] information/ScriptGlobal: Dumping variables to file '/opt/local/var/cache/icinga2/icinga2.vars'
[2020-12-18 17:08:13 +0100] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonUtility::LoadConfigFiles'
[2020-12-18 17:08:13 +0100] notice/cli: Notifying umbrella process (PID 771) about the config loading success
[2020-12-18 17:08:13 +0100] notice/cli: Worker process successfully loaded its config
[2020-12-18 17:08:13 +0100] notice/cli: Waiting for the umbrella process to let us doing the actual work
[2020-12-18 17:08:13 +0100] notice/cli: The umbrella process let us continuing
[2020-12-18 17:08:13 +0100] information/ConfigItem: Triggering Start signal for config items
[2020-12-18 17:08:13 +0100] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonCommand::Run'
[2020-12-18 17:08:13 +0100] notice/cli: Seemless worker (PID 784) stopped, stopping as well

I don’t see anything related to exiting suddenly in the debug output. What could I do to troubleshoot more and find out why is icinga2 daemon exiting with code 1?

MichalMMac · December 18, 2020, 5:51pm

When I started icinga2 daemon with --no-config it did not exit. I did a little bit more digging.

Turn out the problem is in the mainlog.conf configuration.

object FileLogger "main-log" {
  severity = "information"
  path = LogDir + "/icinga2.log"
}

mainlog.conf config itself is valid. When I remove it, icinga2 daemon starts without abrupt exit. I think I can live without this feature so problem “solved”.

However I am not sure what is wrong here. LogDir is /opt/local/var/log/icinga2 which does exist and root have no problem to write file there even on macOS with SIP.

dokon · December 21, 2020, 10:52am

Hi !

This might be true to the root user but icinga should use its own user account which is icinga. Also beside that SIP under Big Sur prevents for security reasons almost all non system accounts in writing in system directories like /var/log.

This might not be a “cool” or secure suggestion but could you disable the SIP and also take a look into the “security & privacy” tab of the system preferences if there shows up an allowance question ?

Regards

David

Screen Shot 2020-12-21 at 11.51.10 AM

MichalMMac · December 21, 2020, 8:55pm

I’ll disable SIP for sake of testing and report back.

However

Nothing Icinga related to allow in Security & Privacy. I can try to give icinga2 binary full disc access to test filesystem access related issue.
There is no difference in described behavior launching icinga daemon under root or dedicated account.
icinga2 daemon runs without this problem on the Catalina which has same SIP/ read only root volume restrictions.
/var/log is not protected by sip.
Icinga has been compiled to use /opt/local as it’s base directory. This should make it stay out of the way of SIP protected directories.