Icinga2 not starting on monitored machines

[pierreact]

Hello,

I’m looking for tips and ideas on how to understand/troubleshoot an issue on my new icinga setup.
I did some research but haven’t found results.
I grep’ed to try and spot a redefinition but I can’t see it.

When I restart icinga2 on monitored servers, restart fails with the following message:

On server running the agent:

    Feb 14 17:59:57 registry.int.mydomain.tld icinga2[11724]: [2021-02-14 17:59:57 +0100] information/cli: Icinga application loader (version: r2.8.1-1)
    Feb 14 17:59:57 registry.int.mydomain.tld icinga2[11724]: [2021-02-14 17:59:57 +0100] information/cli: Loading configuration file(s).
    Feb 14 17:59:57 registry.int.mydomain.tld icinga2[11724]: [2021-02-14 17:59:57 +0100] critical/config: Error: Object 'registry.int.mydomain.tld' of type 'Zone' re-defined: in /var/lib/icinga2/api/zones/registry.int.mydomain.tld/_et
    Feb 14 17:59:57 registry.int.mydomain.tld icinga2[11724]: Location: in /var/lib/icinga2/api/zones/registry.int.mydomain.tld/_etc/registry.int.mydomain.tld.conf: 1:0-1:40
    Feb 14 17:59:57 registry.int.mydomain.tld icinga2[11724]: /var/lib/icinga2/api/zones/registry.int.mydomain.tld/_etc/registry.int.mydomain.tld.conf(1): object Zone "registry.int.mydomain.tld" {
    Feb 14 17:59:57 registry.int.mydomain.tld icinga2[11724]:                                                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    Feb 14 17:59:57 registry.int.mydomain.tld icinga2[11724]: /var/lib/icinga2/api/zones/registry.int.mydomain.tld/_etc/registry.int.mydomain.tld.conf(2):   endpoints = [ "registry.int.mydomain.tld" ]
    Feb 14 17:59:57 registry.int.mydomain.tld icinga2[11724]: /var/lib/icinga2/api/zones/registry.int.mydomain.tld/_etc/registry.int.mydomain.tld.conf(3):   parent = "master.mon.int.mydomain.tld"
    Feb 14 17:59:57 registry.int.mydomain.tld systemd[1]: icinga2.service: Main process exited, code=exited, status=1/FAILURE
    Feb 14 17:59:57 registry.int.mydomain.tld systemd[1]: icinga2.service: Failed with result 'exit-code'.

On master:

    # cat /etc/icinga2/zones.d/registry.int.mydomain.tld/registry.int.mydomain.tld.conf
    object Zone "registry.int.mydomain.tld" {
      endpoints = [ "registry.int.mydomain.tld" ]
      parent = "master.mon.int.mydomain.tld"
    }

    object Endpoint "registry.int.mydomain.tld" {
      host = "95.216.143.73"
    }

    object Host "registry.int.mydomain.tld" {
      import "generic-host"
      address = "95.216.143.73"
      vars.http_vhosts["http"] = {
        http_uri = "/"
      }
      vars.notification["mail"] = {
        groups = [ "icingaadmins" ]
      }
      vars.client_endpoint = name
    }

# icinga2 object list --type Zone
    Object 'master.mon.int.mydomain.tld' of type 'Zone':
      % declared in '/etc/icinga2/zones.conf', lines 11:1-11:20
      * __name = "master.mon.int.mydomain.tld"
      * endpoints = [ "master.mon.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.conf', lines 12:3-12:26
      * global = false
      * name = "master.mon.int.mydomain.tld"
      * package = "_etc"
      * parent = ""
      * source_location
        * first_column = 1
        * first_line = 11
        * last_column = 20
        * last_line = 11
        * path = "/etc/icinga2/zones.conf"
      * templates = [ "master.mon.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.conf', lines 11:1-11:20
      * type = "Zone"
      * zone = ""

    Object 'global-templates' of type 'Zone':
      % declared in '/etc/icinga2/zones.conf', lines 24:1-24:30
      * __name = "global-templates"
      * endpoints = null
      * global = true
        % = modified in '/etc/icinga2/zones.conf', lines 25:3-25:15
      * name = "global-templates"
      * package = "_etc"
      * parent = ""
      * source_location
        * first_column = 1
        * first_line = 24
        * last_column = 30
        * last_line = 24
        * path = "/etc/icinga2/zones.conf"
      * templates = [ "global-templates" ]
        % = modified in '/etc/icinga2/zones.conf', lines 24:1-24:30
      * type = "Zone"
      * zone = ""

    Object 'node-1.prod-1.int.mydomain.tld' of type 'Zone':
      % declared in '/etc/icinga2/zones.d/node-1.prod-1.int.mydomain.tld/node-1.prod-1.int.mydomain.tld.conf', lines 1:0-1:45
      * __name = "node-1.prod-1.int.mydomain.tld"
      * endpoints = [ "node-1.prod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-1.prod-1.int.mydomain.tld/node-1.prod-1.int.mydomain.tld.conf', lines 2:3-2:52
      * global = false
      * name = "node-1.prod-1.int.mydomain.tld"
      * package = "_etc"
      * parent = "master.mon.int.mydomain.tld"
        % = modified in '/etc/icinga2/zones.d/node-1.prod-1.int.mydomain.tld/node-1.prod-1.int.mydomain.tld.conf', lines 3:3-3:42
      * source_location
        * first_column = 0
        * first_line = 1
        * last_column = 45
        * last_line = 1
        * path = "/etc/icinga2/zones.d/node-1.prod-1.int.mydomain.tld/node-1.prod-1.int.mydomain.tld.conf"
      * templates = [ "node-1.prod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-1.prod-1.int.mydomain.tld/node-1.prod-1.int.mydomain.tld.conf', lines 1:0-1:45
      * type = "Zone"
      * zone = "node-1.prod-1.int.mydomain.tld"

    Object 'node-1.preprod-1.int.mydomain.tld' of type 'Zone':
      % declared in '/etc/icinga2/zones.d/node-1.preprod-1.int.mydomain.tld/node-1.preprod-1.int.mydomain.tld.conf', lines 1:0-1:48
      * __name = "node-1.preprod-1.int.mydomain.tld"
      * endpoints = [ "node-1.preprod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-1.preprod-1.int.mydomain.tld/node-1.preprod-1.int.mydomain.tld.conf', lines 2:3-2:55
      * global = false
      * name = "node-1.preprod-1.int.mydomain.tld"
      * package = "_etc"
      * parent = "master.mon.int.mydomain.tld"
        % = modified in '/etc/icinga2/zones.d/node-1.preprod-1.int.mydomain.tld/node-1.preprod-1.int.mydomain.tld.conf', lines 3:3-3:42
      * source_location
        * first_column = 0
        * first_line = 1
        * last_column = 48
        * last_line = 1
        * path = "/etc/icinga2/zones.d/node-1.preprod-1.int.mydomain.tld/node-1.preprod-1.int.mydomain.tld.conf"
      * templates = [ "node-1.preprod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-1.preprod-1.int.mydomain.tld/node-1.preprod-1.int.mydomain.tld.conf', lines 1:0-1:48
      * type = "Zone"
      * zone = "node-1.preprod-1.int.mydomain.tld"

    Object 'node-2.preprod-1.int.mydomain.tld' of type 'Zone':
      % declared in '/etc/icinga2/zones.d/node-2.preprod-1.int.mydomain.tld/node-2.preprod-1.int.mydomain.tld.conf', lines 1:0-1:48
      * __name = "node-2.preprod-1.int.mydomain.tld"
      * endpoints = [ "node-2.preprod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-2.preprod-1.int.mydomain.tld/node-2.preprod-1.int.mydomain.tld.conf', lines 2:3-2:55
      * global = false
      * name = "node-2.preprod-1.int.mydomain.tld"
      * package = "_etc"
      * parent = "master.mon.int.mydomain.tld"
        % = modified in '/etc/icinga2/zones.d/node-2.preprod-1.int.mydomain.tld/node-2.preprod-1.int.mydomain.tld.conf', lines 3:3-3:42
      * source_location
        * first_column = 0
        * first_line = 1
        * last_column = 48
        * last_line = 1
        * path = "/etc/icinga2/zones.d/node-2.preprod-1.int.mydomain.tld/node-2.preprod-1.int.mydomain.tld.conf"
      * templates = [ "node-2.preprod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-2.preprod-1.int.mydomain.tld/node-2.preprod-1.int.mydomain.tld.conf', lines 1:0-1:48
      * type = "Zone"
      * zone = "node-2.preprod-1.int.mydomain.tld"

    Object 'node-3.preprod-1.int.mydomain.tld' of type 'Zone':
      % declared in '/etc/icinga2/zones.d/node-3.preprod-1.int.mydomain.tld/node-3.preprod-1.int.mydomain.tld.conf', lines 1:0-1:48
      * __name = "node-3.preprod-1.int.mydomain.tld"
      * endpoints = [ "node-3.preprod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-3.preprod-1.int.mydomain.tld/node-3.preprod-1.int.mydomain.tld.conf', lines 2:3-2:55
      * global = false
      * name = "node-3.preprod-1.int.mydomain.tld"
      * package = "_etc"
      * parent = "master.mon.int.mydomain.tld"
        % = modified in '/etc/icinga2/zones.d/node-3.preprod-1.int.mydomain.tld/node-3.preprod-1.int.mydomain.tld.conf', lines 3:3-3:42
      * source_location
        * first_column = 0
        * first_line = 1
        * last_column = 48
        * last_line = 1
        * path = "/etc/icinga2/zones.d/node-3.preprod-1.int.mydomain.tld/node-3.preprod-1.int.mydomain.tld.conf"
      * templates = [ "node-3.preprod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-3.preprod-1.int.mydomain.tld/node-3.preprod-1.int.mydomain.tld.conf', lines 1:0-1:48
      * type = "Zone"
      * zone = "node-3.preprod-1.int.mydomain.tld"

    Object 'node-2.prod-1.int.mydomain.tld' of type 'Zone':
      % declared in '/etc/icinga2/zones.d/node-2.prod-1.int.mydomain.tld/node-2.prod-1.int.mydomain.tld.conf', lines 1:0-1:45
      * __name = "node-2.prod-1.int.mydomain.tld"
      * endpoints = [ "node-2.prod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-2.prod-1.int.mydomain.tld/node-2.prod-1.int.mydomain.tld.conf', lines 2:3-2:52
      * global = false
      * name = "node-2.prod-1.int.mydomain.tld"
      * package = "_etc"
      * parent = "master.mon.int.mydomain.tld"
        % = modified in '/etc/icinga2/zones.d/node-2.prod-1.int.mydomain.tld/node-2.prod-1.int.mydomain.tld.conf', lines 3:3-3:42
      * source_location
        * first_column = 0
        * first_line = 1
        * last_column = 45
        * last_line = 1
        * path = "/etc/icinga2/zones.d/node-2.prod-1.int.mydomain.tld/node-2.prod-1.int.mydomain.tld.conf"
      * templates = [ "node-2.prod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-2.prod-1.int.mydomain.tld/node-2.prod-1.int.mydomain.tld.conf', lines 1:0-1:45
      * type = "Zone"
      * zone = "node-2.prod-1.int.mydomain.tld"

    Object 'node-3.prod-1.int.mydomain.tld' of type 'Zone':
      % declared in '/etc/icinga2/zones.d/node-3.prod-1.int.mydomain.tld/node-3.prod-1.int.mydomain.tld.conf', lines 1:0-1:45
      * __name = "node-3.prod-1.int.mydomain.tld"
      * endpoints = [ "node-3.prod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-3.prod-1.int.mydomain.tld/node-3.prod-1.int.mydomain.tld.conf', lines 2:3-2:52
      * global = false
      * name = "node-3.prod-1.int.mydomain.tld"
      * package = "_etc"
      * parent = "master.mon.int.mydomain.tld"
        % = modified in '/etc/icinga2/zones.d/node-3.prod-1.int.mydomain.tld/node-3.prod-1.int.mydomain.tld.conf', lines 3:3-3:42
      * source_location
        * first_column = 0
        * first_line = 1
        * last_column = 45
        * last_line = 1
        * path = "/etc/icinga2/zones.d/node-3.prod-1.int.mydomain.tld/node-3.prod-1.int.mydomain.tld.conf"
      * templates = [ "node-3.prod-1.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/node-3.prod-1.int.mydomain.tld/node-3.prod-1.int.mydomain.tld.conf', lines 1:0-1:45
      * type = "Zone"
      * zone = "node-3.prod-1.int.mydomain.tld"

    Object 'registry.int.mydomain.tld' of type 'Zone':
      % declared in '/etc/icinga2/zones.d/registry.int.mydomain.tld/registry.int.mydomain.tld.conf', lines 1:0-1:40
      * __name = "registry.int.mydomain.tld"
      * endpoints = [ "registry.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/registry.int.mydomain.tld/registry.int.mydomain.tld.conf', lines 2:3-2:47
      * global = false
      * name = "registry.int.mydomain.tld"
      * package = "_etc"
      * parent = "master.mon.int.mydomain.tld"
        % = modified in '/etc/icinga2/zones.d/registry.int.mydomain.tld/registry.int.mydomain.tld.conf', lines 3:3-3:42
      * source_location
        * first_column = 0
        * first_line = 1
        * last_column = 40
        * last_line = 1
        * path = "/etc/icinga2/zones.d/registry.int.mydomain.tld/registry.int.mydomain.tld.conf"
      * templates = [ "registry.int.mydomain.tld" ]
        % = modified in '/etc/icinga2/zones.d/registry.int.mydomain.tld/registry.int.mydomain.tld.conf', lines 1:0-1:40
      * type = "Zone"
      * zone = "registry.int.mydomain.tld"

    Object 'director-global' of type 'Zone':
      % declared in '/etc/icinga2/zones.conf', lines 36:1-36:29
      * __name = "director-global"
      * endpoints = null
      * global = true
        % = modified in '/etc/icinga2/zones.conf', lines 37:3-37:15
      * name = "director-global"
      * package = "_etc"
      * parent = ""
      * source_location
        * first_column = 1
        * first_line = 36
        * last_column = 29
        * last_line = 36
        * path = "/etc/icinga2/zones.conf"
      * templates = [ "director-global" ]
        % = modified in '/etc/icinga2/zones.conf', lines 36:1-36:29
      * type = "Zone"
      * zone = ""

On both nodes, icinga2 is the one coming with ubuntu.

    # icinga2 --version
    icinga2 - The Icinga 2 network monitoring daemon (version: r2.8.1-1)

    Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
    License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.

    Application information:
      Installation root: /usr
      Sysconf directory: /etc
      Run directory: /run
      Local state directory: /var
      Package data directory: /usr/share/icinga2
      State path: /var/lib/icinga2/icinga2.state
      Modified attributes path: /var/lib/icinga2/modified-attributes.conf
      Objects path: /var/cache/icinga2/icinga2.debug
      Vars path: /var/cache/icinga2/icinga2.vars
      PID path: /run/icinga2/icinga2.pid

    System information:
      Platform: Ubuntu
      Platform version: 18.04.5 LTS (Bionic Beaver)
      Kernel: Linux
      Kernel version: 4.15.0-135-generic
      Architecture: x86_64

    Build information:
      Compiler: GNU 7.3.0
      Build host: lcy01-amd64-027

On master:

    # icinga2 daemon -C
    information/cli: Icinga application loader (version: r2.8.1-1)
    information/cli: Loading configuration file(s).
    information/ConfigItem: Committing config item(s).
    information/ApiListener: My API identity: master.mon.int.mydomain.tld
    warning/ApplyRule: Apply rule 'backup-downtime' (in /etc/icinga2/conf.d/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere!
    warning/ApplyRule: Apply rule 'apt' (in /etc/icinga2/conf.d/apt.conf: 1:0-1:18) for type 'Service' does not match anywhere!
    information/ConfigItem: Instantiated 1 ApiListener.
    information/ConfigItem: Instantiated 10 Zones.
    information/ConfigItem: Instantiated 8 Endpoints.
    information/ConfigItem: Instantiated 1 ApiUser.
    information/ConfigItem: Instantiated 2 FileLoggers.
    information/ConfigItem: Instantiated 53 Notifications.
    information/ConfigItem: Instantiated 1 NotificationCommand.
    information/ConfigItem: Instantiated 210 CheckCommands.
    information/ConfigItem: Instantiated 5 HostGroups.
    information/ConfigItem: Instantiated 1 IcingaApplication.
    information/ConfigItem: Instantiated 11 Hosts.
    information/ConfigItem: Instantiated 1 UserGroup.
    information/ConfigItem: Instantiated 1 User.
    information/ConfigItem: Instantiated 4 TimePeriods.
    information/ConfigItem: Instantiated 42 Services.
    information/ConfigItem: Instantiated 3 ServiceGroups.
    information/ConfigItem: Instantiated 1 CheckerComponent.
    information/ConfigItem: Instantiated 1 ExternalCommandListener.
    information/ConfigItem: Instantiated 1 IdoMysqlConnection.
    information/ConfigItem: Instantiated 1 NotificationComponent.
    information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
    information/cli: Finished validating the configuration file(s).

Thank you
P.S. When creating a new issue, the template mentions Troubleshooting - Icinga 2. This leads to a 404.

[log1c]

Hi,

first of all: Your Icinga version is very outdated. It is over three years old.
So I suggest you update asap and use the official package repository for that.
https://icinga.com/docs/icinga-2/latest/doc/02-installation/#debianubunturaspbian-repositories

This is your problem:
Object 'registry.int.mydomain.tld' of type 'Zone' re-defined: in /var/lib/icinga2/api/zones/registry.int.mydomain.tld/_etc/registry.int.mydomain.tld.conf

Means that the configuration that the agent receives from the master is re-defining what the agent already knows, because it has the configuration for its own host/endpoint/zone object.

Did you do the setup on the agent with the node wizard?
https://icinga.com/docs/icinga-2/latest/doc/06-distributed-monitoring/#agentsatellite-setup-on-linux
Normally that should be enough for an agent and then add the host object with the exact same name as the endpoint, IIRC.

The troubleshooting link works for me.

Lastly: In future please use markdown to format your posts, especially when they contain config or logs:

Create topics and master Markdown formatting

Edit and write topics Discourse uses Markdown as markup language. If you are familiar with GitHub, you’re good to go. If not, please follow this guide. [48] While you type something into the editor, you’ll recognize many things: live preview as you type on the right side if someone else is replying, a short notice on top of the editor you can navigate to a different page, and still reply to the “old” topic sessions are stored, which means that a draft text can be started on your Macbook…

This way the posts can be read more fluently