Error after updateting to Icinga 2.11 in FreeBSD

Hi,

We update our FreeBSD machines from 11.2 to 11.3. In that update Icinga2 was update to 2.11 and we run in to the problem related of to the inlcusion of config files that is documented here.

This is the output of the original error.

icinga2 daemon -C
[2019-11-12 16:50:45 +0000] information/cli: Icinga application loader (version: r2.11.0-1)
[2019-11-12 16:50:45 +0000] information/cli: Loading configuration file(s).
[2019-11-12 16:50:45 +0000] warning/config: Ignoring directory ‘/usr/local/etc/icinga2/zones.d/master’ for unknown zone ‘master’.
[2019-11-12 16:50:45 +0000] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/director-global’ for unknown zone ‘director-global’.
[2019-11-12 16:50:45 +0000] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/master’ for unknown zone ‘master’.
[2019-11-12 16:50:45 +0000] information/ConfigItem: Committing config item(s).
[2019-11-12 16:50:45 +0000] information/ApiListener: My API identity: icinga.foo.bar
[2019-11-12 16:50:45 +0000] critical/config: Error: Endpoint object for ‘icinga.foo.bar’ is missing.
Location: in /usr/local/etc/icinga2/features-enabled/api.conf: 4:1-4:24
/usr/local/etc/icinga2/features-enabled/api.conf(2): * The API listener is used for distributed monitoring setups.
/usr/local/etc/icinga2/features-enabled/api.conf(3): */
/usr/local/etc/icinga2/features-enabled/api.conf(4): object ApiListener “api” {
^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/etc/icinga2/features-enabled/api.conf(5):
/usr/local/etc/icinga2/features-enabled/api.conf(6): ticket_salt = TicketSalt

[2019-11-12 16:50:45 +0000] critical/config: 1 error
[2019-11-12 16:50:45 +0000] critical/cli: Config validation failed. Re-run with ‘icinga2 daemon -C’ after fixing the config

Then I edited the file zones.conf (that is included in icinga2.conf) and added:

include_recursive “zones.d/master/”

In that directory I have the configuration files for every host in the form of host1.foo.bar, host2.foo.bar, … hostn.foo.bar

After that I got the following error:

 icinga2 daemon -C
[2019-11-12 16:46:50 +0000] information/cli: Icinga application loader (version: r2.11.0-1)
[2019-11-12 16:46:50 +0000] information/cli: Loading configuration file(s).
[2019-11-12 16:46:50 +0000] critical/config: Error: Object 'host1.foo.bar' of type 'Endpoint' re-defined: in /usr/local/etc/icinga2/zones.d/master/host1.foo.bar.conf: 1:0-1:45; previous definition: in /usr/local/etc/icinga2/zones.d/master//host1.foo.bar.conf: 1:0-1:45
Location: in /usr/local/etc/icinga2/zones.d/master/host1.foo.bar.conf: 1:0-1:45
/usr/local/etc/icinga2/zones.d/master/host1.foo.bar.conf(1): object Endpoint "host1.foo.bar" {
                                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/etc/icinga2/zones.d/master/host1.foo.bar.conf(2):     host = "185.108.76.92"
/usr/local/etc/icinga2/zones.d/master/host1.foo.bar.conf(3): }


[2019-11-12 16:46:50 +0000] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.

From that error log I understund that it is reading two times the same file. I renamed host1.foo.bar.conf to host1.foo.bar.conf-back to see if the error changes. Then I got the same error than before but for host2.foo.bar.

After that I try including manually each .conf file instead of doing it recursive:

include "zones.d/master/host1.foo.bar
include "zones.d/master/host2.foo.bar

include "zones.d/master/hostn.foo.bar

I got the same error. I don’t understand how am I duplicating the configuration.

Thanks a lot.

Hi,

remove the following lines from your icinga2.conf file:

include_recursive "zones.d/master/"
include "zones.d/master/host1.foo.bar"
include "zones.d/master/host2.foo.bar"
…
include "zones.d/master/hostn.foo.bar"

It is not necessary to include the directories below the zones.d folder. Directories with and associated Zone object are included automatically. If you now include the directory zones.d manually it will be loaded twice by the config renderer.

Please show us your zones.conf file.

All zone and endpoint objects need to be configured outside the zones.d directory in the zones.conf file.

Best regards
Michael

2 Likes

Thanks @mcktr

It is not necessary to include the directories below the `zones.d` folder. Directories with and associated Zone object are included automatically. If you now include the directory `zones.d` manually it will be loaded twice by the config renderer.

That use to be the case until version 2.11 of Icinga, that is why my system used to work and now it does not work.

BTW, I include

include_recursive "zones.d/master/"

or

include "zones.d/master/host1.foo.bar" 
include "zones.d/master/host2.foo.bar" 
… 
include "zones.d/master/hostn.foo.bar"

Not both.

If I don’t include either I get the first error message posted on the first post.

Thanks

Icinga 2.11 got stricter with configuration that should have never worked but was not blocked by older versions. This is why this might have worked in the past.

Including something below zones.d specifically was never intended to work because it’s hardcoded to work on the whole directory.

You need to place a zone object in your zones.conf file with the name you are using for your directory. e.g. for your directory called master you need a Zone object in zones.conf that’s called master as well. This way Icinga knows it has to include the directory zones.d/master.

2 Likes

Tanks @twidhalm, this is what I have at the moment in zones.conf

object Zone "global-templates" {
        global = true
}

include_recursive "zones.d/master/"

Inside master I have .conf files similar to:

object Endpoint “h1.foo.bar” {
host = “xxx.xxx.xxx.xxx”
}

object Host “h1.foo.bar” {
import “generic-host”

address = "xxx.xxx.xxx.xxx"

vars.os = "FreeBSD"
vars.ssh_port = 2326

}

object Zone “h1.foo.bar” {
endpoints = [ “h1.foo.bar”, ]
parent = “master”
}

I would do some testing from what I read in your comment.

Just for the record, I am pretty new with Icinga2 and I am administrating what someone else implemented before. I don’t have contact with that someone else at the moment.

That said I am trying to understand how things work.

I see that under zones.d/master/ I have a file with a name like monitoring.foo.bar. At the end of the file I have:

object Zone "master" {
 	endpoints = [ "monitoring.foo.bar", ]
 }

On the configuration file of the other hosts they refer to master as parent.

 object Zone "host1.foo.bar" {
     endpoints = [ "host1.foo.bar", ]
     parent = "master"
 }

So my understanding is that the master is monitoring.foo.bar and the rest connect to the parent.


Should I do something like?

object Zone "master" { 
  endpoints = [ "monitoring.foo.bar", "host1.foo.bar",.... "hostn.foo.bar"]

Best

I think I understand your recommendation now. I moved the configuration for the master from zones.d/master/monitoring.foo.bar to zones.conf. So now I have in zones.d:

object Zone "global-templates" {
	global = true
}

object Endpoint "monitoring.foo.bar" {
	host = "xxx.xxx.xxx.xxx"
}

object Host "monitoring.foo.bar"  {
    import "generic-host"

    address = "xxx.xxx.xxx.xxx"
   
    vars.os = "FreeBSD"
    vars.ssh_port = 2376
}

object Zone "master" {
	endpoints = [ "monitoring.foo.bar", ]
}

The error messages have changed now:

[2019-11-14 14:36:32 +0000] information/cli: Loading configuration file(s).
[2019-11-14 14:36:32 +0000] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/director-global' for unknown zone 'director-global'.
[2019-11-14 14:36:32 +0000] information/ConfigItem: Committing config item(s).
[2019-11-14 14:36:32 +0000] information/ApiListener: My API identity: monitoring.foo.bar
[2019-11-14 14:36:32 +0000] critical/config: Error: Validation failed for object 'monitoring.foo.bar!load' of type 'Service'; Attribute 'command_endpoint': Command endpoint must not be set.
Location: in /usr/local/etc/icinga2/conf.d/services/load.conf: 1:0-1:19
/usr/local/etc/icinga2/conf.d/services/load.conf(1): apply Service "load" {
                                                     ^^^^^^^^^^^^^^^^^^^^
/usr/local/etc/icinga2/conf.d/services/load.conf(2):   import "generic-service"
/usr/local/etc/icinga2/conf.d/services/load.conf(3): 


[2019-11-14 14:36:32 +0000] critical/config: Error: Validation failed for object 'monitoring.foo.bar!ssh' of type 'Service'; Attribute 'command_endpoint': Command endpoint must not be set.
Location: in /usr/local/etc/icinga2/conf.d/services/ssh.conf: 1:0-1:18
/usr/local/etc/icinga2/conf.d/services/ssh.conf(1): apply Service "ssh" {
                                                    ^^^^^^^^^^^^^^^^^^^
/usr/local/etc/icinga2/conf.d/services/ssh.conf(2):   import "generic-service"
/usr/local/etc/icinga2/conf.d/services/ssh.conf(3): 

This warnings related to directory inclusion are gone:

[2019-11-12 16:50:45 +0000] warning/config: Ignoring directory ‘/usr/local/etc/icinga2/zones.d/master’ for unknown zone ‘master’.
[2019-11-12 16:50:45 +0000] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/master’ for 

This warning related inclusion still persists:

[2019-11-12 16:50:45 +0000] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/director-global’ for unknown zone ‘director-global’.

I would research this issue.

Hi,

Yes, moving the definition of the Zone object to zones.conf was what I meant.

I’m still wondering, why you get the info, that the directories are ignored. Could you use icinga2 object list --type zone to verify that your core knows about the master zone?

The errormessage aber command_endpoint not to be set is new to me, honestly. I can just imagine that it comes from a check on a single node master. Please use ignore where to remove the check from the master and try to revalidate the configuration with icinga2 daemon -C. You can re-add it later with a slighty changed configuration - removing is just for debugging.

I get an output full of green and blue colors, that seems to say that everything is ok there. Here an example of one of the many definition shown:

Object 'host1.foo.bar.foo.bar' of type 'Zone':
  % declared in '/usr/local/etc/icinga2/zones.d/master/host1.foo.bar.foo.bar.conf', lines 14:1-14:35
  * __name = "host1.foo.bar.foo.bar"
  * endpoints = [ "host1.foo.bar.foo.bar" ]
    % = modified in '/usr/local/etc/icinga2/zones.d/master/host1.foo.bar.foo.bar.conf', lines 15:5-15:44
  * global = false
  * name = "host1.foo.bar.foo.bar"
  * package = "_etc"
  * parent = "master"
    % = modified in '/usr/local/etc/icinga2/zones.d/master/host1.foo.bar.foo.bar.conf', lines 16:5-16:21
  * source_location
    * first_column = 1
    * first_line = 14
    * last_column = 35
    * last_line = 14
    * path = "/usr/local/etc/icinga2/zones.d/master/host1.foo.bar.foo.bar.conf"
  * templates = [ "host1.foo.bar.foo.bar" ]
    % = modified in '/usr/local/etc/icinga2/zones.d/master/host1.foo.bar.foo.bar.conf', lines 14:1-14:35
  * type = "Zone"
  * zone = "master"

So it looks like the configuration in zones.d/master has been imported correctly as it seems to show the configuration or every zone.

I do not understand how to do the ignore where thing however I comment out the Object Host "monitoring.foo.bar" the following lines from zones.conf

object Zone "global-templates" {
	global = true
}

object Endpoint "monitoring.foo.bar" {
	host = "xxx.xxx.xxx.xxx"
}

#object Host "monitoring.foo.bar"  {
#    import "generic-host"
#   address = "xxx.xxx.xxx.xxx"   
#    vars.os = "FreeBSD"
#   vars.ssh_port = 2376
#}

object Zone "master" {
	endpoints = [ "monitoring.foo.bar", ]
}

Now the monitoring is working. There are some issues I would have to check, but is working !!!

I am still having this warnings:

[2019-11-15 14:01:13 +0000] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/director-global' for unknown zone 'director-global'.
[2019-11-15 14:01:13 +0000] warning/ApplyRule: Apply rule 'backup-downtime' (in /usr/local/etc/icinga2/conf.d/downtimes/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere!
[2019-11-15 14:01:13 +0000] warning/ApplyRule: Apply rule 'HTTP: a.d' (in /usr/local/etc/icinga2/conf.d/services/http_a.d.conf: 1:0-1:24) for type 'Service' does not match anywhere!
1 Like

The last warning messages are no problem. They just show you that Icinga found uncommon configuration which might be nonetheless what you want.

The first says, that you get synced a zone name director-global (it’s a new default in case someone uses Director) which is ignored because you have to explicitely add it to your configuration.

In zones.conf:

object Zone "director-global" {
  global = true
}

The second two warnings mean that you have apply rules that never match. This is ok it’s just a warning if it’s not what you want.