Master, Satellites and Agents

[rsx (Roland Sommer)]

@MartinW

Should I just directly edit the config file and replace “master” with “satellite”?

The zone name doesn’t matter (and yes, that’s confusing). The ip address of the endpoint is crucial and it has to be the ip address of the satellite.

The host object has to be in the satellite zone and this is configured at the director. You could verfiy this with icinga2 object list -n wk2016.local.

[MartinW (Martin)]

Hi,

yeah… this is getting more and more confusing, so what icinga2 object list tells me is:

[root@icinga-master icinga2]# icinga2 object list -n wk2016.local
Object ‘wk2016.local’ of type ‘Zone’:
% declared in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_zones.conf’, lines 11:1-11:38

• __name = “wk2016.local”
• endpoints = [ “wk2016.local” ]
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_zones.conf’, lines 13:5-13:46
• global = false
• name = “wk2016.local”
• package = “director”
• parent = “icinga-master”
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_zones.conf’, lines 12:5-12:28
• source_location
  • first_column = 1
  • first_line = 11
  • last_column = 38
  • last_line = 11
  • path = “/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_zones.conf”
• templates = [ “wk2016.local” ]
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_zones.conf’, lines 11:1-11:38
• type = “Zone”
• zone = “icinga-master”

Object ‘wk2016.local’ of type ‘Endpoint’:
% declared in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_endpoints.conf’, lines 11:1-11:42

• __name = “wk2016.local”
• host = “192.168.10.58”
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_endpoints.conf’, lines 12:5-12:25
• log_duration = 0
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_endpoints.conf’, lines 13:5-13:21
• name = “wk2016.local”
• package = “director”
• port = “5665”
• source_location
  • first_column = 1
  • first_line = 11
  • last_column = 42
  • last_line = 11
  • path = “/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_endpoints.conf”
• templates = [ “wk2016.local” ]
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/agent_endpoints.conf’, lines 11:1-11:42
• type = “Endpoint”
• zone = “icinga-master”

Object ‘wk2016.local’ of type ‘Host’:
% declared in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/hosts.conf’, lines 24:1-24:38

• __name = “wk2016.local”
• action_url = “”
• address = “192.168.10.58”
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/hosts.conf’, lines 28:5-28:28
• address6 = “”
• check_command = “hostalive”
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/director-global/host_templates.conf’, lines 6:5-6:31
• check_interval = 300
• check_period = “”
• check_timeout = null
• command_endpoint = “”
• display_name = “wk2016”
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/hosts.conf’, lines 27:5-27:34
• enable_active_checks = true
• enable_event_handler = true
• enable_flapping = false
• enable_notifications = true
• enable_passive_checks = true
• enable_perfdata = true
• event_command = “”
• flapping_threshold = 0
• flapping_threshold_high = 30
• flapping_threshold_low = 25
• groups = [ “CUST1-Group” ]
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/hosts.conf’, lines 29:5-29:27
• icon_image = “”
• icon_image_alt = “”
• max_check_attempts = 3
• name = “wk2016.local”
• notes = “”
• notes_url = “”
• package = “director”
• retry_interval = 60
• source_location
  • first_column = 1
  • first_line = 24
  • last_column = 38
  • last_line = 24
  • path = “/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/hosts.conf”
• templates = [ “wk2016.local”, “Windows-Services-Basic” ]
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/icinga-master/hosts.conf’, lines 24:1-24:38
  % = modified in ‘/var/lib/icinga2/api/packages/director/cc5bce90-b4c1-4b47-b9fe-cbcb68a997a8/zones.d/director-global/host_templates.conf’, lines 5:1-5:38
• type = “Host”
• vars = null
• volatile = false
• zone = “icinga-master”

[MartinW (Martin)]

Then I went into the icinga director and tried to select the correct zone for the host, namely satellite-zone, however I can only chose the master zone …

Okay, so I had a look at the zones.conf on the master:

[root@icinga-master icinga2]# cat /etc/icinga2/zones.conf
/*

• Generated by Icinga 2 node setup commands
• on 2020-08-07 10:21:39 +0200
  */

object Endpoint “icinga-master” {
host = “192.168.1.3”
}

object Endpoint “satellite” {
host = “192.168.10.58”
}

object Zone “icinga-master” {
endpoints = [ “icinga-master” ]
}

object Zone “satellite” {
endpoints = [“satellite”]
parent = “icinga-master”
}
object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}

there the satellite zone exists …

[rsx (Roland Sommer)]

I’d assume you didn’t run kick start wizard after you added the satellite’s objects to your zones.conf. And that’s the reason why is does not appear in the list.

BTW: For better readability please format your posts as described here.

[MartinW (Martin)]

@rsx

kick start wizard

Thanks, it seems to be missing (Configuration → Modules → Configuration)

[MartinW (Martin)]

and tried to do it over CLI:

[root@icinga-master icinga2]# icingacli director kickstart run
ERROR: RuntimeException in /usr/share/icingaweb2/modules/director/library/Director/Data/Db/DbObject.php:853 with message: Storing icinga_apiuser[] failed: SQLSTATE[23000]: Integrity constraint violation: 1048 Column ‘object_name’ cannot be null, query was: INSERT INTO icinga_apiuser (object_name, object_type, disabled, password, client_dn, permissions) VALUES (?, ?, ?, ?, ?, ?) {array (
‘id’ => NULL,
‘object_name’ => NULL,
‘object_type’ => ‘external_object’,
‘disabled’ => ‘n’,
‘password’ => NULL,
‘client_dn’ => NULL,
‘permissions’ => NULL,
)}

[rsx (Roland Sommer)]

The error message is clear “Please change the endcoding…” means you didn’t create the director database with CREATE DATABASE director CHARACTER SET 'utf8';

[MartinW (Martin)]

Yeah, I saw that, however it is incorrect.

[MartinW (Martin)]

Hi,

these seems like a bug.

I went into Applications and set the database charset (just a text field) to utf8 and now icinga director works. However, Kickstart is still missing…

[MartinW (Martin)]

Hi,

okay I got one step further.

You have to create “kickstart.ini” under the path: /etc/icingaweb2/modules/director

[config]
endpoint = your-endpoint-here
username = your-api-user-here
password = your-api-user-password-here

Then run:
icingacli director kickstart run

Go into the webinterface Icinga-Director → Activitylogs and accept.

Now I am able to set the host wk2016.local to the z one “satellite”.
However, still it looks as though the icinga-master server tries to check the wk2016 server… When I have a look at the host I can see that “check-source” is set to icinga-master for the “hostalive” plugin. Where can I set this to the satellite-server?

Thanks,

[rsx (Roland Sommer)]

Check source is only updated when a check is triggered to run. Maybe you did not wait long enough? You still can verify the host object with icinga2 object list... and it need to have the satellite as zone.

[MartinW (Martin)]

@rsx

k is triggered to run. Maybe you did not wait long enough? You still can verify the host object with icinga2 obj

Hi,

thanks.

Yes, I checked it with object list and zone seems correct:

• type = “Host”
• vars = null
• volatile = false
• zone = “satellite”

However, if I look into the webinterface now, I can see the following error messages:

Remote Icinga instance ‘wk2016.local’ is not connected to ‘icinga-master’

so I checked with icinga2 ca list on master and satellite, however no new certs have to be approved.

going onto the satellite and checking the icinga logs, I can see:

020-09-04 14:17:17 +0200] information/ApiListener: New client connection for identity ‘wk2016.local’ from [192.168.10.58]:53817
[2020-09-04 14:17:17 +0200] warning/ApiListener: No data received on new API connection from [192.168.10.58]:53817 for identity ‘wk2016.local’. Ensure that the remote endpoints are properly configured in a cluster setup.
[2020-09-04 14:17:20 +0200] information/ApiListener: Reconnecting to endpoint ‘wk2016.local’ via host ‘192.168.10.76’ and port ‘5665’
[2020-09-04 14:17:20 +0200] warning/ApiListener: Unexpected certificate common name while connecting to endpoint ‘wk2016.local’: got ‘monitoring’
[2020-09-04 14:17:20 +0200] information/ApiListener: Finished reconnecting to endpoint ‘wk2016.local’ via host ‘192.168.10.76’ and port ‘5665’

Where to search now?
Could this be an issue of the agent, satellite and master’s zone.conf file?

[rsx (Roland Sommer)]

@MartinW

Unexpected certificate common name while connecting to endpoint ‘wk2016.local’: got ‘monitoring’

It looks like your certificates CN is not identical with the host object ‘wk2016.local’ != ‘monitoring’. I mentioned already the certificate checks.

[MartinW (Martin)]

Thanks, I must have missed that and even going through the thread I cannot see the “certificate checks”.

How would I check them?

Thanks

[rsx (Roland Sommer)]

You’ll find them here.

[MartinW (Martin)]

Hi Roland,

sorry for the late reply what I had some (well earned) holidays

So first thing I did is check if satellite’s cert and master’s cert is the same:

root@monitoring:/tmp/certs# ls -l
total 8
-rw-r–r-- 1 root root 1720 Sep 21 08:49 master_ca.crt
-rw-r–r-- 1 root root 1720 Sep 21 08:48 satellite_ca.crt
root@monitoring:/tmp/certs# diff master_ca.crt satellite_ca.crt
root@monitoring:/tmp/certs#

So this looks good.

However, if I want to use pki tools (on ubuntu) to check the certs in detail, I get:

[quote]root@monitoring:/var/lib/icinga2/certs# pki verify --cacert ca.crt
FileNotFoundError: [Errno 2] No such file or directory: ‘/usr/share/pki/java-home/bin/java’
[/quote]

Java is installed and java_home is exported:

root@monitoring:/var/lib/icinga2/certs# echo $JAVA_HOME
/usr/lib/jvm/java-11-openjdk-amd64

Strictly no icinga problem, but maybe you stumbled accross that one as well using PKI on ubuntu?

Thanks

[MartinW (Martin)]

What I did now is using openssl s_client connect … to test connections from:
Agent to Satellite
Satellite to Master

this looks good as well.

Only thing:

Verify return code: 19 (self signed certificate in certificate chain)

however, this is okay I guess

[MartinW (Martin)]

anybody can help me?