Situation
Hi Community!
Im Trying to add a new host, to my existing setup, which is managed by puppet. I have a distributet setup with one master and multiple agents. The master and most agents run Debian. The new server although is not managed by puppet so i have to add it manually to the system. The new server runs Debian and is used to self-host our gitlab repos and to run a few docker instances. I used the node setup wizard, for the agent setup. All servers run Icinga2 version r2.12.3-1.
My Problem is, that all checks, which have the host itself as the endpoint, stay unnkown. The host itself shows as up, and the my ping check works fine. most other checks show:
Remote Icinga instance 'git.xxxxxxx.de' is not connected to 'icinga.xxxxxxxxx'
Logs
This keeps showing up in the debug log on the agent:
[2021-02-10 16:08:31 +0100] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'icinga.xxxxxxxx': Error: End of file
icinga2: icinga::NetString::ReadStringFromStream(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) (+0x3d1) [0x55902e9e35b1]
icinga2: icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) (+0x49) [0x55902e8d6199]
icinga2: icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) (+0x119) [0x55902e8e3469]
icinga2: <unknown function> (+0x67c2dd) [0x55902e8c62dd]
icinga2: <unknown function> (+0x648992) [0x55902e892992]
libboost_context.so.1.67.0: make_fcontext (+0x2f) [0x7fd37be231ef]
[2021-02-10 16:08:31 +0100] warning/JsonRpcConnection: API client disconnected for identity 'icinga.xxxxxxxxxx'
Snippet from master debug log when filtering for the host in question:
[2021-02-10 16:03:00 +0100] debug/CheckerComponent: Check finished for object 'git.xxxxxx.de!gitlab-services'
[2021-02-10 16:03:01 +0100] debug/ApiListener: Not connecting to Endpoint 'git.xxxxxxx.de' because we re already trying to connect to it.
[2021-02-10 16:03:07 +0100] debug/CheckerComponent: Scheduling info for checkable 'git.xxxxxxx.de!disk' (2021-02-10 16:03:07 +0100): Object 'git.xxxxxxxxx.de!disk', Next Check: 2021-02-10 16:03:07 +0100(1.61297e+09).
[2021-02-10 16:03:07 +0100] debug/CheckerComponent: Executing check for 'git.xxxxxxxx.de!disk'
[2021-02-10 16:03:07 +0100] debug/Checkable: Update checkable 'git.xxxxxx.de!disk' with check interval '60' from last check time at 2021-02-10 15:59:16 +0100 (1.61297e+09) to next check time at 2021-02-10 16:04:05 +0100 (1.61297e+09).
[2021-02-10 16:03:07 +0100] debug/CheckerComponent: Check finished for object 'git.xxxxxxx.de!disk'
[2021-02-10 16:03:11 +0100] debug/ApiListener: Not connecting to Endpoint 'git.xxxxxx.de' because we re already trying to connect to it.
[2021-02-10 16:03:14 +0100] debug/CheckerComponent: Scheduling info for checkable 'git.xxxxxx.de!procs_higher' (2021-02-10 16:03:14 +0100): Object 'git.xxxxxx.de!procs_higher', Next Check: 2021-02-10 16:03:14 +0100(1.61297e+09).
Prior problems (maybe badly fixed?)
The certificate was signed on demand. Although when i first tried singing it, the directory /var/lib/icinga2/ca was missing. I made the directory myself, copied over icinga.xxxxxxxxx.key and icinga.xxxxxxx.crt from /var/lib/icnga2/certs as ca.crt and ca.key and changed owner and group to nagios. I was able to sign the certificate on the master after this, but the servers still aren’t connecting correctly. Maybe the problem lies somwhere here?
My Environment / Config
enabled features on the master: api checker ido-pgsql mainlog notification
enabled features on the agent: api checker debuglog mainlog
the master has icingaweb2. version 2.8.2
Master
on the maste, I have two directories which are not managed by puppet.
/etc/icinga2/manual-entries/etc/icinga2/zones.d/master/manual-entries
In the first one there is the zone config. I cant have it in zones.conf, because it would be overwritten by the puppet.
object Endpoint "git.xxxxxxxx.de" {
host = "git.xxxxxxxx.de"
port = "5665"
}
object Zone "git.xxxxxxxx.de" {
endpoints = [ "git.xxxxxxxx.de", ]
parent = "master"
}
in the second one there’s a subdirectory git.xxxxxxxxx.de/ with git.xxxxxxxxx.de.conf for the host config and services.conf.
the Host config:
object Host "git.xxxxxxx.de" {
import "generic-host"
address = "xxxxxxxxx"
address6 = "xxxxxxxxxxxx"
vars.kritikalitaet = "kritisch"
vars.docker = true
vars.https = true
vars.client_endpoint = name
}
Agent
the zone config on the agent is:
object Endpoint "icinga.xxxxxxxxx" {
}
object Zone "master" {
endpoints = [ "icinga.xxxxxxxxx" ]
}
object Endpoint "git.xxxxxxx.de" {
}
object Zone "git.xxxxxxx.de" {
endpoints = [ "git.xxxxxxx.de" ]
parent = "master"
}
object Zone "global-templates" {
global = true
}
object Zone "director-global" {
global = true
}
Puppet
I am using the icinga2 modules for puppet. This is some of my master config:
class profile::icinga::master {
$icinga_host = lookup('infra::icinga')
class { '::icinga2':
confd => manual-entries,
constants => {
'ZoneName' => 'master',
'TicketSalt' => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
},
}
class { '::icinga2::feature::api':
pki => 'puppet',
ticket_salt => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
accept_commands => true,
accept_config => false,
ca_host => '127.0.0.1',
endpoints => {
$icinga_host => {
'host' => '127.0.0.1',
},
},
zones => {
'master' => {
'endpoints' => [$icinga_host],
},
},
}
I hope this is enough information, please tell me if you need any more.
I’ve tried for quite some time to solve this problem already, but now i think im stuck…
I’ll be very glad for any help or tips. 
