Objects I create on the master are not visible to the satelite - why?

Icinga 2
1

I have a problem that I can’t get past - I see this a lot in icinga2.log on the master (I have a master and a satelite zone, and I have recently upgraded both to version 2.10):

[2019-03-15 13:28:07 +0000] information/ApiListener: New client connection from [146.179.60.11]:53402 (no client certificate)
[2019-03-15 13:28:07 +0000] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-15 13:28:07 +0000] information/HttpServerConnection: Request: PUT /v1/objects/services/cx1-141-9-2.cx1.hpc.ic.ac.uk%21check_fs (from [146.179.60.11]:53402), user: masteradm)
[2019-03-15 13:28:07 +0000] critical/config: Error: An object with type 'Service' and name 'cx1-141-9-2.cx1.hpc.ic.ac.uk!check_fs' already exists (in /var/lib/icinga2/api/packages/_api/4f4482f9-394c-4ecf-86d6-e5adeb32995b/conf.d/services/cx1-141-9-2.cx1.hpc.ic.ac.uk!check_fs.conf: 1:0-1:24), new declaration: in /var/lib/icinga2/api/packages/_api/4f4482f9-394c-4ecf-86d6-e5adeb32995b/conf.d/services/cx1-141-9-2.cx1.hpc.ic.ac.uk!check_fs.conf: 1:0-1:24
Location: in /var/lib/icinga2/api/packages/_api/4f4482f9-394c-4ecf-86d6-e5adeb32995b/conf.d/services/cx1-141-9-2.cx1.hpc.ic.ac.uk!check_fs.conf: 1:0-1:24
/var/lib/icinga2/api/packages/_api/4f4482f9-394c-4ecf-86d6-e5adeb32995b/conf.d/services/cx1-141-9-2.cx1.hpc.ic.ac.uk!check_fs.conf(1): object Service "check_fs" {
                                                                                                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/packages/_api/4f4482f9-394c-4ecf-86d6-e5adeb32995b/conf.d/services/cx1-141-9-2.cx1.hpc.ic.ac.uk!check_fs.conf(2):  check_command = "dummy"
/var/lib/icinga2/api/packages/_api/4f4482f9-394c-4ecf-86d6-e5adeb32995b/conf.d/services/cx1-141-9-2.cx1.hpc.ic.ac.uk!check_fs.conf(3):  check_interval = "1800"

[2019-03-15 13:28:07 +0000] critical/config: 1 error
[2019-03-15 13:28:07 +0000] warning/TlsStream: TLS stream was disconnected.
[2019-03-15 13:28:07 +0000] information/HttpServerConnection: HTTP client disconnected (from [146.179.60.11]:53402)

The reason this happens is that I create host and service objects dynamically, through the REST API:

  1. The script checks if the satelite knows the service
  2. If not, it creates it with a PUT to the master

However, this is not then communicated back to the satelite - sometimes, but not always. So, next time the script will try to create the object, but can’t, of course, and the log fills up with this. Is there any way to troubleshoot this? It seems to only occur when there is significant load; if I try to stop the cron jobs that geerate to updates, and run the checks manually, it works like a charm.

2

Seems you’re bitten by this bug.

Cheers,
Michael

3

Oh, interesting. Can I avoid this by slowing down the pace of updates? Is there perhaps an option to rate-limit the updates or something similar, like limiting the number of threads listening for connections? I can live with temporary connection failures on the client side, if I know I don’t run into this problem.

4

Unfortunately there is no way to throttle the sync.

It should occur only rarely and the workaround is to restart the sync by deleting /var/lib/icinga2/api/zones/* and /var/lib/icinga2/api/packages/* while the Icinga 2 service is stopped.

5

It is a problem of concurrent API connections, so limiting them one-by-one may help up until there’s a fix for a) the network stack b) config sync in 2.11.