Edit: This issue is very similar to the following - but that one was mainly about the fact that I hadn’t configured the zone correctly in the service objects; this is not the issue now:
However, the symptoms seem to be the same:
- I have a master zone, a satelite, and a number of clients
- All service checks are passive, and the first time they are run, they create their own host and service objects, if necessary
This appears to work fine - except that it seems to be unrealiable. After I create a new ServiceGroup and restart the master/satelite, all host and service objects disappear, but are the recreated. I did this before the weekend, everything seemed OK, sensible data were seen, but now it turns out that no updates have been received since.
When I try to run the check scripts manually, they get a return code 404, so the service object doesn’t exist. However, when I query the servers in different ways, they do exist - I have tried through the API and using the console (“icinga2 console --connect ‘https://some:firstname.lastname@example.org:5665/’”), and I find the service objects. It is a bit of a disaster TBH, if I can’t get it to work reliably - or at least can find a reliable way to troubleshoot this sort of issue. Any ideas, please?
I have had a look in the debug log and the icinga2.log on the satelite - the only thing that stands out is the fact that I see this sort of thing an awful lot in the icinga2.log - in fact, it seems to happen for all updates:
[2019-03-11 10:31:23 +0000] information/ApiListener: New client connection from [192.168.103.61]:52792 (no client certificate) [2019-03-11 10:31:23 +0000] information/HttpServerConnection: Request: POST /v1/actions/process-check-result?service=cx1-138-16-1.cx1.hpc.ic.ac.uk%21sdr_list (from [192.168.103.61]:52792, user: client-pki-ticket-cx1-admin) [2019-03-11 10:31:23 +0000] warning/TlsStream: TLS stream was disconnected.
On the master, there are no log entries relating to the updates. These problems seem to arise every time I add a ServiceGroup at the moment - I do this by changing
icinga2 daemon -C and restarting the icinga server on master and satelite. Is there anything I need to do differently?