I am running icinga version 2.10.3-1 on the master.
I ran into an issue recently where one of my servers locked up but icinga did not send any notifications about the issue. Once i rebooted the machine I stopped the icinga service to see what would happen and it basically just keeps trying to execute the check but does not actually send any warnings or go to a Critical state. I checked the icinga2 logs and the only message relating to the server is "information/ApiListener: Finished reconnecting to endpoint ‘server-test’ via host ‘220.127.116.11’ and port ‘5665’ . But how is it connecting if the icinga service is down on the client? I checked to see if the master node was connected to the client via lsof command and grepping for the IP address but it does not show up. I had this issue once before and what helped solve this issue was changing the hostalive command to cluster-zone but this did not help this time.