Best way for a secure remote "Master" instance (only view) in case of network issues on the office

Icinga 2
1

Hello,

at the moment we have several satellites ( a satellite per datacenter) which connects to our two masters (via VPN) on our office network. The problem we want to solve is: if the office network is down we are blind … because we can’t see anything. So we need a Icinga2 instance outside the office network (may on a Linode / AWS … / ) which gives us the possibility to check for the healthstate on the DCs.
It should only show us the state of the hosts / services … and nothing more, as it is needed only a few times per year.

Just to add a 3rd Icinga2 instance would not work … as the Icinga2 / Icingaweb2 can not connect to the same DB …

Any suggestions ?

cu denny

2

You could host a 3rd individual icinga master on AWS/linode and post passive check results to it.

DC fetches service states by itself-> sends it agentless to the third icinga machine → icinga machine sends notifications if there is no check result or a critical one

2 Likes
3

hi,

thanks for the reply … but … how does the 3rd Icinga node knows, which services exists ? At this time … we have per DC ~100 hosts with ~50 checks per host.
The masters are managed with Icingaweb Director + Puppet (for plugins and the file stuff).

cu denny

4

  1. In Icinga director you create a service template with the checkcommand passive

  2. Create a service on “YOUR-DC1” using that service template and give it a name (passive-testservice1)

  3. Create an icinga API user to submit passive checkresults

  4. write a script that collect the data on the dc and send your passive check result according to: https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#process-check-result

I would secure that via vpn and use the vpn ip address to submit the result

curl -k -s -S -i -u icingaapiuser:icingaapiuserpw -H 'Accept: application/json' \
 -X POST 'https://icinga-on-aws:5665/v1/actions/process-check-result' \
-d '{ "type": "Service", "filter": "host.name==\"YOUR-DC1\" && service.name==\"passive-testservice1\"", "exit_status": 2, "plugin_output": "You are all gonna die done here...", "performance_data": [ "rta=5000.000000ms;3000.000000;5000.000000;0.000000", "pl=100%;80;100;0" ], "check_source": "yourdc1", "pretty": true }'

Send your result in an interval (5m) smaller than the check interval of the service template eg 10m otherwise your check flips to unknown because of the passive check command

If you send “exit_status”: 0 The check flips to OK

Example:
Service template: retryinterval 1m, check interval 10m, max attempts 3
Your result managed my some task scheduler interval: 5m

1 Like
5

hi,

big thanks ! That sounds pretty good :slight_smile:

6

Hi,

I’ve found also a 2nd solution … API + Dashing ! I totally forgotten Dashing (thanks to HomeOffice). So I can run a container on our external docker hosts, which are already have access to the Icinga satellite. With a revers proxy and LDAP authentication … I can access the dashing from the outside, without too much work. Ok, business checks are not available and also no notifications … but hey … we need it only 1-2 times per year :slight_smile:

1 Like