Getting the 'is_reachable' value from Icinga API

geoffhouse · February 3, 2021, 10:14am

Hi. I’m writing a web application which connects via the Icinga2 API.

We have simple host → parent dependencies (usually the local router).
When a parent host goes down, all the dependent hosts have a nice red dot in the icingaweb UI showing that they’re unreachable.

Can I access this value in the API?
I know there’s the last_reachable value, but it’s not the same.
For example the host in the screenshot above has the following results in the API:

"attrs": {
            "__name": "myhostname",
            "acknowledgement": 0.0,
            "acknowledgement_expiry": 0.0,
            "acknowledgement_last_change": 0.0,
            "action_url": "",
            "active": true,
            "address": "172.1.1.2",
            "address6": "",
            "check_attempt": 1.0,
            "check_command": "beagle-hostalive",
            "check_interval": 300.0,
            "check_period": "",
            "check_timeout": null,
            "command_endpoint": "",
            "display_name": "My Display Name",
            "downtime_depth": 0.0,
            "enable_active_checks": true,
            "enable_event_handler": true,
            "enable_flapping": false,
            "enable_notifications": true,
            "enable_passive_checks": true,
            "enable_perfdata": true,
            "event_command": "",
            "flapping": false,
            "flapping_current": 0.0,
            "flapping_last_change": 0.0,
            "flapping_threshold": 0.0,
            "flapping_threshold_high": 30.0,
            "flapping_threshold_low": 25.0,
            "force_next_check": false,
            "force_next_notification": false,
            "groups": [
                "Group 1",
                "Group 2"
            ],
            "ha_mode": 0.0,
            "handled": false,
            "icon_image": "",
            "icon_image_alt": "",
            "last_check": 1612226517.014076,
            "last_check_result": {
                "active": true,
                "check_source": "bcn-bh-icingamaster101",
                "command": [
                    "/usr/lib/nagios/plugins/check_ping",
                    "-H",
                    "172.26.134.210",
                    "-c",
                    "5000,100%",
                    "-w",
                    "3000,80%"
                ],
                "execution_end": 1612226517.014026,
                "execution_start": 1612226512.978128,
                "exit_status": 0.0,
                "output": "PING OK - Packet loss = 0%, RTA = 0.59 ms",
                "performance_data": [
                    "rta=0.592000ms;3000.000000;5000.000000;0.000000",
                    "pl=0%;80;100;0"
                ],
                "schedule_end": 1612226517.014076,
                "schedule_start": 1612226512.9778428,
                "state": 0.0,
                "ttl": 0.0,
                "type": "CheckResult",
                "vars_after": {
                    "attempt": 1.0,
                    "reachable": true,
                    "state": 0.0,
                    "state_type": 1.0
                },
                "vars_before": {
                    "attempt": 1.0,
                    "reachable": true,
                    "state": 0.0,
                    "state_type": 1.0
                }
            },
            "last_hard_state": 0.0,
            "last_hard_state_change": 1607060168.122394,
            "last_reachable": true,
            "last_state": 0.0,
            "last_state_change": 1607060168.122394,
            "last_state_down": 0.0,
            "last_state_type": 1.0,
            "last_state_unreachable": 0.0,
            "last_state_up": 1612226517.014108,
            "max_check_attempts": 3.0,
            "name": "myhostname",
            "next_check": 1612345164.28,
            "next_update": 1612345472.3523662,
            "notes": "",
            "notes_url": "",
            "original_attributes": null,
            "package": "director",
            "paused": false,
            "previous_state_change": 1607060168.122394,
            "problem": false,
            "retry_interval": 60.0,
            "severity": 0.0,
            "source_location": {
                "first_column": 1.0,
                "first_line": 748.0,
                "last_column": 28.0,
                "last_line": 748.0,
                "path": "/var/lib/icinga2/api/packages/director/731ad983-7262-4f51-9476-77076e86f0ea/zones.d/master/hosts.conf"
            },
            "state": 0.0,
            "state_type": 1.0,
            "templates": [
                "Ubiquiti Unifi",
                "Beagle Device"
            ],
            "type": "Host",
            "vars": {
                "host_checkdns": true,
                "host_location": "",
                "host_manufacturer": "Ubiquiti",
                "host_model": "Unifi",
                "host_notificationgroups": [],
                "host_owner": "Department 1",
                "host_parenthost": "myparenthostname"
            },
            "version": 0.0,
            "volatile": false,
            "zone": "master"
        },
        "joins": {},
        "meta": {},
        "name": "myhostname",
        "type": "Host"

“myparenthostname” is down, but “myhostname” was up when the parent went down.
The dependency kicked in and there were no more checks for the host (which would find that the host was indeed ‘down’)
I’m guessing this is why last_reachable is still ‘true’ in this case, but that’s not much good to use in my web app.
Is there a way to get the value used for that red dot on the IcingaWeb UI?

(edited to add the following output from the DB:)
This the output from the relevant result in the mySql database - showing that the database knows that it’s not reachable (which is correct).

mysql> select * from icinga_hoststatus where host_object_id="990" \G
*************************** 1. row ***************************
                hoststatus_id: 66
                  instance_id: 1
               host_object_id: 990
           status_update_time: 2021-02-02 00:41:57
                       output: PING OK - Packet loss = 0%, RTA = 0.59 ms
                  long_output:
                     perfdata: rta=0.592000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0
                 check_source: bcn-bh-icingamaster101
                current_state: 0
             has_been_checked: 1
          should_be_scheduled: 1
        current_check_attempt: 1
           max_check_attempts: 3
                   last_check: 2021-02-02 00:41:57
                   next_check: 2021-02-02 00:46:51
                   check_type: 0
            last_state_change: 2020-12-04 05:36:08
       last_hard_state_change: 2020-12-04 05:36:08
              last_hard_state: 0
                 last_time_up: 2021-02-02 00:41:57
               last_time_down: NULL
        last_time_unreachable: NULL
                   state_type: 1
            last_notification: NULL
            next_notification: 2021-02-02 00:41:53
        no_more_notifications: 0
        notifications_enabled: 1
problem_has_been_acknowledged: 0
         acknowledgement_type: 0
  current_notification_number: 0
       passive_checks_enabled: 1
        active_checks_enabled: 1
        event_handler_enabled: 1
       flap_detection_enabled: 0
                  is_flapping: 0
         percent_state_change: 0
                      latency: 0.000335
               execution_time: 4.035898
     scheduled_downtime_depth: 0
   failure_prediction_enabled: 0
     process_performance_data: 1
             obsess_over_host: 0
     modified_host_attributes: 0
          original_attributes: null
                event_handler: NULL
                check_command: beagle-hostalive
        normal_check_interval: 5
         retry_check_interval: 1
   check_timeperiod_object_id: 0
                 is_reachable: 0
           endpoint_object_id: 283
1 row in set (0.00 sec)

Thanks!

Someone · February 3, 2021, 11:32am

Hello @geoffhouse !
As far as i know, the is_reachable flag is not available directly from api, it can be only accessed from IDO, the reason is that computations required for it are heavy, and so are done only once by IDO, however you can have a similar information in checkresults events with the flags vars_after.reachable and vars_before.reachable, if you get a change between them then you should be in sync with IDO as long as you catch the events on the fly.
You can get checkresults objects in various ways like eventstream or directly requesting api for checkable objects, example :

{
  "check_result": {
    [...]
    "type": "CheckResult",
    "vars_after": {
      "attempt": 1,
      "reachable": true,
      "state": 0,
      "state_type": 1
    },
    "vars_before": {
      "attempt": 1,
      "reachable": true,
      "state": 0,
      "state_type": 1
    }
  },
[...]
}

eventstream general information :
https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#event-streams

check result object definition :
https://icinga.com/docs/icinga-2/latest/doc/08-advanced-topics/#advanced-value-types-checkresult

geoffhouse · February 4, 2021, 8:09am

That’s really helpful - thank you. I’m glad I’m not going mad …

So are you saying that I need to store the values of vars_before and vars_after, or just that I need to compare them?
For the host listed above, I know it’s not reachable, but both the before and after are ‘true’.

I’m easily confused!

theFeu · February 9, 2021, 10:14am

Hey there
I would like to ask you to mark an answer as the solution, if it resolves your topic.

geoffhouse · February 9, 2021, 1:08pm

Hey @theFeu
It kinda does, but I was hoping someone might be able to help a little further.
At the moment I’m considering a number of really hacky options to get this value out of the database - and I really don’t want to have to

Thanks!
Geoff

theFeu · February 9, 2021, 1:33pm

Fair enough!
From what I (and a colleague I just chatted about this topic with) can tell @Someone’s solution is pretty much on point.

You might be able to access the data via the web API, but I’m just assuming that would work - haven’t looked into it

geoffhouse · February 9, 2021, 3:34pm

I’d LOVE it to be available via the web API, but it’s not (unless I’m going mad).

I think I’m going to just have to interrogate the database for every host (after checking the web API) - which is just embarrassing.

Someone · February 9, 2021, 6:19pm

Your question is legit, but i’m afraid i cant help much more about it, sorry.
I admit i’m also curious why on this point icinga check result output in api and ido are not in sync.

To build up my answer i mostly used this + doc i previously linked :

github.com/Icinga/icinga2

Add "is_reachable" to host object

opened 03:45PM - 18 Dec 18 UTC

closed 01:21PM - 07 May 19 UTC

thomseddon

bug area/api

## Expected Behavior When requesting a host object via the api (https://x.x.x.x…:5665/v1/objects/hosts?hosts=1234), the `is_reachable` field should be returned ## Current Behavior The `is_reachable` field is not returned. ## Possible Solution Include `is_reachable` in API response. ## Steps to Reproduce (for bugs) 1. `curl -k -s -X GET -H 'Accept: application/json' -H 'Content-Type: application/json' -u xx:xx 'https://x.x.x.x:5665/v1/objects/hosts?hosts=1234'` 2. 3. 4. ## Context For example, the host below is not reachable due to a failed dependency. We have "disable_checks" enabled and as the dependency failed after the last check, no further checks were executed. As the last executed check succeeded there is no way to determine that the host is unreachable via the API. icingaweb2 shows the host is not reachable as determined by the "is_reachable" field which is available via the IDO interface but not the API :( ``` # curl -k -s -X GET -H 'Accept: application/json' -H 'Content-Type: application/json' -u xx:xx 'https://x.x.x.x:5665/v1/objects/hosts?hosts=1988' | jq { "results": [ { "attrs": { "__name": "1988", "acknowledgement": 0, "acknowledgement_expiry": 0, "action_url": "", "active": true, "address": "10.0.0.1", "address6": "", "check_attempt": 1, "check_command": "ping4", "check_interval": 30, "check_period": "", "check_timeout": null, "command_endpoint": "", "display_name": "hostname123", "downtime_depth": 0, "enable_active_checks": true, "enable_event_handler": true, "enable_flapping": true, "enable_notifications": true, "enable_passive_checks": true, "enable_perfdata": true, "event_command": "", "flapping": false, "flapping_current": 0, "flapping_last_change": 1515772203.052052, "flapping_threshold": 0, "flapping_threshold_high": 30, "flapping_threshold_low": 25, "force_next_check": false, "force_next_notification": false, "groups": [ "hosts-priority-high" ], "ha_mode": 0, "icon_image": "", "icon_image_alt": "", "last_check": 1545096024.779654, "last_check_result": { "active": true, "check_source": "example.telcom.network", "command": [ "/usr/lib64/nagios/plugins/check_ping", "-4", "-H", "10.0.0.1", "-c", "5000,100%", "-p", "5", "-w", "500,100%" ], "execution_end": 1545096024.779608, "execution_start": 1545096020.74619, "exit_status": 0, "output": "PING OK - Packet loss = 0%, RTA = 19.74 ms", "performance_data": [ "rta=19.742001ms;500.000000;5000.000000;0.000000", "pl=0%;100;100;0" ], "schedule_end": 1545096024.779654, "schedule_start": 1545096020.745869, "state": 0, "type": "CheckResult", "vars_after": { "attempt": 1, "reachable": true, "state": 0, "state_type": 1 }, "vars_before": { "attempt": 1, "reachable": true, "state": 0, "state_type": 1 } }, "last_hard_state": 0, "last_hard_state_change": 1544816401.983204, "last_reachable": true, "last_state": 0, "last_state_change": 1544816401.983204, "last_state_down": 1544816260.430792, "last_state_type": 1, "last_state_unreachable": 1543767570.038673, "last_state_up": 1545096024.779696, "max_check_attempts": 3, "name": "1988", "next_check": 1545147302.29, "notes": "", "notes_url": "", "original_attributes": null, "package": "_etc", "paused": false, "retry_interval": 10, "severity": 8, "source_location": { "first_column": 1, "first_line": 7081, "last_column": 18, "last_line": 7081, "path": "/etc/icinga2/zones.d/masters/hosts.conf" }, "state": 0, "state_type": 1, "templates": [ "1988", "host-priority-high" ], "type": "Host", "vars": { "location_id": "8", "location_name": "A Location, UK", "ping_cpl": 100, "ping_crta": 5000, "ping_packets": 5, "ping_wpl": 100, "ping_wrta": 500, "priority": 2, "priority_str": "example", "sla": "24x7", "subnet_id": "310" }, "version": 0, "volatile": false, "zone": "masters" }, "joins": {}, "meta": {}, "name": "1988", "type": "Host" } ] } ``` ## Your Environment * Version used (`icinga2 --version`): r2.8.1-1 * Operating System and version: Amazon Linux AMI * Enabled features (`icinga2 feature list`): api checker command ido-mysql mainlog notification * Icinga Web 2 version and modules (System - About): 2.4.1 * Config validation (`icinga2 daemon -C`): N/A * If you run multiple Icinga 2 instances, the `zones.conf` file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes. N/A

However, to be really sure of whats going and why it’s different it’s required to dig in the code, @theFeu and icinga team can give a much better explanation about it (no pun intended).
From what i understood icinga is supposed to use the same functions to compute is_reachable value and the one from checkresult object.

what version of icinga are you using ? the same behavior for icinga api and ido comes with 2.11 based on github

Someone · February 9, 2021, 6:35pm

Also, i wanted to add something, i’m not using it (yet), but you may also have the information you are looking for in icingadb throught redis after if you are using version 2.12+, however i dont know if the reachable flag in icingadb behave in the same way than ido.

geoffhouse · February 11, 2021, 9:08pm

Version r2.12.3-1

I’m kind of hoping that IDO is right and the API is wrong, because the host isn’t reachable. It has a dependency to the local router host - which is down. So I can’t reach it.

Unless I’m misunderstanding.

Thanks for your help so far!