Icinga Remotechecks via API

[b3n4kh (ben)]

Is it possible to show the last check result of checks triggerd via command_endpoint?
Or is there a way to define a check on the agent and just tell the master to gather the check results?

Backstory:
I got a pretty basic distributed monitoring setup with a single master, two satellites and some agents. This setup works just fine, for distributed monitoring. Configuration gets synced via Puppet so I got no need for Icinga Director.
The Agents, tend to lose connection to their Master / Satellites due to the nature of their connection (point2point radio, sometimes even satellite) so if they come back online the replay their log, this works fine too.

The question is, is there a way for the agent, to show the last check result of the remote check via API or CLI? The only way I found would be defining the checks on the Agent (In conf.d) as well as on the Master (zones.d), but that feels strange (problems are described here it only “works” for me since the checks are identical). Or is there even a way to define a check on the agent and just tell the master to gather the check results?

[stevie-sy]

Hi,

that’s easy. Do a request like written in the docs and you get a JSON with every value from the check.

e.g. for service

curl -k -s -u xxx:xxx ‘https://your-icinga-server:5665/objects/services?service=hostname!servicename’

e.g. for host

curl -k -s -u xxx ‘https://your-icinga-server:5665/objects/hosts?host=hostname’

[b3n4kh (ben)]

This will only show me checks configured locally in conf.d. Not checks that are triggerd via command_endpoint from an icinga master. But getting the outupt of this command would be exactly what I need. Please tell me if my question is unclear, so I will try to rephrase.

[stevie-sy]

Pls show us the output after you do the api request.

Because if i do this, if get everything including the last performance data the check gots.

[b3n4kh (ben)]

If I execute it on the agent:

$ curl -k -s -u xxx:xxx ‘https://agent1.tld:5665/objects/services’
{
  "results": []
}

If I execute the same check on the Master

$ curl -k -s -u xxx:xxx ‘https://master.tld:5665/objects/services?service=agent1.tld!disk’
{
    "results": [
        {
            "attrs": {
                "__name": "agent1.tld!disk",
                "acknowledgement": 0.0,
                "acknowledgement_expiry": 0.0,
                "action_url": "",
                "active": true,
                "check_attempt": 1.0,
                "check_command": "disk",
                "check_interval": 600.0,
                "check_period": "",
                "check_timeout": null,
                "command_endpoint": "agent1.tld",
                "display_name": "disk",
                "downtime_depth": 0.0,
                "enable_active_checks": true,
                "enable_event_handler": true,
                "enable_flapping": false,
                "enable_notifications": true,
                "enable_passive_checks": true,
                "enable_perfdata": true,
                "event_command": "",
                "flapping": false,
                "flapping_current": 0.0,
                "flapping_last_change": 0.0,
                "flapping_threshold": 0.0,
                "flapping_threshold_high": 30.0,
                "flapping_threshold_low": 25.0,
                "force_next_check": false,
                "force_next_notification": false,
                "groups": [],
                "ha_mode": 0.0,
                "handled": false,
                "host_name": "agent1.tld",
                "icon_image": "",
                "icon_image_alt": "",
                "last_check": 1573560339.284133,
                "last_check_result": {
                    "active": true,
                    "check_source": "agent1.tld",
                    "command": [
                        "/usr/lib64/nagios/plugins/check_disk",
                        "-c",
                        "92%",
                        "-w",
                        "90%",
                        "-X",
                        "none",
                        "-X",
                        "tmpfs",
                        "-X",
                        "sysfs",
                        "-X",
                        "proc",
                        "-X",
                        "configfs",
                        "-X",
                        "devtmpfs",
                        "-X",
                        "devfs",
                        "-X",
                        "mtmfs",
                        "-X",
                        "tracefs",
                        "-X",
                        "cgroup",
                        "-X",
                        "fuse.gvfsd-fuse",
                        "-X",
                        "fuse.gvfs-fuse-daemon",
                        "-X",
                        "fdescfs",
                        "-X",
                        "overlay",
                        "-X",
                        "nsfs",
                        "-X",
                        "squashfs",
                        "-m",
                        "-p",
                        "/",
                        "-p",
                        "/opt",
                        "-p",
                        "/var"
                    ],
                    "execution_end": 1573560339.284034,
                    "execution_start": 1573560339.276218,
                    "exit_status": 2.0,
                    "output": "DISK CRITICAL - free space: / 2414 MB (60.50% inode=97%); /opt 344 MB (69.37% inode=94%); /var 1871 MB (94.04% inode=100%);",
                    "performance_data": [
                        "/=1575MB;398;319;0;3990",
                        "/opt=152MB;49;39;0;496",
                        "/var=118MB;198;159;0;1990"
                    ],
                    "schedule_end": 1573560339.284133,
                    "schedule_start": 1573560339.284133,
                    "state": 2.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": true,
                        "state": 2.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": true,
                        "state": 2.0,
                        "state_type": 1.0
                    }
                },
                "last_hard_state": 2.0,
                "last_hard_state_change": 1573547559.134351,
                "last_reachable": true,
                "last_state": 2.0,
                "last_state_change": 1573547381.753345,
                "last_state_critical": 1573560339.285917,
                "last_state_ok": 0.0,
                "last_state_type": 1.0,
                "last_state_unknown": 0.0,
                "last_state_unreachable": 0.0,
                "last_state_warning": 0.0,
                "max_check_attempts": 5.0,
                "name": "disk",
                "next_check": 1573560639.199642,
                "notes": "",
                "notes_url": "",
                "original_attributes": null,
                "package": "_etc",
                "paused": false,
                "previous_state_change": 1573547381.753345,
                "problem": true,
                "retry_interval": 60.0,
                "severity": 136.0,
                "source_location": {
                    "first_column": 1.0,
                    "first_line": 18.0,
                    "last_column": 21.0,
                    "last_line": 18.0,
                    "path": "/etc/icinga2/zones.d/smn/agent1.tld.conf"
                },
                "state": 2.0,
                "state_type": 1.0,
                "templates": [
                    "disk",
                    "generic-service"
                ],
                "type": "Service",
                "vars": {
                    "disk_cfree": "92%",
                    "disk_partitions": [
                        "/",
                        "/opt",
                        "/var"
                    ],
                    "disk_wfree": "90%"
                },
                "version": 0.0,
                "volatile": false,
                "zone": "smn"
            },
            "joins": {},
            "meta": {},
            "name": "agent1.tld!disk",
            "type": "Service"
        }
    ]
}

[b3n4kh (ben)]

You can see the services on your agent that you configured on your master? How did you configure them, maybe I just missed an option on the master?

My Services are defined in /etc/icinga2/zones.d/<zone>/<agent>.conf
And look like this example below:

object Service "disk" {
  command_endpoint = "agent1.tld"
  host_name = "agent1.tld"
  display_name = "disk"
  check_command = "disk"
}

Which works perfect for checking on the Master, and I see everything in my icingaweb2, I just can’t access the results from the client.

[stevie-sy]

Yes, but including the JSON you have this:

“execution_end”: 1573560339.284034,
“execution_start”: 1573560339.276218,
“exit_status”: 2.0,
“output”: “DISK CRITICAL - free space: / 2414 MB (60.50% inode=97%); /opt 344 MB (69.37% inode=94%); /var 1871 MB (94.04% inode=100%);”,
“performance_data”: [
“/=1575MB;398;319;0;3990”,
“/opt=152MB;49;39;0;496”,
“/var=118MB;198;159;0;1990”
],

and a little bit underneath

“last_hard_state”: 2.0,
“last_hard_state_change”: 1573547559.134351,
“last_reachable”: true,
“last_state”: 2.0,
“last_state_change”: 1573547381.753345,
“last_state_critical”: 1573560339.285917,
“last_state_ok”: 0.0,
“last_state_type”: 1.0,
“last_state_unknown”: 0.0,
“last_state_unreachable”: 0.0,
“last_state_warning”: 0.0,

a few lines further

“next_check”: 1573560639.199642,

if I understand your question correctly, this is what you want to know

[b3n4kh (ben)]

Yes this is what I want to know, but on the agent and not on the master
On the agent I only get:

{
  "results": []
}

[stevie-sy]

if you want this directly from the satellite or agent, you have to configure the api feature also there.

[b3n4kh (ben)]

The feature is configured, thats why I’m getting an output, I am also authenticated, I can successful call
$ curl -k -s -u xxx:xxx ‘https://agent1.tld:5665/v1/status’ and get output of the agent including what I configured in zones.conf. But I can’t access Checks configured on the Master from the agent.

[stevie-sy]

could you show us the api.conf, icinga2 Feature list, icinga2 daemon -c from this node?

[b3n4kh (ben)]

root@agent1.tld # cat /etc/icinga2/features-enabed/api.conf
object ApiListener "api" {
  accept_commands = true
  accept_config = true
}

In my conf.d there is only one file:

root@agent1.tld # cat /etc/icinga2/conf.d/api-user.conf
object ApiUser "root" {
  client_cn = "agent1.tld"
  permissions = [ "*" ]
}

root@agent1.tld # icinga2 daemon -C
[2019-11-12 13:41:16 +0100] information/cli: Icinga application loader (version: 2.11.2-1)
[2019-11-12 13:41:16 +0100] information/cli: Loading configuration file(s).
[2019-11-12 13:41:16 +0100] information/ConfigItem: Committing config item(s).
[2019-11-12 13:41:16 +0100] information/ApiListener: My API identity: agent1.tld
[2019-11-12 13:41:16 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2019-11-12 13:41:16 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2019-11-12 13:41:16 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-11-12 13:41:16 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2019-11-12 13:41:16 +0100] information/ConfigItem: Instantiated 3 Zones.
[2019-11-12 13:41:16 +0100] information/ConfigItem: Instantiated 2 Endpoints.
[2019-11-12 13:41:16 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2019-11-12 13:41:16 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2019-11-12 13:41:16 +0100] information/ConfigItem: Instantiated 229 CheckCommands.
[2019-11-12 13:41:16 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2019-11-12 13:41:16 +0100] information/cli: Finished validating the configuration file(s).

[b3n4kh (ben)]

What should I be able to see? I don’t think I will be able to see the services since they are never difend on the agent, should I be able to see the check result? What endpoint would I see services defined on the master? Is there any documentation on this?

[stevie-sy]

Api seems to be configured correctly.

But with “icinga2 daemon -c” you see which configuration a satellite or agent gets from the config master. In your case your output shows you, that there are no hosts, services configured. Or in other word your agent/satellite doesn’t gets this from the (config) master. Maybe something is wrong in the cluster config.

[b3n4kh (ben)]

Thank you for taking time to help me narrow down the issue!

I do not have config sync down to the agent, only to the satellite inbetween, the actual check is triggerd on a satellite via “command_endpoint” as I said in the Original Post.

As far as I know it is not possible to sync the check down to the agent, my “global-template” config sync only copies “checkcommands” since services and hosts can not be copied via Global Zone Config sync

Note : Checkable objects (hosts and services) cannot be put into a global zone.

[stevie-sy]

ok, how do you configure your Icinga Cluster and your hosts/services? Do you use director, Ansible or other config tools or do you write the config files?

[b3n4kh (ben)]

I wrote in my OP that I use puppet. Are you familiar with how command_endpoints work?

[dnsmichi (Michael Friedrich)]

@b3n4kh

The question is, is there a way for the agent, to show the last check result of the remote check via API or CLI?

No. That’s because the agent doesn’t have a physical host/service object locally, it just executes a given received command with arguments in a virtual context. The executed command is logged though with enabling notice/debug log for further troubleshooting.

Cheers,
Michael

[stevie-sy]

for your questions about command_endpoint you could check this docs:

• https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#pin-checks-in-a-zone
• https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#top-down-command-endpoint
• https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#three-levels-with-masters-satellites-and-agents