How to use passive or dummy command in heart-beat context

Hi,

I have to monitor about 1500 “nodes” (hosts), the main constraint is that icinga2 can’t reach hosts.

However, each host can send data to icinga.
On each host, a service is configured to send a POST to /v1/actions/process-check-result every 2 minutes.

I’ve tried to setup hosts with dummy or passive command as check_command, with template like:

template Host "test_dummy_template" {
    check_command = "dummy"
    max_check_attempts = "1"
    check_interval = 2m
    retry_interval = 2m
    check_timeout = 2m
    enable_active_checks = false
    enable_passive_checks = true
    vars.dummy_state = 2
    vars.dummy_text = "No news from this site"
}

And host like that:

object Host "test_dummy_host" {
    check_command = "dummy"
    max_check_attempts = "1"
    check_interval = 2m
    retry_interval = 2m
    check_timeout = 2m
    enable_active_checks = false
    enable_passive_checks = true
    vars.dummy_state = 2
    vars.dummy_text = "No news from this site"
}

Every 120s, each host send check results with these datas:

POST /v1/actions/process-check-result?host=test_dummy_host HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Basic xxxxxxxxxxxxxxxxxx
Connection: keep-alive
Content-Length: 122
Content-Type: application/json
Host: X.X.X.X:5665
User-Agent: HTTPie/2.6.0

{
    "check_source": "icingaserver",
    "exit_status": "0",
    "plugin_output": "PING OK",
    "ttl": "120",
    "type": "Host"
}

Just after the first configuration, the host state is “unknown”, when icinga receive the POST action, the host go to OK.

My problem is, if I stop the process who send the POST actions, the line “Last update” go red, but the host remain “OK”:

The behaviour I try to setup is: “the host remains green as long as icinga receives check-results, in the absence of check-results the host goes to fail”

I have too questions:

  • Is the behavior I want to obtain possible with Icinga?
  • What part of the functioning I did not understand or not well configured?

I hope that my explanations are clear, good day to all,

Best regards,

PS: I’m not sure that below informations are relevant here, but:

> icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.2-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 20.04.3 LTS (Focal Fossa)
  Kernel: Linux
  Kernel version: 5.4.0-99-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 9.3.0
  Build host: runner-hh8q3bz2-project-298-concurrent-0
  OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
> icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql mainlog notification
  • Is the behavior I want to obtain possible with Icinga?

Yes I think it’s possible :slight_smile:

object Host "test_dummy_host" {
    check_command = "dummy"
    max_check_attempts = "1"
    check_interval = 2m
    retry_interval = 2m
    check_timeout = 2m
    enable_active_checks = false
    enable_passive_checks = true
    vars.dummy_state = 2
    vars.dummy_text = "No news from this site"
}

enable_active_checks should be set to true here.

I also noticed that this Host object does not import the Host template - is this what you wanted?

I’m curious if this configuration does what we want:

template Host "dummy-host" {
    check_command = "dummy"
    max_check_attempts = 1
    check_interval = 2m
    vars.dummy_state = 2
    vars.dummy_text = "No news from this site"
}

object Host "test1" {
	import "dummy-host"
}

object Host test2" {
	import "dummy-host"
}

I removed a few attributes in the template definition.

  • enable_active_checks default value is already true.
  • enable_passive_checks default value is already true.
  • check_timeout is already set by the CheckCommand “dummy”.
  • retry_timeout is never used as the Host immediately goes into a HARD state since max_check_attempts is set to 1.
{
    "check_source": "icingaserver",
    "exit_status": "0",
    "plugin_output": "PING OK",
    "ttl": "120",
    "type": "Host"
}

I think it’s also unnecessary to set ttl in this request, since it just overrides the Host’s check_interval to the same value (2 minutes). There is a little bit more info on this in the Advanced Topics section of the Icinga docs: Advanced Topics - Icinga 2

Curious if this works for you :slight_smile:

1 Like

I’d be interested to see if this works. I think I misread this earlier today (perhaps OP did as well):

If the freshness checks fail, Icinga 2 will execute the defined check command unless active checks are disabled.

A lot of negatives. With the change of enable_active_checks to true, it seems Icinga 2 will execute the defined check command if active checks are enabled (what we want).

A lot of negatives. With the change of enable_active_checks to true, it seems Icinga 2 will execute the defined check command if active checks are enabled (what we want).

Haha good point. I submitted a patch for the docs with some perhaps clearer wording:
https://github.com/Icinga/icinga2/pull/9215/commits/1aedbd563e0b2dc955171448830814ad4568f07c

1 Like

Yes, of course, you are right! it is obvious, how could I have missed it!
Active check must be enable to run dummy/passive command.

Changing this field solved my problem, and everything work as expected.

It can be clearer if default value is printed/remembered.
Icingaweb show “- please chose -”, “- please choose (default Yes) -” would be comforting.

Thanks a lot for your solution and improvements,

Best regards

1 Like

I have seen that other topics are marked as resolved, but I don’t see how to do it at all.

The FAQ clearly explains why to do it, but does not address how to do it.