History for passive check results

Hi at all,

we have a question how you handle the following case or if you have any ideas before I create a possible feature request:
For some devices we have to use passive check results. It’s specifically about SNMP Traps, Syslog or backup scripts which sends a message that it’s finished (like the idea from the icinga book). To process SNMP traps and syslog messages we use logstash. Very often it happens that some devices sends in short interval a lot of diffrent messages.

Here is the problem:
If the script or devices sends messages with the same state but with a diffrent text, in the history you’ll see only one message with the latest text of the message. If you have to analyse this (e.g. after the night) this is a big problem if you lost one or more error messages. Think about SLA in this case.

We have two ideas for a workarround:

  • Let logstash write an extra log file (this is what we do now and if a colleague ask I take a look on it)
  • the script or logstash have to send fake message before with another state (e.g. ok) and then send the real message. Now it is a state change for icinga and you will see every passive check result in the history. Not really nice, but should work.

thank you

This sound similar to my question (which does not have any suggestions yet, unfortunately).

And I give you there an answer :grin: - Forgot your questions. It’s really similar

That’s mainly because the state history actually is a state change history. Not every check result is recorded here.

I’d recommend to collect such events in Elasticsearch / Loki / etc. and use that for problem analysis and correlation then.

Cheers,
Michael

@dnsmichi thanks for this.
So that means the state change concern only if the sate field itself is changing (CRITICAL -> OK) and not in combination with the message / plugin output. Or in other words if the state is the same from the privious and the new message, but the Output is changed nothing happens in the history.

That leaves only to provocate a state change for the icinga history, write a log file and/or ship the message to elastic.