State history is not updated when plugin output changes without a state change Show more lines

Icinga 2
1

Hello,

While testing I noticed a behavior that makes it difficult to display the latest plugin output for active problems.

I have a service called “Deneme1”.

When the service changes state from OK to CRITICAL, a record is written into the state_history table and the plugin output is stored correctly.

Example:

State: CRITICAL
Plugin Output: Deneme 12

I can see this value both in Icinga Web and in the icingadb.state_history table.

However, if the service remains in CRITICAL state and only the plugin output changes, the state_history table is not updated.

Example:

Initial state:
CRITICAL
Plugin Output: Deneme 12

Later:
CRITICAL
Plugin Output: Deneme 123

In Icinga Web I can see the updated plugin output (Deneme 123), but no new record is written into icingadb.state_history and the existing state_history record still contains “Deneme 12”.

As a result, external reporting tools that rely on state_history cannot display the latest plugin output for ongoing problems.

Is this the expected behavior?

Would it be possible to make plugin output changes available in history even if the state itself does not change?

For example:

  • create a new state_history entry when output changes
    or
  • update the latest history record with the current output

This would make it easier to build reporting and dashboard integrations that need the latest output of an active problem.

Thanks.

image
image857×542 19.4 KB

image
image861×447 60.8 KB

2

"the reason for this is, that the most recent non hard state change information is stored in the redis server only.

The sql database hold the hard state relevant for “reboot consistant” sla calculation."

So an OK following an OK with a different output is not always in the database
and a CRITICAL following a CRITICAL with a different output is not always in the database too
I have written it down here:

3

While @moreamazingnick already explained why this is the case, I would urge you not to query the relational database from external tools: the schema might change, things might break, or external calls might introduce database locking. If you want to query service state information, please use the Icinga 2 API: Icinga2 Api - Icinga 2

4

If it helps you could use icingadb-web api to query hosts:
username demo
password demo

https://demo.icinga.com/icingadb/services?service.name~File%2A&format=json

create a filter or not, and copy link address of the export->json
works for hosts services and so on.

Bildschirmfoto 2026-09-08 um 15.52.59
Bildschirmfoto 2026-09-08 um 15.52.59786×542 68 KB

and it is machine readable json. just use basic auth on your script.
you can also copy the query url directly and just set the header in your script to

               'headers' => [
                    'Accept' => 'application/json',
                    'Content-Type' => 'application/json',
                ],

with that you wont need &format=json

5

Thanks everyone for the explanations.

Now it makes sense.

I was assuming that state_history would also contain plugin output changes while a service remains in the same state, but after testing and reviewing your explanations I understand that only hard state relevant information is persisted to the SQL database, while the latest volatile state information is kept in Redis.

This explains why I can see updated plugin output in Icinga Web, but not in state_history or service_state.

Our use case is to display the latest plugin output for active problems in an external dashboard. Since this information is not guaranteed to be available in the SQL database, we will most likely move to the Icinga 2 API (or IcingaDB Web JSON endpoints) instead of relying on state_history.

Thank you for pointing me in the right direction and for the additional examples.:folded_hands: