IDO does not log eventhandlers for clients

Icinga 2
1

Not sure if this is a bug, a limitation or just not a feature, but I noticed something trying to get some historical data out of IDO today.

The icinga_eventhandlers table only shows event handlers that ran on the masters, not the clients. For example, that event_index-lock-remover handler runs quite often.

icinga=# select object_id,name1 from icinga_objects where name1 like('event_%') order by object_id;
 object_id |              name1              
-----------+---------------------------------
       411 | event_test_ido_check
       412 | event_index-lock-remover
      2704 | event_ido-failover
      2705 | event_test_output
      2706 | event_flagdestroyer
     24754 | event_ntpd-starter
     24755 | event_unix-process-handler
     24757 | event_linux-log-cleanup-handler
     48580 | event_catalina-log-nixer
     55205 | event_solaris-svc-restart
     55206 | event_event_ssh
     55963 | event_test-event
     56200 | event_array-index-restart
     56831 | event_average-request-count
     57140 | event_filenotfound-announce
     57493 | event_pagerduty-api
     59475 | event_dangerous-service-restart
     59778 | event_[redacted]
     63491 | event_test-downtime-event
     67302 | event_usernotfound-announce
     72068 | event_cf-log-handler
     78803 | event_mailflood-flag-destroyer
     80611 | event_service-macros
     80613 | event_rabbitmq_event_fetch
     80707 | event_hostalive-announce
(25 rows)

icinga=# select distinct command_object_id from icinga_eventhandlers order by command_object_id;
 command_object_id 
-------------------
              2705
              2706
             24754
             55205
             55963
             56831
             57140
             59475
             78803
             80613
             80707
(11 rows)

So event handlers specific to client nodes have never been recorded here. Presently looking around for another way to scrape this data and will likely throw a little note here for future observers, but thought to run it by here before putting in something on github. Is this expected? Thanks!

2

So in trying to see how often an event command ran in a quarter, I recreated the circumstances under which they would have run. This could serve as a decent work around for anyone else trying to do this. All my examples are using postgres.

icinga=# select COUNT(*) from icinga_statehistory where output like('%INDEX IS LOCKED%') and state_time >= '2019-10-01' and state_time < '2020-01-01' and current_check_attempt = '1' and state_type = '0' and state = '2';
 count 
-------
   166
(1 row)
3

Okay, I actually see @dnsmichi described this in a bug I submitted about something else with event handlers here: https://github.com/Icinga/icinga2/issues/7141

This topic was described as on purpose, so I’ll close this, but leave it up as I didn’t see anything on the forum specifically about it.

Event handlers are called whenever a cluster instance processes a check result locally. There is no synchronisation for additional event handler history logging in place, i.e. you won’t see satellite or client event handler calls in the master’s history log in DB IDO or the API events. That’s on purpose as this generates huge network traffic overload - typically unneeded. Neither does the DB IDO feature populate the database table by default, as this information isn’t needed in Icinga Web 2.