Hello,
I have an Influx database (1.7, not enterprise) with metrics pushed to it and stored there.
I’d like to configure Icinga 2 (2.9 on Debian 9, possible to upgrade if required) notifications/alerting based on these metrics. For example, if metric A is above 100, raise critical condition for a certain Icinga 2 service tied to that metric.
Given that I’d have to evaluate not less than 4k metrics per minute (potentially more), what would be the best ways to do that?
- First way I thought of is querying Influx with checks, but it seems quite wasteful and plain wrong.
- Another option I thought of is setting up Kapacitor and somehow tying Kapacitor alerts to Icinga 2 services. Got zero ideas about how to do that though. Kapacitor seems to have SNMP trap event handler and able to post JSON encoded data to an HTTP endpoint. Using SNMP traps for that seems quite weird, so if Kapacitor with post is the answer, what would be starting point to building alerting on input JSON?
If both of mentioned solutions aren’t good, what else could be done to get that alerting made in the optimal way?