Influxdbwriter - include tag only if variable exists

Icinga 2
1

We are using icinga2 to monitor a mix of network devices and servers.
We are also using the InfluxdbWriter to send perfdata to influxdb, we have a new requirement to send an additional tag to influxdb but only for the network devices that we monitor.

Our config looks like this right now:

object InfluxdbWriter "influxdb" {
  host = "influxserver.com"
  port = 8086
  database = "icinga"
  username = "grafana"
  password = "passwordhere"
  host_template = {
    measurement = "$host.check_command$"
    tags = {
      hostname = "$host.name$"
    }
  }
  service_template = {
    measurement = "$service.check_command$"
    tags = {
      hostname = "$host.name$"
      service = "$service.name$"
      instance = "$service.vars.instance$"
    }
  }

And this is working fine, I would like to add a new tag to both the host and the service that will look something like this:

area = "$host.vars.area$"

Where $host.vars.area$ is a custom variable that only exists for our network devices.

I’ve tried doing this and it works fine for the network devices but for other hosts where the variable doesn’t exist it literally writes “$host.vars.area$” into the tag in influx.
From reading the docs it sounded to me like if the host doesn’t have this variable it should be ignored but either I misunderstood or there is a bug?

Does anyone know how to achieve what I’m looking for here? Can we have 2 InfluxdbWriter objects and apply them based on host or service groups?

Cheers
Martin

2

Hi,

a short peek into the code tells me that missing macros are not set into the tags structure. The code sits there since 2017, so I’d be interested in the debug log lines where the metrics are built and sent to InfluxDB.

Never seen that Icinga would leave the macro string “as is” in there, this was a common pitfall with 1.x command lines where Icinga 2 is stronger in ignoring or warning about this.

Cheers,
Michael

3

Hi Michael, thanks for the reply.
If I understood you correctly then I should be able to include that extra line in the tags section and if the variable doesn’t exist for that host it will be ignored?

I enabled debug logs, trying to find anything that may be relevant in there:

I see lines like this quite frequently:

[2019-07-19 09:37:07 +0000] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 1, rate: 19.1667/s (1150/min 5378/5min 5385/15min);
[2019-07-19 09:37:07 +0000] debug/InfluxdbWriter: Timer expired writing 120 data points
[2019-07-19 09:37:07 +0000] notice/InfluxdbWriter: Reconnecting to InfluxDB on host '10.123.80.66' port '8086'.

and a few of these:

[2019-07-19 09:35:46 +0000] warning/InfluxdbWriter: Ignoring invalid perfdata value: 'replication     max'=NaNs;3;5
Context:
        (0) Processing check result for 'servername.net!pgactivity_longest_query'

But I don’t think either help us here
Is there something else that I should be looking out for?

Just for reference we are running version: r2.10.5-1

4

Hi,

I’m looking for lines like Checkable 'bla' adds to metric list:. Right after, the full string is dumped which is written to InfluxDB’s API. But as far as I can see, this only is available in git master for 2.11, and hasn’t been backported into 2.10.x.

So, our best guess would be looking onto the InfluxDB side if something like query logging is available. I don’t know much about InfluxDB which could help troubleshoot unfortunately.

Cheers,
Michael

5

I’ve just tried enabling more debug logs in influx but we don;t really see anything useful there either unfortunately, just lots of lines like this:

Jul 19 12:59:55 influxdb-data0 influxd: [httpd] 10.123.80.66 - icinga [19/Jul/2019:11:59:55 +0000] "POST /write?db=icinga&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.1" 204 0 "-" "Icinga/r2.10.5-1"

Just, had a little look at the icinga2 github and spotted this pull request - https://github.com/Icinga/icinga2/pull/7226 - I may be completely off here but would this be a potential solution to my problem?

Cheers
Martin

6

Hi,

no, that PR talks about adding fields next to tags and just duplicates the code logic from the latter to the first. I’m undecided whether the PR makes sense or not, especially with performance in mind. Each call to the MacroResolver methods is extremely expensive.

It definitely doesn’t solve your problem, since you only got tags as config attribute inside the templates.

One thing I can thing of - you forgot the closing $ sign and that got written to InfluxDB. Do you have proof somewhere for this string? E.g. in a test environment.

Cheers,
Michael

7

Fair enough, thanks for checking though :slight_smile:

Which closing $ sign are you referring to?

This has all been done in our staging environment so far… here’s an example of some raw data from influxdb (in this example the variable is L3Area and not just area):

> select * from snmp  order by time desc limit 100
name: snmp
time                hostname                                  instance                layer3area         metric                      service                     unit    value
----                --------                                  --------                ----------         ------                      -------                     ----    -----
1563544117000000000 fan1.staging.dc2                          $service.vars.instance$ Staging Area       5min                        Cisco-ASR900-CPU            percent 14
1563544117000000000 fan1.staging.dc2                          $service.vars.instance$ Staging Area       1min                        Cisco-ASR900-CPU            percent 11
1563544117000000000 fan1.staging.dc2                          $service.vars.instance$ Staging Area       15min                       Cisco-ASR900-CPU            percent 14
1563544116000000000 activationcheck.ab2.staging..net          $service.vars.instance$ $host.vars.L3Area$ iso.3.6.1.4.1.2021.9.1.9.1  disk-space-used                     13
1563544114000000000 icinga-master1.ab2.staging..net           $service.vars.instance$ $host.vars.L3Area$ iso.3.6.1.4.1.2021.9.1.9.1  disk-space-used                     29

You can see it works for the first few entries (this host has the variable in icinga).
The next 2 lines show the problem (they don’t have the variable in icinga)

The $service.vars.instance$ is another example of the same problem.

Here’s a little extract from the icinga config file for the host object:

object Host "fan1.staging.dc2" {
  import "Router-FAN"
  address = "10.124.2.6"
  display_name = "fan1.staging.dc2"
  vars.L3Area = "Staging Area"
  vars.cabinetName = "dc2"
  vars.geolocation = "0, 0"
  ....

Here’s a snippet of the raw config form my features-enabled/influx file

  host_template = {
    measurement = "$host.check_command$"
    tags = {
      hostname = "$host.name$"
      layer3area = "$host.vars.L3Area$"
    }
  }
  service_template = {
    measurement = "$service.check_command$"
    tags = {
      hostname = "$host.name$"
      service = "$service.name$"
      instance = "$service.vars.instance$"
      layer3area = "$host.vars.L3Area$"
    }
  }

cheers
Martin

8

Ok, that smells like a bug. Can you collect the details from this topic and move this into a GitHub issue please?

9

Done - https://github.com/Icinga/icinga2/issues/7341

Thanks for your time and have a good weekend!