No performance data in Influxdb

Icinga 2
,
1

Hello,

I’m quite sure there is somewhere the answer for my question but cannot find it.
The situation - trying to collect performance data to Influxdb but don’t see any there.

OS: CentOS 8.1
Icinga version: 2.11.4-1
Influxdb version: 1.8.0

icinga2 feature list
Disabled features: compatlog debuglog elasticsearch gelf graphite opentsdb statusdata syslog
Enabled features: api checker command ido-mysql influxdb livestatus mainlog notification perfdata

Config /etc/icinga2/features-enabled/influxdb.conf

library "perfdata"
object InfluxdbWriter "influxdb" {
  host = "127.0.0.1"
  port = 8086
  database = "icinga2"
  username = "icinga2"
  password = "icinga2"
  host_template = {
    measurement = "$host.check_command$"
    tags = {
        hostname = "$host.name$"
    }
  }
  service_template = {
    measurement = "$service.check_command$"
    tags = {
      hostname = "$host.name$"
      service = "$service.name$"
    }
  }
}

Influxdb

> show users
user          admin
icinga2       false

> show grants for icinga2
database privilege
icinga2  ALL PRIVILEGES

> show databases
name: databases
name
_internal
icinga2

> show measurements

and here is nothing.

Also, selinux package is installed:

yum install icinga2-selinux

ps -eZ | grep icinga2
system_u:system_r:icinga2_t:s0  10782 ?        00:00:00 icinga2
system_u:system_r:icinga2_t:s0  10797 ?        00:00:03 icinga2
system_u:system_r:icinga2_t:s0  10798 ?        00:00:14 icinga2

Don’t know why there is 3 processes.

setsebool -P icinga2_can_connect_all true (need to allow Icinga to connect to all ports)

I can see performance data in IcingaWeb2 interface.
Where can I check else why it’s not saving data to Influxdb?

2

Icinga’s logfile will tell you.There you should find sometime like:

[2020-07-10 06:30:32 +0200] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 1, rate: 1.38333/s (83/min 426/5min 1280/15min);

or an error message.

3

Indeed, there is the message:

[2020-07-10 13:42:33 +0300] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 0, rate: 4.08333/s (245/min 1128/5min 3380/15min)

getting it every 5 min
what should I get from this message?

4

It tells you how many items are sent to influxdb. Maybe you should enable debuglog to get more detailed information.

5

Got it in debuglog (last entry), nothing about Influxdb

 [2020-07-10 15:32:04 +0300] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0',  active_checks_enabled = '1',  check_command = 'check_apc_ups',  check_source = 'Icinga.local',  check_type = '0',  current_check_attempt = '1',  current_notification_number = '0',  current_state = '0',  endpoint_object_id = 289,  event_handler_enabled = '1',  execution_time = '0.455354',  flap_detection_enabled = '0',  has_been_checked = '1',  instance_id = 1,  is_flapping = '0',  is_reachable = '1',  last_check = FROM_UNIXTIME(1594384324),  last_hard_state = '0',  last_hard_state_change = FROM_UNIXTIME(1583410950),  last_state_change = FROM_UNIXTIME(1583410950),  last_time_ok = FROM_UNIXTIME(1594384324),  latency = '0.000385',  long_output = '',  max_check_attempts = '5',  next_check = FROM_UNIXTIME(1594384383),  next_notification = FROM_UNIXTIME(1594385727),  normal_check_interval = '1',  notifications_enabled = '1',  original_attributes = 'null',  output = 'OK: Battery Rack doesn\'t need a replacement',  passive_checks_enabled = '1',  percent_state_change = '0',  perfdata = '',  problem_has_been_acknowledged = '0',  process_performance_data = '1',  retry_check_interval = '0.500000',  scheduled_downtime_depth = '0',  service_object_id = 888,  should_be_scheduled = '1',  state_type = '1',  status_update_time = FROM_UNIXTIME(1594384324

The only entry with influxdb (sorry for long log):

[2020-07-10 15:38:16 +0300] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0',  active_checks_enabled = '1',  check_command = 'icinga',  check_source = 'Icinga.local',  check_type = '0',  current_check_attempt = '1',  current_notification_number = '0',  current_state = '0',  endpoint_object_id = 289,  event_handler_enabled = '1',  execution_time = '0.001407',  flap_detection_enabled = '0',  has_been_checked = '1',  instance_id = 1,  is_flapping = '0',  is_reachable = '1',  last_check = FROM_UNIXTIME(1594384696),  last_hard_state = '0',  last_hard_state_change = FROM_UNIXTIME(1581600003),  last_state_change = FROM_UNIXTIME(1581600003),  last_time_ok = FROM_UNIXTIME(1594384696),  latency = '0.000138',  long_output = '',  max_check_attempts = '5',  next_check = FROM_UNIXTIME(1594384753),  next_notification = FROM_UNIXTIME(1594385727),  normal_check_interval = '1',  notifications_enabled = '1',  original_attributes = 'null',  output = 'Icinga 2 has been running for 17 minutes and 18 seconds. Version: 2.11.4-1',  passive_checks_enabled = '1',  percent_state_change = '0',  perfdata = 'api_num_conn_endpoints=19 api_num_endpoints=20 api_num_http_clients=0 api_num_json_rpc_anonymous_clients=1 api_num_json_rpc_relay_queue_item_rate=7.350000 api_num_json_rpc_relay_queue_items=1 api_num_json_rpc_sync_queue_item_rate=0 api_num_json_rpc_sync_queue_items=0 api_num_json_rpc_work_queue_item_rate=2.850000 api_num_not_conn_endpoints=1 checkercomponent_checker_idle=564 checkercomponent_checker_pending=1 idomysqlconnection_ido-mysql_queries_rate=5.216667 idomysqlconnection_ido-mysql_queries_1min=313 idomysqlconnection_ido-mysql_queries_5mins=1955 idomysqlconnection_ido-mysql_queries_15mins=5834 idomysqlconnection_ido-mysql_query_queue_items=0 idomysqlconnection_ido-mysql_query_queue_item_rate=7.566667 **influxdbwriter_influxdb_work_queue_items=0** **influxdbwriter_influxdb_work_queue_item_rate=2.533333 influxdbwriter_influxdb_data_queue_items=0** livestatuslistener_livestatus_connections=0 active_host_checks=0.033333 passive_host_checks=0 active_host_checks_1min=2 passive_host_checks_1min=0 active_host_checks_5min=73 passive_host_checks_5min=0 active_host_checks_15min=216 passive_host_checks_15min=0 active_service_checks=2.483333 passive_service_checks=0 active_service_checks_1min=149 passive_service_checks_1min=0 active_service_checks_5min=1020 passive_service_checks_5min=0 active_service_checks_15min=3075 passive_service_checks_15min=0 current_pending_callbacks=0 current_concurrent_checks=6 remote_check_queue=0 min_latency=0 max_latency=0.020107 avg_latency=0.000907 min_execution_time=0.000119 max_execution_time=14.790000 avg_execution_time=1.492752 num_services_ok=451 num_services_warning=5 num_services_critical=29 num_services_unknown=10 num_services_pending=0 num_services_unreachable=29 num_services_flapping=0 num_services_in_downtime=0 num_services_acknowledged=19 num_services_handled=48 num_services_problem=44 uptime=1038.828546 num_hosts_up=63 num_hosts_down=7 num_hosts_pending=0 num_hosts_unreachable=0 num_hosts_flapping=0 num_hosts_in_downtime=0 num_hosts_acknowledged=2 num_hosts_handled=2 num_hosts_problem=7 last_messages_sent=1594384695.744403 last_messages_received=1594384696.314402 sum_messages_sent_per_second=2.250000 sum_messages_received_per_second=2.750000 sum_bytes_sent_per_second=182.050000 sum_bytes_received_per_second=553.233333',  problem_has_been_acknowledged = '0',  process_performance_data = '1',  retry_check_interval = '0.500000',  scheduled_downtime_depth = '0',  service_object_id = 296,  should_be_scheduled = '1',  state_type = '1',  status_update_time = FROM_UNIXTIME(1594384696) WHERE service_object_id = 296