Enable debug for icingadb to troubleshoot redis heartbeat issue

Icinga 2 Icinga DB
1

After configuring TLS for redis I am noticing that icingadb is dropping the redis heartbeat. It stays up but fails,

Aug 27 18:06:37 servicename.obfuscate.org icingadb[142343]: Connecting to Redis at 'servicename.obfuscate.org:6379'
Aug 27 18:06:37 servicename.obfuscate.org icingadb[142343]: Starting history sync
Aug 27 18:07:37 servicename.obfuscate.org icingadb[142343]: heartbeat: Waiting for Icinga heartbeat
Aug 27 18:08:37 servicename.obfuscate.org icingadb[142343]: heartbeat: Waiting for Icinga heartbeat
Aug 27 18:09:37 servicename.obfuscate.org icingadb[142343]: heartbeat: Waiting for Icinga heartbeat
Aug 27 18:10:37 servicename.obfuscate.org icingadb[142343]: heartbeat: Waiting for Icinga heartbeat
Aug 27 18:11:37 servicename.obfuscate.org icingadb[142343]: retry deadline exceeded
                                                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:163
                                                                   runtime.goexit
                                                                           runtime/asm_amd64.s:1695
                                                                   HA aborted
                                                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).abort.func1
                                                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:131
                                                                   sync.(*Once).doSlow
                                                                           sync/once.go:74
                                                                   sync.(*Once).Do
                                                                           sync/once.go:65
                                                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).abort
                                                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:129
                                                                   github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
                                                                           github.com/icinga/icingadb/pkg/icingadb/ha.go:163
                                                                   runtime.goexit
                                                                           runtime/asm_amd64.s:1695
                                                                   HA exited with an error
                                                                   main.run
                                                                           github.com/icinga/icingadb/cmd/icingadb/main.go:335
                                                                   main.main
                                                                           github.com/icinga/icingadb/cmd/icingadb/main.go:37
                                                                   runtime.main
                                                                           runtime/proc.go:271
                                                                   runtime.goexit
                                                                           runtime/asm_amd64.s:1695

I tried to enabling debuging in /etc/icingadb/config.yml

#logging:
  # Default logging level. Can be set to 'fatal', 'error', 'warn', 'info' or 'debug'.
  # If not set, defaults to 'info'.
  #level: info
  level: debug

then when I restart icingadb

Aug 27 18:23:58 servicename.obfuscate.org icingadb[7961]:   85 |   #level: info
Aug 27 18:23:58 servicename.obfuscate.org icingadb[7961]: > 86 |   level: debug
Aug 27 18:23:58 servicename.obfuscate.org icingadb[7961]:          ^
Aug 27 18:23:58 servicename.obfuscate.org icingadb[7961]:   87 |   # Logging output. Can be set to 'console' (stderr) or 'systemd-journald'.
Aug 27 18:23:58 servicename.obfuscate.org icingadb[7961]:   88 |   # If not set, logs to systemd-journald when running under systemd, otherwise stderr.
Aug 27 18:23:58 servicename.obfuscate.org icingadb[7961]:   89 | #  output:
Aug 27 18:23:58 servicename.obfuscate.org icingadb[7961]:   90 |
Aug 27 18:23:58 servicename.obfuscate.org systemd[1]: icingadb.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 27 18:23:58 servicename.obfuscate.org systemd[1]: icingadb.service: Failed with result 'exit-code'.
Aug 27 18:23:58 servicename.obfuscate.org systemd[1]: Failed to start Icinga DB.

icingadb-redis-server --version
Redis server v=7.0.15 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=d570a2c0475ce571

Icinga DB version: v1.2.0

Build information:
Go version: go1.22.2 (linux, amd64)
Git commit: a0a65af0260b9821e4d72692b9c8fda545b6aeca

System information:
Platform: Red Hat Enterprise Linux
Platform version: 9.4 (Plow)

2

got logging working, its yaml so spacing is important:

logging:
  # Default logging level. Can be set to 'fatal', 'error', 'warn', 'info' or 'debug'.
  # If not set, defaults to 'info'.
  level: debug

This fixed it

3

As you have just reconfigured your Redis, you have to reflect this configuration in three places:

This message indicates an absence of Icinga 2’s heartbeat, being provided via Redis. Thus, I would suggest checking if Icinga 2 works with your reconfigured Redis. If you have already changed the Icinga 2 IcingaDB object and it still does not work, please consider the log files.

1 Like
4

@apenning Thank you so much, using the link provided https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#icingadb I was able to resolve the issue. Again, thank you so much!!