I am setting up a new icinga2 node, and it was working, then a bunch of agents connected and the service seems to crash
Now icingaweb is “icinga is currently not up and running” and icinga2 service is not writting to the database
select status_update_time from icinga_programstatus;
Empty set (0.00 sec)
I tried dropping the icinga2 database, and re creating it, and then repopulating from schema
/usr/share/icinga2-ido-mysql/schema/mysql.sql
and then restarting the service, but no luck.
the icinga2 service is up and running
As i was writting this ticket, 20 min after I rebuilt the db. It suddenly came back to life.
Any help or ideas would be appreciated. since I am worried it may happen again.
Version used (icinga2 --version) r2.12.4-1
Operating System and version : Ubuntu 20.04.2 LTS (Focal Fossa)
Enabled features (icinga2 feature list) api checker debuglog ido-mysql influxdb mainlog notification
Icinga Web 2 version and modules (System - About)
|setup|2.8.2|
|grafana|1.4.2|
|monitoring|2.8.2|
Config validation (icinga2 daemon -C)
valid
If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes
The only enlighteniing things in the icinag2.log file is hosts that cannot connect (because they need their zones and certs updated, something that is in progress)
and the fact that the DB re connection / resuming seems quite slow.
more testing and messing with things and it seems that eventually icinga2 catches up and is happy. but every time i reload the service it gets stuck reloading. and then if i dont notice right away, and restart the service to fix the issue it takes 45min for icinga to catch back up and the cpu load to return to normal levels.
It might help if you share your configuration with us - if you have a lof of hosts and a lot of apply rules that refence the host objects like assign where host.name == "foo" the reload time could be bloated because of that.