Unable to delete nodes in icinga2

prime69 · March 3, 2024, 7:56pm

hi Community

can anyone please direct me to information on how to remove hosts and services from icinga and netbox integration…

as when i delete it from director it automatically recreates again and is on my dashboard…

Thanks

Before you ask a question, you can check the troubleshooting documentation first, maybe you can find an answer here.

Please describe your problem as detailed as possible and don’t forget to use a meaningful title
We also have a markdown formatting guide to help you make your topics more readable!

Give as much information as you can, e.g.

Version used (icinga2 --version)
Operating System and version
Enabled features (icinga2 feature list)
Icinga Web 2 version and modules (System - About)
Config validation (icinga2 daemon -C)
If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes

log1c · March 4, 2024, 12:44pm

I assume, that you made a config deployment after you deleted the host from the Director?

If so:
Sounds like you have an automated import&sync as well as a config deployment in place.
Check the Directors “Automation” section, I guess there you will find an import source for the netbox, a sync rule for creating Director objects as well as a job that automates all that. Disabling the job that automates the import&sync will stop the recreation. BUT it would also disabled the creation of new hosts. Why not delete the host in the netbox?

prime69 · March 4, 2024, 11:44pm

thanks for the insight @log1c

yes we try that earlier but the graphite still displayed the hosts that were deleted from netbox GUI end, somehow its not updating the graphite dashboard…

will have to check whether updates in Netbox it sync with icinga graphite display

is there any file snippet you would like to view that i could put online to help troubleshoot this issue

log1c · March 5, 2024, 8:24am

Hold on a second.
Are we talking about what you see in the Icinga webinterface or the graphite webinterface?

The Icinga webinterface will display the current running config of your Icinga2 monitoring. If you delete something from the Icinga Director and don’t deploy the config afterwards, the webinterface will still show the host.

Graphite is completely separate to the Icinga webinterface, though you can integrate it’s graphs via the graphite module.
Graphite is only for visualising the performance data. And the performance data of a host does not get deleted when you delete it from the Icinga config. Thus the graphite webinterface will still show that hosts performance data. If you don’t want it showing up there anymore you need to delete the graphite perfdata files from your disk.

prime69 · March 5, 2024, 4:57pm

Thanks @log1c that makes sense

so is there a in-built script in icinga2 that cleans out the perfdata of hosts that are removed by the director

it would make practical sense to have an inbuilt script for house keeping every time
a host is removed from icinga2 its performance data is also removed from icinga2

so my requirement is basically house keeping clear out legacy hosts and their performance data as it does not reflect the current environment …

too many reds gives the impression of a chaos environment with everything failing which is not a true reflection of the current environment

so from the insight i gather from your knowledge

i have to manually clear out the performance data of hosts that have been deleted…

is there any automated script…in-built into icinga2

or do i have to manually go into the database on icinga2 and remove the perfdata of hosts
is there a best-practice for such a scenario

Thanks
john

log1c · March 7, 2024, 7:17am

no, there is no such mechnism

the perfdata isn’t stored in the database. when you use graphite for perfdata collection and display you will have a path like /var/lib/graphite/whisper/icinga2 (e.g.) where there is a folder for each host.

but why?
you don’t see the perfdata of old hosts, as they are not monitored anymore. Or do you have someone “spying” through the server or the graphite webinterface explicitly looking for legacy hosts?

I don’t know if graphite has some built-in mechanism to clean old data.
Manually you can do that by simply deleting the folders on disk of hosts that don’t exist anymore. For example with a cronjob running the find command on the graphite perfdata folder and deleting everything that hasn’t been updated in x days.

prime69 · March 7, 2024, 11:04pm

Thanks @log1c for your insight and advise

my apologize i should have presented below earlier in the our discussion, as it would get down to identifying the actual symptoms

please note i dont have graphite but grafana 1.3.4 , when i followed your recent advise i could not find any grafana or graphite folder under /var/lib/

i have already deleted 75 hosts in net box but on the Grafana dashboard the hosts are still showing as down 99 host but it should be (99 - 75) = 24 valid down hosts

and on our default map its still this is how it looks which is not an true indicator of our current environment, the visual display dashboard is incorrect from the existing hosts in icinga/netbox setup

when i try and delete hosts from icinga director for the hosts i deleted in netbox below error when i deploy changes… and its throwing this exception

we should have more green instead of all the red dots below its only when one zooms in does it show more green hosts but i should not have too zoom in

this does not reflect our current true operational environment and its displayed in our noc which from a laymen perspective shows all our systems are down…

thanking you in advance for your assistance

log1c · March 8, 2024, 8:16am

This is not the Grafana dashboard. This is the Icinga Web 2 webinterface.
So your problem really has nothing to do with anything performance data related which is what graphite or grafana and its icingaweb2-modules are for. So just ignore anything related to those.

Your actual problem is, that you can’t deploy your configuration.

Please share your /etc/icinga2/zones.conf file from your master and a screenshot of .../director/dashboard?name=infrastructure#!/director/kickstart

For the error message of your Import Source: make sure the icinga server can actually reach and authenticate against the netbox api. Click on modify after chosing the Import Source and check the data for correctness.

For the error in your Sync Rule: The syncrule has the Purge option enabled to delete host objects from the Director if they are no longer available in the Netbox. The error it is complaining about is that it is trying to delete a host object that is currently configured as a dependency parent host (docs about dependencies: Object Types - Icinga 2). That’s why it currently can’t be deleted. You will need to either disable the whole depedency configuration or remove the “to-be-deleted” host from the other hosts dependency variable.

PS: Am I correct in assuming that you inherited this system?
Because it is really old and outdated.
Just for comparison:
Icinga Director current version is 1.11.1
Icinga Web 2 (monitoring module) current version is 2.12.1
I’m guessing that the icinga2 core version (cli: icinga2 --version) is somewhere around 2.8 or 2.9? Current version is 2.14.2

prime69 · March 9, 2024, 1:15pm

hi @log1c

thanks for narrowing root cause down

please see zone file content below

/
*
 * Generated by Icinga 2 node setup commands
 * on 2019-03-26 01:50:36 +0000
 */


object Endpoint "icinga.xxx.xx.xx" {
}

object Zone "master" {
        endpoints = [ "icinga.xxx.xx.xx" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

the import source looks valid as i can ping the netbox url, access the netbox api via web browser and the api tokens are still valid

as for the sync rule i have enable the purge but still cant deploy changes…what would be the best way to disable the whole dependency, without many changes…

Am I correct in assuming that you inherited this system? yes when i started at this organization
Because it is really old and outdated. indeed working on a better maintenance strategy to get ahead of the delayed upgrades
Just for comparison:
Icinga Director current version is 1.11.1
Icinga Web 2 (monitoring module) current version is 2.12.1
I’m guessing that the icinga2 core version (cli: icinga2 --version) is somewhere around 2.8 or 2.9? Current version is 2.14.2

the version is 2.10.3 which is really old…thank you in advance for your assistance…

log1c · March 12, 2024, 9:55am

check if you have configured Dependencies inside the Director (.../director#!/director/dependencies/applyrules). If yes, disable the config.
If no you will have to search the config under /etc/icinga2/[zones.d|conf.d]

Config and kickstart screenshot looks good.

But it still looks like there is a connection problem between the webserver/director and the icinga2 api.
Is the icinga api user “director_api” still valid?
Check where the user is configured (zones.d/conf.d directory) and if that config is actually imported. The conf.d directory might not be included, check /etc/icinga2/icinga2.conf
Is the icinga2 service running without problems?
Is port 5665 for the icinga2 api listening on the host (use netstat/ss)?

prime69 · March 15, 2024, 4:59am

hi @log1c

sorry for delay please note i am unable to check any of the requested directory as shown below

as when i try and open the files it opens a new file instead of the existing file…and for directory it says no such file or directory…

appreciate your assistance…

Thanking you in advance

drwxr-x— 2 nagios nagios 4096 Mar 7 09:58 conf.d
-rw-r–r-- 1 nagios nagios 1045 Mar 1 09:10 constants.conf
-rw-r–r-- 1 root root 998 Mar 26 2019 constants.conf.orig
drwxr-x— 2 nagios nagios 4096 Oct 9 15:24 features-available
drwxr-x— 2 nagios nagios 4096 Sep 19 11:59 features-enabled
-rw-r–r-- 1 nagios nagios 2040 Mar 1 09:10 icinga2.conf
-rw-r–r-- 1 root root 1721 Mar 26 2019 icinga2.conf.orig
drwx------ 2 nagios nagios 4096 Feb 27 2019 pki
drwxr-xr-x 4 root root 4096 Jul 28 2023 scripts
-rw-r----- 1 nagios nagios 289 Mar 26 2019 zones.conf
-rw-r–r-- 1 root root 1334 Mar 26 2019 zones.conf.orig
drwxr-x— 2 nagios nagios 4096 Feb 27 14:17 zones.d

icingaweb2:
total 40
-rw-rw---- 1 www-data icingaweb2 195 Apr 10 2019 authentication.ini
-rw-rw---- 1 www-data icingaweb2 287 Feb 11 2022 config.ini
drwxrws— 9 www-data icingaweb2 4096 Mar 12 2020 dashboards
drwxrws— 2 www-data icingaweb2 4096 Mar 8 12:09 enabledModules
-rw-rw---- 1 www-data icingaweb2 196 Jul 16 2019 groups.ini
-rw-r----- 1 root icingaweb2 958 Mar 6 10:55 icingatestresources.ini
drwxrws— 9 root icingaweb2 4096 Sep 21 12:33 modules
drwxrws— 3 www-data icingaweb2 4096 Nov 10 2020 preferences
-rw-rw---- 1 www-data icingaweb2 958 Jul 22 2020 resources.ini
-rw-rw---- 1 www-data icingaweb2 2567 Mar 6 16:18 roles.ini

log1c · March 19, 2024, 7:19am

Why? They are all there?!
/etc/icinga2/icinga2.conf → is the conf.d directory included recursively?
if yes, check /etc/icinga2/conf.d/api-users.conf
Also check folders and config files in /etc/icinga2/zones.d/...

prime69 · April 23, 2024, 11:19pm

Thank you @log1c for all your assistance and support…after taking a step back and evaluating my progress over months, my colleague have decided to just built from scratch…and using existing legacy icinga system as a baseline, have managed to get icinga2 to the latest versions running off ubuntu 22.04 and icinga2 version 2.12

Good learning experience, when there is no system documentation and personnel involved with installation with vendor have departed…just do a fresh installation

especially when time restrictions is a factor

God Bless @log1c

Regards