How icinga determines the API identity?

Icinga 2 Core
#1

Hello.

For some reason, I can’t provide more than 2 links in the post. In my opinion, it makes it more difficult to understand, but it is not up to me to decide. So below I use the following aliases:
wrong-link: example.xaas.organisation.com
correct-link: example.organisation.com

I am trying to configure icinga2 to monitor a couple of machines. The icinga2-machine is behind a firewall of our organization, while the monitored machines are outside of it.

When running icinga2 daemon -C I get the following error:

...
[2021-01-11 16:12:46 +0100] information/ApiListener: My API identity: wrong-link
[2021-01-11 16:12:46 +0100] critical/config: Error: Endpoint object for 'wrong-link' is missing.
Location: in /etc/icinga2/features-enabled/api.conf: 4:1-4:24
...

The problem here is that the right DNS name of our icinga2-machine is ‘correct-link’. The DNS server of our organization only provides the ‘wrong-link’, which isn’t visible outside of our local network.

I am not entirely sure whether I understood everything correctly, but it appears to me that if I convince Icinga2 to use ‘correct-link’ as “My API identity”, the problem might be resolved. But I didn’t find a way to do that. I modified all the config files located in the /etc folder removing ‘xaas’ suffix from the DNS names. When I run nslookup XXX.XXX.XXX.XX I get the correct DNS ‘correct-link’ . Not sure why icinga2 still gets the wrong one (with xaas).

I would be happy to provide more information if needed, but I am not even sure what else to provide.

Best,
Sasha

#2

Hi @yakutovicha

what does the NodeName constant look like in /etc/icinga2/constants.conf?

Kind regards
ritzgu

#3

Hi @ritzgu

it is example.organisation.com (as expected).

Best,
Sasha

#4

Relevant lines in icinga source code may be

After tracing GetIdentity() through JsonRpcConnection
GetIdentity() refers to the m_Identity which is passed as identity to the constructor of JsonRpcConnection

It’s not clear to me who calls this constructor and whether one can provide the identity manually.

P.S. After some more searching around, I also came across this:

I.e. icinga stores information on the endpoints in the database. The following tables still contained the old hostname:

icinga.icinga_endpoints ;
icinga.icinga_endpointstatus;  
icinga.icinga_hosts;
icinga.icinga_hoststatus;
icinga.icinga_objects;
icinga.icinga_servicestatus;

Not clear to me whether this has an effect - I tried emptying those tables but the error with icinga daemon -C was unaffected.

#5

In the end, we decided to completely remove mysql and icinga, delete all icinga config files and set it up from scratch again.
It turns out this solved the issue (icinga daemon -C now picks up the correct hostname), i.e. we believe there was some mysterious icinga-internal cache that we were unable to find.

We did remove traces of the old hostname in:

  • any file in /etc
  • cleared /var/cache/icinga2
  • truncated tables in the mysql database that contained the old hostname

Is there another place where icinga stores it?

#6

There is possibily config that is applied under /var/lib/icinga2.

I think of thoses 2 ways to see from where the faulty config come from :

  1. if faulty endpoint is explicitly declared of modified by something it’ll show you
    icinga2 object list --type Endpoint --name wrong-link
  2. if it’s more a code related problem, stracing the icinga process to get from what file/when the DNS call is made for the process can help as well
    strace -f -e network,open -s 9999 -o ./strace-icinga.log systemctl restart icinga2
    Be aware this can output heavy chunks of log.
1 Like