Secondary master - Icinga2 won't start

Icinga 2
1

Hello,

In Icinga2 2.5.4 cluster with 2 nodes. The master node is working fine. Somehow on the secondary master, the icinga2 process won’t start. It reports error about “generic-host” definition unknown which it should read from “templates.conf”

On secondary someone deleted all content under /var/lib/icinga2/api directory. AFAIK, the directory gets sync’d with the master. So in current situation, do I need to manually copy the contents from master - /var/lib/icinga2/api on secondary so it will be able to start ?

Please advise.

Thanks

2

Hi,

That’s quite old and may have bugs, I’d recommend planning an upgrade.

Can you share where the templates.conf file is located? This must be synced e.g. via the global-templates zone in order being available on all endpoints.

Cheers,
Michael

3

It’s not anywhere on the host. It was deleted. Does it have to be in /var/lib/icinga2/api/zones ?

EDIT: Yes that directory needs to have the data. So I just have to copy the content from master’s /var/lib/icinga2/api/ directory?

Thanks

4

I copied global-templates from master /var/lib/icinga2/api/zones to the secondary in the same directory. But I see the same error.

5

Ok I copied over the “global templates” and removed the .authorize file and I could able to start Icinga2 on the secondary node. However it doesn’t have the downtimes and comments sync’d. So when it took over the active endpoint, lots of alerts which should be in the downtime appeared on the web ui. Is there a way to sync the downtimes and comments ? just copy from the master would work if there is no way to sync them?

The downtimes/comments for those hosts should on the satellites. Is that right? If that’s correct, then the downtimes/comments should be sync’d on the secondary master yes?

Thanks

6

Ok looks like Per debug log, the secondary master is discarding updates from all satellites.

warning/ApiListener: Discarding 'config update object' message from

Any idea what could be the problem with that ?

Thanks

It is expected it seems as per

/* discard messages if the sender is in a child zone */
if (!Zone::GetLocalZone()->IsChildOf(endpoint->GetZone())) {
	Log(LogWarning, "ApiListener")
	    << "Discarding 'config update object' message from '"
	    << origin->FromClient->GetIdentity() << "'.";
	return Empty;
}
7

Ok I realized that the secondary node is trying to sync up with the master which should include downtimes/comments etc. I see it is doing that but Icinga2 on the master would crash with “segfault”. This is causing the secondary not getting sync’d up.

icinga2[7478]: segfault at 28 ip 00002b546bc85e42 sp 00002b5578600c20 error 4 in libbase.so[2b546bb38000+1df000]
8

Hello,

Currently the secondary node is out of sync and has old data. How do I remove the old data and sync the latest content to it from master?

Please advise.
Thanks

9

Have you upgraded as advised?

10

No not upgraded but need to get it sync’d to help with the migration to v2.11 afterwards.
Thanks

11

Maybe you upgrade and the sync… will just work?