I’ve been using icinga2 for a while now and I thought I had most of it figured out, but recently I ran into an interesting problem that I couldn’t find the answer to.
So, what I was trying to accomplish is monitor AWS autoscaling groups. Automatically adding and removing instances. I know that Director already has a module for that, but I didn’t want to use it.
I used a similar approach to what director does. I have an dynamic ansible inventory which finds the proper hosts for me, and does some magic to register them to the appropriate satellite. And then it creates a configuration for them on the master as an API package for the respective zone. So far so good.
The package gets properly replicated to the satellite zone, but it doesn’t get loaded by the satellite at all.
Can someone help me understand how satellites load configuration from /var/lib/icinga2/api/zones/ ?
I see that the regular zones configured manually appear slightly different on the satellites - Regular zones follow this structure:
/var/lib/icinga2/api/zones/ZONENAME/_etc/{ZONE_FILES}
While the Package zones don’t have an _etc directory and look like:
/var/lib/icinga2/api/zones/ZONENAME/{PACKAGE_NAME}/{PACKAGE_FILES}
I’ve tried including them recursively in the icinga.conf on my satellite like:
include_recursive “/var/lib/icinga2/api/zones/{ZONE_NAME}/{PACKGE_NAME}” and this does sort of work around the issue, but I still fail to understand why they don’t get loaded anyway.
I’ve looked at similar configuration created by director and director also lacks the _etc directory in its zones.
TLDR. Using Icinga API packages. Configuration for them appears on the master, gets shipped to the correct satellite zones. Doesn’t get loaded by the satellite and therefore I have all of the satellite clients on pending from the POV of my master.
If anyone has had to deal with this, please share some insight on how icinga satts load configuration distributed from the master and why the packages get ignored even though they are present.
The config sync zones exist since 2.1, and as such, the config master holds everything in zones.d, all other endpoints receive the synced configuration and store that underneath /var/lib/icinga2/api/zones.
The API config packages were added in 2.4, and as such, there was a slight change within the /var/lib/icinga2/api/zones directory: The package name is used as primary directory. Objects originating from zones.d are stored within _etc while API created config packages preserve their name instead of that.
Meaning to say, directories and package names which a leading _ are treated internally, while real names originate from user created API packages. The director package is also a user created package, just by an application this time.
You don’t need to include /var/lib/icinga2/api/zones, that’s done automatically on startup. You can verify this by calling the config validation with icinga2 daemon -C -x notice | grep zones.
Some questions for troubleshooting:
Does the zone exist on the satellite which receives the files from the parent master? (zones.conf)
What’s the file tree on that satellite tree /var/lib/icinga2/api/zones or ls -lahR /var/lib/icinga2/api/zones
What’s the debug log when such a sync happens on the satellite?
Which versions are involved on both the master and the satellite? icinga2 --version
Thans a ton for the quick reply. My satellite was indeed one minor version ahead, but that wasn’t the problem. It turns out that it’s not just packages that are problematic, but all satellite zone configuration on this specific satellite. I’m still looking into it, but currently I’ve got this.
[2019-04-10 14:55:59 +0300] notice/config: Ignoring non local config include for zone ‘SAT-ZONE’: We already have an authoritative copy included.
So either you have configured in /etc/icinga2/zones.d on the satellite, or you pushed the config packages to the satellite’s REST API - it then thinks it is authoritative for this configuration. Purge everything in /var/lib/icinga2/api/zones and restart the satellite then.
Also make sure that your custom include_recursive in icinga2.conf on the satellite is purged, that’s wrong.
You are right. I figured out what went wrong… Damn this took me too long.
Initially I think I created the package on my satellite while testing things. This lead to the zone folder being created on the satellite through this package and then even though my configuration was replicating from the master it wasn’t being loaded on the satellite because it thought it had the authoritative version.
deleting the /var/lib/icinga2/api/zones/.* on the satellite and restarting it resolved the problem.
Now my package and regular zone files are being loaded as expected on the satellite host.
Once again, Thanks a ton Michael, your help was priceless. Are you the only active community member for icinga? You are everywhere.
you’re welcome My main intent with providing help and ideas is to push everyone to the level where they come back to this platform and enjoy helping others and giving back. The more expertise we all have, the better and strong our community gets. Especially the first learning curve is sometimes hard
My colleagues are very busy, consider that Icinga and all its projects are maintained by a small group of developers, product managers, consultants with roughly 20 people. We all love what we do, but also need to earn our living whilst fixing bugs, creating features and so on.
I’ve always learned from helping others, improving docs and features. In the past 10 years, this helped building a strong Icinga community, and a safe place for work. That’s a great foundation for the next years and decades, and maybe we’ll meet at some point having some beers or G&Ts together
You’re doing great helping people. I can’t name how many times I’ve found solutions to problems in your comments. I’m also trying to share my knowledge by writing public blogs about interesting things I’ve had to overcome.
I’ve never had the chance to attend an icinga-dedicated meetup, but I’d like to do so in the future, we might as well meet and have a beer at one.
Until then, cheers and have a great time,
Alexander
My main intent with providing help and ideas is to push everyone to the level where they come back to this platform and enjoy helping others and giving back. The more expertise we all have, the better and strong our community gets. Especially the first learning curve is sometimes hard