Create Hostgroups from imported puppet nodes

Problem: in puppet we have two custom facts: “role” and “application” for defining hosts.
I would like to create Icinga Hostgroups for all “roles” and “applications” and assign the hosts to these groups.

I have an import source for the puppetdb (which works) and I can extract the two facts and modify them to have the values:
hostgroup_role = role_${facts.role}
hostgroup_application = application_${facts.application}

In a sync rule I try to create “Host Groups” an set object_name to hostgroup_role
Preview looks OK
BUT: when I “Trigger this Sync” I get:
This Sync Rule failed when last checked at 2019-01-31 14:27:20: Exception while syncing Icinga\Module\Director\Objects\IcingaHostGroup role_nor_db: Trying to recreate icinga_hostgroup ("role_nor_db")

which is clear, because the “roles” are not unique.

Update Policy is “Ignore”

Is there any way to make this work?

Hi,

which Director version are you using? The errors reads like that it sources from a very late check in the database context - similar as you would recreate a hostgroup via REST API.

Maybe the following issues shed some light?



Reading them makes me believe that you need a unique object identifier which allows to ignore duplicates? Maybe any sorts of additional facts which stay the same over time … what else do your facts provide as data?

HTH,
Michael

Hi,

I use the current git master for director :slight_smile:

After reading the first issue, I maybe see my problem: my “puppetdb import” imports Hosts/Nodes, but not Host Groups.

Exporting the two facts as resources will also not work, because the importer will use “certname” also as primary key and I will get duplicates again.

I think I need some kind of data source containing only the unique “roles” and “applications”

Maybe there is a way to query the puppetdb externally and create a CSV file or something like that…

Thanks for pushing me in the (hopefully right) direction :slight_smile:

Hi,

please share your findings here then. Import sources are still a thing where I need to learn a lot, and I am eager to see your real world requirements :slight_smile:

In terms of data … I’ve read that quite often that Tom suggests to only import a partial data set and not the entire PuppetDB because of performance and whatnot. I’m not sure if exported resources could still do the trick, or one uses a “proxy” in the middle to provide only the partial data set.

To get a better picture, can you share some Puppet snippets from the host itself, or a facts list it provides?

In terms of using git master - I would suggest to use the release archives in prod, and have git in a test environment. That way you can really say whether you hit a bug already known in version x.y or just a development regression in master.

Cheers,
Michael

I found a very ugly solution:

Step 1: create custom puppet fact on Icinga2 Master with a dirty perl script:

#!/usr/bin/env perl

use strict;
use warnings;

my $role_hostgroups = qx(curl -s -X GET  http://PUPPETDBHOST:8080/pdb/query/v4/facts/role --data-urlencode 'query=["extract",["value"],["group_by","value"]]' | jq   -r '.[]|.value');
my @role_hostgroups = sort map {"role_$_" } grep { $_ ne 'not_set' } split(/\n/,$role_hostgroups);
my $application_hostgroups = qx(curl -s -X GET  http://PUPPETDBHOST:8080/pdb/query/v4/facts/application --data-urlencode 'query=["extract",["value"],["group_by","value"]]' | jq   -r '.[]|.value');
my @application_hostgroups = sort map {"application_$_" } grep { $_ ne 'not_set' } split(/\n/,$application_hostgroups);

open (my $fact, ">", "/etc/facter/facts.d/icinga_role_application_hostgroups.yaml") or die "Kann /etc/facter/facts.d/icinga_role_application_hostgroups.yaml nicht schreiben: $!";
print $fact "icinga_role_application_hostgroups: [".join(",",@role_hostgroups,@application_hostgroups)."]\n";
close $fact;

That creates a fact:
[ "role_ROLE1", "role_ROLE2", ... "application_APP1", "application_APP2", ... ]

This script is startet by cron every 5 minutes, so new “roles” or “applications” are available a few minutes later.


Step 2: Add some code to our puppet manifests for the icing2 master:

icinga2_master::hostgroups { $::icinga_role_application_hostgroups: }

and in “icinga2_master/hostgroups.pp”:

define icinga2_master::hostgroups (
  $hostgroup = $name,
)
  {

    icinga2::object::hostgroup { "${hostgroup}":
      target => '/etc/icinga2/conf.d/hostgroups.conf',
      assign => [
        "${hostgroup} == role_ + host.vars.puppet_role",
        "${hostgroup} == application_ + host.vars.puppet_application",
      ],
    }

  }

With the next puppet run it creates the hostgroups and assigns the hosts to it.


Conclusion: in german I would call it: “Von hinten durch die Brust ins Auge”, but at least it works…

I will try, if the fileshipper module can read CSV and create Hostgroups from that, so my perl script could just write the CSV. Or the “curl” could write it directly. I will try this…

1 Like

Solution 2: “Use Fileshipper”, they (Tom :slight_smile: ) said. “It works”, they said. “It may ruin your nerves with assign_filters”, I say…

Steps:

  • Create a CSV from the two curl calls and add a column header “hostgroup”
  • add this file to a fileshipper source directory (don’t forget to handcraft imports.ini first :frowning: )
  • create import source with that file
  • create sync rule with object_name = ${hostgroup}

That creates all groups in director!

I also tried to add an assign_filter there, but I need:
assign where "${hostgroup}" == "role_"+host.vars.role
but I could not convince director to create this…
It always created something with “contains” or “= null” or “in =” …

So I added a “groups” modifier to the hosts sync rule, which works at the end.

This is much nicer than the first puppet solution, but still not perfect…

What I really hate is the fact, that director tries to understand user provided filters and creates something unpredictable from that! Why not simply use the provided string and only replace ${variable}?

I wouldn’t call that “hate”, it is just a different thought within the design, maybe forgotten, or just needs a different perspective. I do think that this likely needs better logging, or more insights into what’s going on. I had that problem with the kickstart wizard a while back too, and opened this issue. Did not have the time to submit a PR yet, but it is on my list.

The problem with hostgroups is that a host can be a member of 1 or many, and as such the renderer attempts to create this automatically. I guess that’s merely the reason for providing a suitable web form filter, and automagically transforming this into the correct Icinga 2 DSL in the background. Not so easy to tackle that DSL from a config frontend, long story.

In terms of the first solution: You’re putting the config into conf.d, and depending on your distributed setup you might need this definition on all HA masters or satellites too. Maybe change that into a global zone :slight_smile: I agree that it is not perfect, but hey, you’ve found a way to make it work and learned a lot, that’s what counts the most :slight_smile:

I don’t know if that’s possible at the host import stage, but you could also try to populate the groups attribute during sync when creating the host objects from the node. During a short research, I’ve found this issue which looks promising.

Cheers,
Michael