Host is not part of a hostgroup even though it should be

Occassionaly it happens that a host is not part of a hostgroup even though it should be.

After detailed issue analysis it was found that the information is missing in the icinga2 database. For instance, in icingaweb2 GUI xx01.a.b.c.d hostgroup has 29 members, even though there should be 30 members with the hostname xx01…. Matching is done with an assign rule stading the prefix of a system.

Example:

object HostGroup "xx01.a.b.c.d" {
    assign where match("xx01*.a.b.c.d", host.name)
}

Icinga2 database:

first check hostgroup_id of our hostgroup name

icinga_2=# select * from icinga_hostgroups where alias like 'xx01%';
 hostgroup_id | instance_id | config_type | hostgroup_object_id | alias | notes | notes_url | action_url | config_hash
--------------+-------------+-------------+---------------------+-----------------------+-------+-----------+------------+------------------------------------------------------------------
 28 | 1 | 1 | 5281 | xx01.a.b.c.d | | | | hjhdjshdjxhjdshjshdjhdjsnhjs
(1 row)

check host_object_id of host which hostgroup is missing in icingaweb2
Since xx01-1.a.b.c.d is missing, its host data can be found with:
icinga_2=# select * from icinga_hosts where alias like ‘xx01-1.%’;
Here we got:

its host_id = 123
its host_object_id = 1234

select all members of given hostgroup (by id)

icinga_2=# select * from icinga_hostgroup_members where hostgroup_id = 28;
 hostgroup_member_id | instance_id | hostgroup_id | host_object_id | session_token
---------------------+-------------+--------------+----------------+---------------
 38 | 1 | 28 | 2288 |
 203 | 1 | 28 | 2453 |
 228 | 1 | 28 | 2478 |
 281 | 1 | 28 | 2532 |
 357 | 1 | 28 | 2608 |
 403 | 1 | 28 | 2654 |
 423 | 1 | 28 | 2674 |
 694 | 1 | 28 | 2947 |
 882 | 1 | 28 | 3135 |
 884 | 1 | 28 | 3137 |
 908 | 1 | 28 | 3161 |
 1323 | 1 | 28 | 3579 |
 1610 | 1 | 28 | 3867 |
 1711 | 1 | 28 | 3968 |
 1723 | 1 | 28 | 3980 |
 1750 | 1 | 28 | 4007 |
 1801 | 1 | 28 | 4058 |
 1928 | 1 | 28 | 4186 |
 1947 | 1 | 28 | 4205 |
 1961 | 1 | 28 | 4219 |
 2062 | 1 | 28 | 4320 |
 2080 | 1 | 28 | 4338 |
 2088 | 1 | 28 | 4346 |
 2091 | 1 | 28 | 4349 |
 2108 | 1 | 28 | 4366 |
 2376 | 1 | 28 | 4636 |
 2557 | 1 | 28 | 4818 |
 2583 | 1 | 28 | 4844 |
 2607 | 1 | 28 | 4868 |
(29 rows)

Here we expect 30 rows.
Unfortunatelly host_object_id 1234 is not part of hostgroup members.

It looks like a bug in icinga2. Why is that happend and how to fix it?

Additional information about our icinga2 installation:

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.19.0-0.bpo.1-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: cb654124b660

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Thanks in advance,

Best regards

This is an annoying bug and is there any way to trigger icinga2 that it recreates the objects in the database or does a validation of the information in the database?

Is there a way to see if a host is part of a hostgroup from the point of view in icinga2 in order to check if it is only a ‘bug’ of the database representation or its also the view of the icinga2 master?

I can print the hostgroup object using the command
icinga2 object list --type HostGroup --name …
and see the host object using
icinga2 object list --type Host --name …
but in the output i don’t see if a host is part of a hostgroup or not…

I’ve stumbled upon the same problem, albeit in a much simpler case: three HostGroups with two Hosts each, yet we end up with one group complete, one group with only one host and one group empty.

I think what happens is this:

The config compiler is multi-threaded, but there’s no depedency between Host and HostGroup objects. Thus processing HostGroups may finish before the Hosts are done and then you’ll end up with partially filled HostGroups. The same can happen to ServiceGroups.

There’s some information in GitHub issue #8116 and there’s also a patch, but since there was no reply whether the patch works or not the patch has been sitting stale since.

Also I think the patch may be incomplete as in: the HostGroups will contain all Hosts, but now you may fail assigning Services to HostGroups for the same reason as above.

Nevertheless I’ve built patched packages and I will try to test them and let you know the results. This may take a while, since I run a huge setup with many parties involved.

In the long term there’s propably no other way than to get rid of Host- and ServiceGroups in our configs. The documentation doesn’t say so yet, but whenever the issue comes up the Icinga2-developers clearly state that these groups were brought over from Icinga1, don’t integrate well into Icinga2 and are best avoided.