Added single service executes on master instead of host

odaroloc · June 7, 2020, 11:39pm

I added a single service using Director (entire system built with Director) to a host. The check executes on the master instead of the host. If I go into the master icinga2 directory structure and find the latest modified directory, i.e.:
sudo vi /var/lib/icinga2/api/packages/director/25682ce4-f50b-4ab8-b7e7-ce7e86b6cdf1/zones.d/director-global/service_templates.conf
I find the incorrectly executing host service template is missing the last 2 lines which are included in all the other templates, i.e.:

enable_notifications = true
command_endpoint = host_name

service_templates.conf

template Service “check-rpi-temp” {
check_command = “rpi-temp”
command_endpoint = host_name
}

template Service “check-mdstat” {
check_command = “check-mdstat”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-shutdown-button-service” {
check_command = “shutdown-button-service”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-spa-temp” {
check_command = “check-spa-temp”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-hostalive” {
check_command = “hostalive”
command_endpoint = host_name
}

template Service “check-fail2ban” {
check_command = “fail2ban”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-apt” {
check_command = “apt”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-disk” {
check_command = “disk”
enable_notifications = true
command_endpoint = host_name
vars.disk_ereg_path = “/media/backup”
}

template Service “check-load” {
template Service “check-rpi-temp” {
check_command = “rpi-temp”
command_endpoint = host_name
}

template Service “check-mdstat” {
check_command = “check-mdstat”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-shutdown-button-service” {
check_command = “shutdown-button-service”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-spa-temp” {
check_command = “check-spa-temp”
enable_notifications = true
command_endpoint = host_name
} check_command = “load”
enable_notifications = true
command_endpoint = host_name
vars.load_cload1 = “15”
vars.load_cload15 = “6”
vars.load_cload5 = “9”
vars.load_wload1 = “7.5”
vars.load_wload15 = “4.5”
vars.load_wload5 = “6”
}

template Service “check-ssh” {
check_command = “ssh”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-swap” {
check_command = “swap”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-rpi-firmware” {
check_command = “rpi-firmware”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-pi-sd-card” {
check_command = “disk”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-rpi-temp” {
check_command = “rpi-temp”
command_endpoint = host_name
}

template Service “check-mdstat” {
check_command = “check-mdstat”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-shutdown-button-service” {
check_command = “shutdown-button-service”
enable_notifications = true
command_endpoint = host_name
}

template Service “check-spa-temp” {
check_command = “check-spa-temp”
enable_notifications = true
command_endpoint = host_name
}

If I add these lines to the file, it works correctly, for a while. I can see the host change in the “Dashboard>Service>Check Execution>Check Source” line.
Perhaps a new directory structure is generated during a refresh, and the new structure reverts to the same error.
As I am using Director for all changes and additions, I consider this a bug. I previously added an different single service to another host and this consistently works correctly. I’m happy to supply whatever information is needed.

On the Master:

icinga@master-bldr:~ $ icinga2 --version
[2020-06-07 17:42:21 -0600] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error “Operation not permitted”
[2020-06-07 17:42:21 -0600] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error “Operation not permitted”
icinga2 - The Icinga 2 network monitoring daemon (version: r2.11.2-1)

Copyright (c) 2012-2020 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
Platform: Raspbian GNU/Linux
Platform version: 10 (buster)
Kernel: Linux
Kernel version: 4.19.97-v7+
Architecture: armv7l

Build information:
Compiler: GNU 8.3.0
Build host: runner-LTrJQZ9N-project-297-concurrent-0

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

On the host:

spa@spatemp:~ $ icinga2 --version
[2020-06-07 17:43:55 -0600] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error “Operation not permitted”
[2020-06-07 17:43:55 -0600] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error “Operation not permitted”
icinga2 - The Icinga 2 network monitoring daemon (version: r2.11.2-1)

Copyright (c) 2012-2020 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
Platform: Raspbian GNU/Linux
Platform version: 10 (buster)
Kernel: Linux
Kernel version: 4.19.118+
Architecture: armv6l

Build information:
Compiler: GNU 8.3.0
Build host: runner-LTrJQZ9N-project-297-concurrent-0

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

odaroloc · June 17, 2020, 10:47pm

Since posting 10 days ago, this happened again, but the modifications to the service template were more severe. Icinga would not start due to the jumbled statements and duplicated host template entries. Hand editing of service_templates.conf was required again to correct the errors and launch the monitor.
I also noted that this post was viewed 36 times but there were no replies.

theFeu · June 19, 2020, 3:10pm

Hello,
it seems like no one knew an answer to your issue.
I will forward this to @tgelf on our internal channels, as I hope he might be able to help you out with this.
If you think that something is wrong in the tool / you found a bug, it might be for the best to open an issue on GitHub, as this is more ingrained in the developers workflow

Thank you for your patience,
Feu

tgelf · June 19, 2020, 3:15pm

Hi @odaroloc,

enabling Run on agent for your service (or better: the related service template) as shown in the documentation should get you covered.

Cheers,
Thomas

odaroloc · July 9, 2020, 1:33am

Hi tgelf, thanks for your response. All my service templates have Run on agent enabled. But single services, without a template since they are only intended to and can run on one host, do not have an (obvious?) Run on agent option under Director. Yes, I could create a service template for a single service, but I wanted to try the single service option. If the corruption occurs again I may do so in order to avoid the problem, I hope.

However, I think you miss my point. My Director created setup was working correctly and then failed due to internal service template file corruption. The single service required a manual correction to the file. This happened twice, but not in the 20 days since. This is a bug in Director/Icinga2, which I will submit as such., per Feu’s suggestion. Thanks again.

tgelf · July 13, 2020, 4:30pm

Hi @odaroloc,

it’s very, very unlikely that “internal service template file corruption” wipes away exactly those two lines. The “Run on agent” flag toggles command_endpoint = host_name, there should be no other magic involved. I also replied to your GitHub issue, we can keep the discussion either here or there. I’d suggest to close the GitHub issue, as this doesn’t seem to be a bug at all.

Please have a look at your Deployment log and make sure that it show successful deployments. Please do not manually tweak those files on disk, the next deployment will wipe them away.

Best,
Thomas