Icinga2 secondary master server getting restarted on a regular interval

Give as much information as you can, e.g.

We have the below ansible task for icinga . The jenkins build runs every two hours and the icinga restarts on every build.

- name: Setup client
  shell: |
    icinga2 node setup --ticket {{ticket.stdout}} \
    --cn {{agent_fqdn}} \
    --endpoint {{groups.icinga_primary_master[0]}} \
    --zone master \
    --parent_zone master \
    --parent_host {{master_ip}} \
    --trustedcert {{icinga_certs_dir}}{{icinga_master_cert_file}} \
    --accept-config \
    --disable-confd
  register: setup
  • name: Restart Icinga in secondary master
    systemd:
    name: icinga2
    state: restarted
    daemon_reload: yes
    enabled: yes
    when: setup.changed
  • Version used (icinga2 --version)
    icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.4-1)

System information:
Platform: CentOS Linux
Platform version: 7 (Core)
Kernel: Linux
Kernel version: 3.10.0-1160.6.1.el7.x86_64
Architecture: x86_64

Build information:
Compiler: GNU 11.2.1
Build host: runner-hh8q3bz2-project-322-concurrent-0
OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

  • Enabled features (icinga2 feature list)

Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql mainlog

  • Config validation (icinga2 daemon -C)

master2-icinga ~ # icinga2 daemon -C
[2022-07-05 07:20:17 +0000] information/cli: Icinga application loader (version: r2.13.4-1)
[2022-07-05 07:20:17 +0000] information/cli: Loading configuration file(s).
[2022-07-05 07:20:17 +0000] information/ConfigItem: Committing config item(s).
[2022-07-05 07:20:17 +0000] information/ApiListener: My API identity: master2-icinga.int.cloud.ruckuswireless.com
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 5 Users.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 2913 Services.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 9 ServiceGroups.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 4 Zones.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 4 NotificationCommands.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 6689 Notifications.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 432 Hosts.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 24 HostGroups.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 4 Endpoints.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 6 ApiUsers.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 269 CheckCommands.
[2022-07-05 07:20:20 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2022-07-05 07:20:20 +0000] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’

  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes

include “/etc/icinga2/agents.conf”

object Endpoint “master-icinga.int.************” {
}

object Endpoint NodeName {
}

object Zone “master” {
endpoints = [
NodeName,
“master-icinga.int.**********”,
]
}

object Zone “global-templates” {

global = true

}

object Zone “global-commands” {
global = true
}

Hi and welcome to the community.

Can you please describe the problem you are having and what you would have expected?
Because tbh I don’t get it and the ansible part looks like it does what it should (setting up a server with the icinga agent und reloading the service).

Why do you run this at an interval?
To set up a secondary master you need to set it up as a satellite first and than manually edit some config files. This is explained in the docs here: Distributed Monitoring - Icinga 2

Hi @log1c

Thank you for your response.
So we have a GitOps Project model , we have the jenkins pipeline which runs every two hours and so are the ansible tasks.

My doubt is, the client setup (one of the ansible task) is running when the jenkins job is triggered.
Icinga is restarted whenever the client-setup is changed.
I wanted to know what is changing??

Can I just reload instead restart the icinga service.

We have the below ansible task for icinga . The jenkins build runs every two hours and the icinga restarts on every build. Do we have to set up the client everytime or just once is enough ?

- name: Setup client
  shell: |
    icinga2 node setup --ticket {{ticket.stdout}} \
    --cn {{agent_fqdn}} \
    --endpoint {{groups.icinga_primary_master[0]}} \
    --zone master \
    --parent_zone master \
    --parent_host {{master_ip}} \
    --trustedcert {{icinga_certs_dir}}{{icinga_master_cert_file}} \
    --accept-config \
    --disable-confd
  register: setup
  • name: Restart Icinga in secondary master
    systemd:
    name: icinga2
    state: restarted
    daemon_reload: yes
    enabled: yes
    when: setup.changed

Tbh I’m not sure why/what is changing. Maybe adding some debug output helps.

But also I don’t understand why you would run the 2nd master setup (or any icinga agent setup for satellites/clients) periodically. This should be a one time thing. If you want to run it periodically as a kind of “quality control” you have to make sure all your tasks are idempotent.

As the 2nd master also requires some more (manual) steps after this initial setup (config modification, feature enablement) you are not done after this one task.