Notifications is not working on HA configuration

Hello,

I’ve discovered that notifications is only working with mail and not working my other application. Do you know how I can check it the HA configuration is provoking the issue?

as I can see on the notification:

→ HA cluster active, this endpoint does not have the authority (paused=true). Skipping.

Hi @alperez,
hard to say with that information, is the notification feature active on both master nodes?

Hello,

Yes, we have active notifications in both master nodes. As I could see, only works notifications related with “HOSTS” all the notifications with services are not working when the nodeB is active. To check it that could be an issue with the HA configuration I’ve stopped the nodeB and I was testing all the notifications creating some alerts and forcing these alerts and works. So this is why I think that the issue is related with the nodeB HA configuration.

In the other hand, I’ve raised another ticket here related to “check now” funcionatily and it’s the same issue, if nodeB is working “check now” option and some services applies rules are hang.

I didn’t do the configuration of HA was another person and sincerelly i don’t know how he did it. as I can see on the document must exists a CA certicated that for me not exists in none master nodes and besides we have two satellites configured. So my problem also is that I cannot check it if the HA is correctly configured or not.

This is why I’ve created the thread.

Please share from both masters

Also if you can only send mails from one of both masters check your MTA (postfix most likley) if it is configured correctly on both masters.

Hi,

Information added. Right now Master2 is down because I’ve stopped the icinga2 daemon.

In the other hand the problem is not with the sendmail both nodes are sending mails is not working or script to use the oncall.
I’ve tested manually the scripts and works so the problem is with MASTER2, besides, services apply rules are hang with MASTER 2 configuration check now is not working and satellites are hang also.

MASTER 1 icinga2 daemon -C

[2024-07-23 08:12:19 +0200] information/cli: Icinga application loader (version: 2.13.2-1)
[2024-07-23 08:12:19 +0200] information/cli: Loading configuration file(s).
[2024-07-23 08:12:19 +0200] information/ConfigItem: Committing config item(s).
[2024-07-23 08:12:19 +0200] information/ApiListener: My API identity: eurvlii75429.xmp.net.intra
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1 SyslogLogger.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 3 Users.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 7 TimePeriods.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1700 Services.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 4 Zones.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1 NotificationCommand.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 96 HostGroups.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 2856 Notifications.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 4 Downtimes.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 438 Hosts.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 4 Endpoints.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 12 Comments.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 256 CheckCommands.
[2024-07-23 08:12:19 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2024-07-23 08:12:19 +0200] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2024-07-23 08:12:19 +0200] information/cli: Finished validating the configuration file(s).

MASTER 1 zones.conf

/*

  • Generated by Icinga 2 node setup commands
  • on 2023-02-17 17:36:43 +0100
    */

/Master endpoint/
object Endpoint “MASTER 1” {
host = “MASTER 1”
}

object Endpoint “MASTER 2” {
host = “MASTER 2”
}

/Satellite-1/
object Endpoint “SATELLITE-1” {
host = “SATELLITE-1”
}

/Satellite-2/
object Endpoint “SATELLITE-2” {
host = “SATELLITE-2”
}

object Zone “master” {
endpoints = [ “MASTER 1” , “MASTER 2”]
}

object Zone “zone-emea” {
endpoints = [ “SATELLITE-1” , “SATELLITE-2”]
parent = “master”
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}

icinga2 feature list
Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb ido-pgsql influxdb influxdb2 livestatus opentsdb perfdata statusdata
Enabled features: api checker command ido-mysql mainlog notification syslog

Master2 icinga2 daemon -C

[2024-07-23 11:03:39 +0200] information/cli: Icinga application loader (version: 2.13.2-1)
[2024-07-23 11:03:39 +0200] information/cli: Loading configuration file(s).
[2024-07-23 11:03:39 +0200] warning/ConfigCompiler: Cannot compile file ‘/etc/icinga2/features-enabled/notification.conf’: Error: Function call ‘std::ifstream::open’ for file ‘/etc/icinga2/features-enabled/notification.conf’ failed with error code 13, ‘Permission denied’

Context:
(0) Compiling configuration file ‘/etc/icinga2/features-enabled/notification.conf’
[2024-07-23 11:03:39 +0200] information/ConfigItem: Committing config item(s).
[2024-07-23 11:03:39 +0200] information/ApiListener: My API identity: eurvlii75430.xmp.net.intra
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1 SyslogLogger.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 3 Users.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 6 TimePeriods.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1480 Services.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 4 Zones.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1 NotificationCommand.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 94 HostGroups.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1923 Notifications.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 4 Downtimes.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 449 Hosts.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 4 Endpoints.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 12 Comments.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 256 CheckCommands.
[2024-07-23 11:03:40 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2024-07-23 11:03:40 +0200] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2024-07-23 11:03:40 +0200] information/cli: Finished validating the configuration file(s).

MASTER 2 zones.conf

/*

  • Generated by Icinga 2 node setup commands
  • on 2024-04-11 20:22:29 +0200
    */

object Endpoint “MASTER 1” {
host = “MASTER 1 IP”
port = “5665”
}

object Endpoint “MASTER 2” {
host = “MASTER 2”
}

object Zone “master” {
endpoints = [ “MASTER 1” , “MASTER 2” ]
}

/Satellite-1/
object Endpoint “SATELLITE-1” {
host = “SATELLITE-1”
}

/Satellite-2/
object Endpoint “SATELLITE-2” {
host = “SATELLITE-2”
}

object Zone “zone-emea” {
endpoints = [ “SATELLITE-1” , “SATELLITE-2”]
parent = “master”
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}

icinga2 feature list
Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb ido-pgsql influxdb influxdb2 livestatus opentsdb perfdata statusdata
Enabled features: api checker command ido-mysql mainlog notification syslog

the icinga user ( nagios on debianish OSes) does not have read permissions on that file :slight_smile:

1 Like

so… what’s the permission that must exists on /etc/icinga2 folder? because I can see that zones.conf is root:root and other inside of features also the owner and the group is root:root

That folder and it’s contents should belong to the user running the icinga2 service.
icinga on rhel/rpm-based systems
nagios on debian-based systems

How come that it is changed on your setup? Did someone copy all the files from one node to the other?
The installation process will place the files with the correct ownership.

Hello,

Sincerelly I don’t know how it was intalled the second node. Because as I don’t have access to the repo. I think that all packages were download and installed manually but I don’t know if there was a copy or not.

This is also the thing that I’m triying to understand.

Hi again,

Do you have another idea? I was thinking that could be an issue with HA because we have both masters “UP”.

What’s the normal behaviour of HA configuration for icinga2?

Because the “checks” for service apply rules are not working with both masters “UP” I think that something internall, is not ok.

Regards

Hello,

I’m still facing to this issue. Each time that nodeB is up with icinga service. All services are hang and notifications are not working.

Do you have any idea?

It could be due that We are configured MariaDB only in nodeA?

Regards

Seems I have missed your last post.

Normal behavior is an active/active cluster, so both nodes running, having the same zones and features configured and share hosts/checks.
See the docs for more detailed description: Technical Concepts - Icinga 2

If all services stop working when you turn on node b, than there is a problem with the configuration. I guess both masters think that they are the only master.
Turn on debug logs and check what is written there.

Check that the zones.conf matches the HA master setup from the docs Distributed Monitoring - Icinga 2

Check that only the config master (generally the one you set up first) has files/directories und /etc/icinga2/zones.d

Fix that as well. As written above the enabled features of both masters have to be the same.

Hi, many many thanks for your respone.

But last question about HA configuration. If we want to have two masters active active with same configuration this means that we need to have icingaweb2 in both servers, icinga director in both servers and a DB in both servers?

Or. do I need to have it icingaweb2 in both servers and director in both servers and the database in a different server?

I was doing at home some test with different configurations and for me as I can see best configuration it has been, configuring icingaweb2, icinga-director and Postgres in a server, and one master in another server and other master in another server. My test was working fine when I’ve put down one master monitors were moved to the another and we I’ve started again the master1 was working all again.

But I have a question about icinga agent, is it mandatory install on the same place that icingaweb2? Do you know if we need to have, icingaweb2 and master1 and master2 connected to Database all the time?

Regards

Icinga Web 2 is completely separate from the Icinga 2 core. You don’t need the webinterface for a running icinga cluster. It is “just” for displaying purposes.

I would suggest, as you are running a HA setup, to install the webinterface on a separate server. Same for the database, install it on a dedicated server.

Both masters need to be connected to the database.

When configuring the Director make sure it is only connected to the configuration master.

1 Like