Setup a Virtual master

Icinga 2
1

So we have a standalone Icinga2 server at a location and would like to set up a Master via VM just incase if any problem happens then the other can take over. I am pretty new to this and would like to explore more and set up a nice environment. Can someone guide me
So that basically means we will have 2 masters one at vm and the other standalone.

  • Version used (icinga2 --2.10.1)

  • Operating System and version -----Debian 11

  • Icinga Web 2 version and modules (System - About)
    |grafana||1.3.4||
    | — | — | — | — |
    |idoreports||0.9.1||
    |ipl||v0.3.0||
    |monitoring||2.10.1||
    |setup||2.10.1||
    |test||2.10.1||

2

Welcome

Why not directly go with a HA setup?

3

I would also opt for the HA setup.

A separate master would make things more difficult I guess:

  • config master
  • agent connection

Might be a solution, though I’m not sure how that works:

  • a complete clone of the master (same name, ca, ip, …) in the second location that is started when the other is down.
4

As per this document, it setup one as a master and second one as agent. but with the naming convention it says first master/second master. if i follow this process then one would support another right? means if one goes down then other will take over.

5

Yes and no, some functions run load balanced and others run in a fail over mode.

Also have a look at the documentation not just the blog.

6

Ok, then I would have to look more as we need another master not an agent since the already setup icinga is standalone.

7

If you set up Icinga 2 in a cluster the nodes in the same zone (here: master) run simultaneously (active-active).
By default:

  • They will distribute the checks between them
  • Both will send notifications
  • both write into the database
  • both receive updates from child nodes
  • both sign cert requests

Therefore both masters have to have the same features enabled.
If one of the masters goes down, the remaining one will take over the checks.

For the configuration there is only one master (the first to be set up), meaning you won’t be able to do config changes/deploys when the config master is down.

8

I did not properly understand what this means
For the configuration there is only one master (the first to be set up), meaning you won’t be able to do config changes/deploys when the config master is down.

Also can you please share document where it guides to setup another master with already having a master.

Thanks

9

High-Availability Master Docs

The config master holds the configuration for hosts, services, templates, … under /etc/icinga2/[conf.d|zones.d]
This configuration get replicated to the second master via the API, when both are connected.
If the config master is down you obviously can’t change the config there, and therefore the second master will not receive any new configuration, but will remain running with the config it was last sent via the API.
As config received via the API resides in a different folder a is basically read-only changes there will not affect (and could possibly destroy) the configuration.

10

can we make config changes on second master when the first is down and later the first gets updated with config which were updated in second…Sorry for asking so many question as I want to understanf completely before starting. Hope you dont mind. Thanks for the documentation link

11

No, that won’t work.
The config master holds authority over the config and won’t accept config update from other nodes.

12

Maybe not the first original config master but would it be possible to make the second master node the new config master and add a new node to master zone that will then be receive the config form the former second master that is now the config master?

Maybe this can help in managing the setup: lfops/roles/icinga2_master at main · Linuxfabrik/lfops · GitHub

13

Hi
I am back.
So i set up a master with web UI and one satellite.
The web interface is up and running with master IP. Also i have added a check for satellite host.
I made config as per How to set up High-Availability Masters
After this i followed few more steps as per Initial Sync for new Endpoints in a Zone where i copied the content(/var/lib/icinga2/icinga2.state and /var/lib/icinga2/api/packages/_api) from master to satellite (this is to make satellite also a master)
I tested to shutdown the master to see if the other node will take over but it didn’t …may be i am incomplete with the process.
Could you please guide what would be the next steps.

14

Can you post the zones.conf of original master and the new master?
Also have a look into the logs and review the output of icinga2 feature list on both nodes.

15

Master 1
/*

  • Generated by Icinga 2 node setup commands
  • on 2022-09-24 17:27:59 +0200
    */

object Endpoint “ftest01.xyz-software.com” {
}

object Endpoint “ftest02.xyz-software.com” {
host = “192.168.1.246”
}

object Zone “master” {
endpoints = [ “ftest01.xyz-software.com”, “ftest02.xyz-software.com” ]
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}

Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

Master 2
/*

  • Generated by Icinga 2 node setup commands
  • on 2022-09-24 17:34:08 +0200
    */

object Endpoint “ftest01.xyz-software.com” {
host = “192.168.1.244”
port = “5665”
}

object Zone “master” {
endpoints = [ “ftest01.xyz-software.com”, “ftest02.xyz-software.com” ]
}

object Endpoint “ftest02.xyz-software.com” {
}

object Zone “global-templates” {
global = true
}

//object Zone “director-global” {
// global = true
//}
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql mainlog

16

I see no problem except that director global isn’t global if nodes don’t have it :wink:
Did you also enable the HA config of the enabled features like ido-mysql?

NodeName (/etc/icinga2/constants.conf) are matching the endpoints?

17

I actually commented the director part in master2 afterwards since it was not sync…yes the ido-mysql feature is already enabled.
Is it something to do with director? I mean any installation …i see something in this link Installation - Icinga Director

18

master 1
/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = “/usr/lib/nagios/plugins”

/* The directory which contains the Manubulon plugins.

  • Check the documentation, chapter “SNMP Manubulon Plugin Check Commands”, for details.
    */
    const ManubulonPluginDir = “/usr/lib/nagios/plugins”

/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.

  • Check the documentation, chapter “Plugins Contribution”, for details.
    */
    const PluginContribDir = “/usr/lib/nagios/plugins”

/* Our local instance name. By default this is the server’s hostname as returned by hostname --fqdn.

/* Our local zone name. */
const ZoneName = “master”

/* Secret key for remote node tickets */
const TicketSalt = “*****************************”

master 2
/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = “/usr/lib/nagios/plugins”

/* The directory which contains the Manubulon plugins.

  • Check the documentation, chapter “SNMP Manubulon Plugin Check Commands”, for details.
    */
    const ManubulonPluginDir = “/usr/lib/nagios/plugins”

/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.

  • Check the documentation, chapter “Plugins Contribution”, for details.
    */
    const PluginContribDir = “/usr/lib/nagios/plugins”

/* Our local instance name. By default this is the server’s hostname as returned by hostname --fqdn.

/* Our local zone name. */
const ZoneName = “ftest02.xyz-software.com”

/* Secret key for remote node tickets */
const TicketSalt = “”

19

Just to be clear, I meant:

object IdoMysqlConnection "ido-mysql" {
  ...
  enable_ha = true

And the DB needs to be on a third machine so that both masters share the same DB for enable_ha to work.

Also I have on both master nodes the accept_config and accept_commands set to true:

object ApiListener "api" {

  accept_config = true
  accept_commands = true

}

but the first one also has ticket_salt = TicketSalt

20

I am using database for each node. so in that case i set enable_ha = false…
So there are 2 conf file one with api.conf and other api.conf.orig…In api.conf.orig i see this on both nodes
object ApiListener “api” {
//accept_config = false
//accept_commands = false

ticket_salt = TicketSalt
}
should i change it to true and uncomment it?