Self Monitoring icinga2 master (Agent Based)

Hello

New in icinga2. So may be the question is already answered or known to community.

I have installed a master icinga2 server on CentOS
I have installed client icinga2 agent on SUSE
I have graphite
I have director

Now before I ran the $icinga2 node wizard the master server server’s self monitoring was done by icinga2 automatically. But as soon as I ran that and changed it to master it stopped monitoring self.
Can you share any document which can help me to setup both agent and agentless monitoring.
Currently under /etc/icinga2/zones.d/master

I have 2 files
services.conf → Contains service defination

suse.domain.conf → Contains Below

object Endpoint “suse.domain” {
host = “suse.domain”
}

object Zone “suse.domain” {
endpoints = [ “suse.domain” ]
parent = “master”
}
object Host “suse.domain” {
check_command = “hostalive”
address = “suse.domain”
vars.client_endpoint = name //follows the convention that host name == endpoint name
vars.os = “Linux”
}

I guess you answered the last question of the node wizard with yes:

Do you want to disable the inclusion of the conf.d directory [Y/n]:

This removes the example config which is part of the package. Hence, the host is no longer “self monitored”. But this is ok as you want to use the director. Means start creating a host object for the master and create any required service objects.

Thank You. Understood why it removed the self monitoring. But on the Master Server since already icinga2 is installed as server - will I be installing another icinga2 as agent on the icinga2 master server ? My other servers have icinga2 agents configured

Also can we have both agentless and agent based monitoring in icinga2 at the same time? If yes do you have any step by step docs around it

No, you just need one icinga instance per node, one is configured as master and other are configured as clients. Have you already read this chapter? It’s a good starting point to get knowledge about the basics as it explains the concept and all details.

Hello Roland

Sorry, but this is still a little unclear to me, despite me browsing through available docs.

I have already installed Icinga server on ServerX to remotely monitor other host services.
I also would like to setup 2nd pillar, which is client-based monitoring.

Can i create master node on the same ServerX then? Seems exclusion of conf.d directory makes it not possible.

Thanks and Regards

I have already installed Icinga server on ServerX to remotely monitor other
host services.

This means you have an Icinga Master, and it is managing Agents (those are the
“other hosts” you mention).

I also would like to setup 2nd pillar, which is client-based monitoring.

I don’t understand this - do you mean you want a second master for high
availability?

Can i create master node on the same ServerX then?

ServerX is a Master, if it’s already remotely monitoring other host services
(which you say above it is).

Seems exclusion of conf.d directory makes it not possible.

Ther is no difference at all in what you install on an:

  • Icinga Master
  • Icinga Satellite
  • Icinga Agent

Which machine is which is purely determined y the “parent” settings in your
configuration files.

Agents have a Satellite as their parent (or the Master if you’re not using
Satellites) and Satellites have the Master as their parent.

Does that help to clarify?

Antony.

Hello Antony

Master setup guide:
https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#distributed-monitoring-setup-master

says:

The setup wizard will ensure that the following steps are taken:
(…)

  • Update the [icinga2.conf] to disable the conf.d inclusion, and add the api-users.conf file inclusion.

Therefore my service monitoring setup will vanish, wont it?

Ah, sorry, I thought you were saying that you have ServerX running Icinga and
remotely monitoring other host services, but you were confused about the
definitions of Master, Satellite and Agent.

You’re right, the Master setup does disable the inclusion of conf.d in the
icinga.conf file, so you should move any configuration you have in there to
something under zones.d (I suggest something like /etc/icinga2/zones.d/master
since these will only be configuration files you have created for the master so
far), and then you create anything specific to the Agents under something along
the lines of /etc/icinga2/zones.d/agentone etc.

See https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/
#configuration-modes

“Next, you need to define two zones. There is no naming convention, best
practice is to either use master, satellite/agent-fqdn or to choose region
names for example Europe, USA and Asia, though.”

See also https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/
#best-practice

regarding setting up the “global zone” which contains stuff which needs to be
the same on every machine.

Regards,

Antony.

Removed some of my posts to reduce bias.

After master node wizard:

Master zone name [master]: agentbased
Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]: n
Do you want to disable the inclusion of the conf.d directory [Y/n]: y

i get service start failure (code=exited, status=139) with no trace of problems in logs…

Turning off API feature brings the service back

2 questions please:

  1. is API neccessary to operate (if i dont want to send http status commands)?
  2. i can see that my red hat server is not listening on 5665, so cannot setup client?

Yes, it is required at least by the director

Icinga’s agent requires a port to connect and default is 5665/tcp.

Thanks, Roland

  1. I do not use Director, as the Linux server has no X module yet, just plain text
    Therefore I think i can leave API disabled, which is a good news since it causes some memory violation 139 codes

  2. Where should one search if Icinga is up and running but netstat shows no trace of listening on ports?

Regards

Icingaweb2 also requires API once the deprecated command pipe was removed

The port is closed when you disable API.

Many thanks

I enabled the API again and after icinga2.service restart, I get: “main process exited, code=exited, status=139/n/a”

> # systemctl status -l icinga2.service
> ● icinga2.service - Icinga host/service/network monitoring system
>    Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
>    Active: failed (Result: exit-code) since Wed 2020-09-02 07:24:45 UTC; 29s ago
>   Process: 122212 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=exited, status=139)
>   Process: 122205 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/sysconfig/icinga2 (code=exited, status=0/SUCCESS)
>  Main PID: 122212 (code=exited, status=139)
> 
> Sep 02 07:24:45  icinga2[122212]: [2020-09-02 07:24:45 +0000] information/ConfigItem: Instantiated 1 Endpoint.
> Sep 02 07:24:45  icinga2[122212]: [2020-09-02 07:24:45 +0000] information/ConfigItem: Instantiated 1 ApiUser.
> Sep 02 07:24:45  icinga2[122212]: [2020-09-02 07:24:45 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
> Sep 02 07:24:45  icinga2[122212]: [2020-09-02 07:24:45 +0000] information/ConfigItem: Instantiated 235 CheckCommands.
> Sep 02 07:24:45  icinga2[122212]: [2020-09-02 07:24:45 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
> Sep 02 07:24:45  icinga2[122212]: [2020-09-02 07:24:45 +0000] information/cli: Closing console log.
> Sep 02 07:24:45  systemd[1]: Started Icinga host/service/network monitoring system.
> Sep 02 07:24:45  systemd[1]: icinga2.service: main process exited, code=exited, status=139/n/a
> Sep 02 07:24:45  systemd[1]: Unit icinga2.service entered failed state.
> Sep 02 07:24:45  systemd[1]: icinga2.service failed.

Blind guess: Something wrong with your certificates. Did you run icinga2 node wizard or icinga2 api setup?

BTW: did you change your name here or did you reuse a thread?

  1. I reaused a thread, which i understand is not advised on this Forum, but am used to this approach to minimize number of threads… :wink:

  2. Yes, I did run api setup, and the root password is stored properly in api-users.conf

  3. Yes, I did run node wizard per below:

Master zone name [master]: agentbased
Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]: n
Do you want to disable the inclusion of the conf.d directory [Y/n]: y

Afterwards, the content of zones.conf is:

/*
 * Generated by Icinga 2 node setup commands
 * on 2020-08-28 13:58:32 +0000
 */

object Endpoint "linuxserver.domain.com" { }

object Zone "agentbased" {
        endpoints = [ "linuxserver.domain.com" ] }

object Zone "global-templates" {
        global = true }

object Zone "director-global" {
        global = true }

It’s not wanted and even turns into an disadvantage as users might have ignored this thread and you’ll get less attention.

Hmm, I’m getting more confused about your config (and this is one result of reusing a thread). You don’t need to run api setup and node wizard. Hence, I’d recommend to start a new thread and give more details about your setup and what you have already done.

Thanks Roland, will starte new one then:

139 error code for API - when trying to start icinga2 service in agent based mode