Icinga2 Server
Not update from 2 Aug
zones.conf on master nodes:
`/*
- Generated by Icinga 2 node setup commands
- on 2019-07-25 18:54:09 +0300
*/
object Endpoint “master2.icinga2.hostkey.ru” {
host = “master2.icinga2.hostkey.ru”
port = “5665”
}
object Endpoint “master1.icinga2.hostkey.ru” {
host = “master1.icinga2.hostkey.ru”
port = “5665”
}
object Zone “master” {
endpoints = [ “master2.icinga2.hostkey.ru”, “master1.icinga2.hostkey.ru” ]
}
object Endpoint “monitoring.hostkey.ru” {
}
object Zone “monitoring.hostkey.ru” {
endpoints = [ “monitoring.hostkey.ru” ]
parent = “master”
}
object Zone “global-templates” {
global = true
}
object Zone “director-global” {
global = true
}
`
After 2-3 mon backend down
CentOS Linux release 7.6.1810 (Core
icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)
Copyright © 2012-2019 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: CentOS Linux
Platform version: 7 (Core)
Kernel: Linux
Kernel version: 3.10.0-957.10.1.el7.x86_64
Architecture: x86_64
Build information:
Compiler: GNU 4.8.5
Build host: afcad697467e
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
icinga2 feature list
Disabled features: compatlog debuglog gelf graphite influxdb notification opentsdb perfdata
Enabled features: api checker command elasticsearch ido-pgsql livestatus mainlog statusdata syslog
icinga2 daemon -C
[2019-08-20 10:38:17 +0300] information/cli: Icinga application loader (version: r2.10.5-1)
[2019-08-20 10:38:17 +0300] information/cli: Loading configuration file(s).
[2019-08-20 10:38:17 +0300] information/ConfigItem: Committing config item(s).
[2019-08-20 10:38:17 +0300] information/ApiListener: My API identity: monitoring.hostkey.ru
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 726 Services.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 LivestatusListener.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 SyslogLogger.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 140 Hosts.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 FileLogger.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 10 NotificationCommands.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1741 Notifications.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 ElasticsearchWriter.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 5 HostGroups.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 ApiListener.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 10 Downtimes.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 34 Comments.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 CheckerComponent.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 97 Zones.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 StatusDataWriter.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 96 Endpoints.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 6 ApiUsers.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 6 Users.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 215 CheckCommands.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 1 IdoPgsqlConnection.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 2 UserGroups.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 5 ServiceGroups.
[2019-08-20 10:38:18 +0300] information/ConfigItem: Instantiated 3 TimePeriods.
[2019-08-20 10:38:18 +0300] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2019-08-20 10:38:18 +0300] information/cli: Finished validating the configuration file(s).
Object ‘ug.hostkey.nl’ of type ‘Endpoint’:
% declared in ‘/etc/icinga2/repository.d/endpoints/ug.hostkey.nl.conf’, lines 1:0-1:30
- __name = “ug.hostkey.nl”
- host = “”
- log_duration = 86400
- name = “ug.hostkey.nl”
- package = “_etc”
- port = “5665”
- source_location
- first_column = 0
- first_line = 1
- last_column = 30
- last_line = 1
- path = “/etc/icinga2/repository.d/endpoints/ug.hostkey.nl.conf”
- templates = [ “ug.hostkey.nl” ]
% = modified in ‘/etc/icinga2/repository.d/endpoints/ug.hostkey.nl.conf’, lines 1:0-1:30 - type = “Endpoint”
- zone = “”
Client Object ‘ug.hostkey.nl’ of type ‘Endpoint’:
icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)
Copyright © 2012-2019 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: CentOS Linux
Platform version: 7 (Core)
Kernel: Linux
Kernel version: 3.10.0-862.2.3.el7.x86_64
Architecture: x86_64
systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-08-19 15:10:21 MSK; 19h ago
Process: 28365 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/sysconfig/icinga2 (code=exited, status=0/SUCCESS)
Main PID: 28373 (icinga2)
CGroup: /system.slice/icinga2.service
├─28373 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
└─28406 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
Aug 19 15:10:21 ug.hostkey.nl icinga2[28373]: [2019-08-19 15:10:21 +0300] information/ConfigObject: Restoring program state from file ‘/var/lib/icinga2/icinga2.state’
Aug 19 15:10:21 ug.hostkey.nl icinga2[28373]: [2019-08-19 15:10:21 +0300] information/ConfigObject: Restored 262 objects. Loaded 3 new objects without state.
Aug 19 15:10:21 ug.hostkey.nl icinga2[28373]: [2019-08-19 15:10:21 +0300] information/ConfigItem: Triggering Start signal for config items
Aug 19 15:10:21 ug.hostkey.nl icinga2[28373]: [2019-08-19 15:10:21 +0300] information/FileLogger: ‘main-log’ started.
Aug 19 15:10:21 ug.hostkey.nl icinga2[28373]: [2019-08-19 15:10:21 +0300] information/ApiListener: ‘api’ started.
Aug 19 15:10:21 ug.hostkey.nl icinga2[28373]: [2019-08-19 15:10:21 +0300] information/ApiListener: Started new listener on ‘[0.0.0.0]:5665’
Aug 19 15:10:21 ug.hostkey.nl icinga2[28373]: [2019-08-19 15:10:21 +0300] information/CheckerComponent: ‘checker’ started.
Aug 19 15:10:21 ug.hostkey.nl icinga2[28373]: [2019-08-19 15:10:21 +0300] information/ConfigItem: Activated all objects.
Aug 19 15:10:21 ug.hostkey.nl icinga2[28373]: [2019-08-19 15:10:21 +0300] information/cli: Closing console log.
Aug 19 15:10:21 ug.hostkey.nl systemd[1]: Started Icinga host/service/network monitoring system.
Master2
master2
`service icinga2 status
Redirecting to /bin/systemctl status icinga2.service
● icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-08-19 14:21:48 MSK; 20h ago
Process: 22441 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/sysconfig/icinga2 (code=exited, status=0/SUCCESS)
Main PID: 22449 (icinga2)
CGroup: /system.slice/icinga2.service
├─19698 /usr/lib64/nagios/plugins/check_ping -H 8.8.8.8 -c 5000,100% -w 3000,80%
├─19699 /usr/bin/ping -n -U -W 30 -c 5 8.8.8.8
├─22449 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
└─22484 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: API client disconnected for identity ‘foreman.hostkey.ru’
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: Removing API client for endpoint ‘foreman.hostkey.ru’. 1 API clients left.
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: Removing API client for endpoint ‘foreman.hostkey.ru’. 1 API clients left.
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: API client disconnected for identity ‘chat.hostkey.ru’
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: API client disconnected for identity ‘chat.hostkey.ru’
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: Removing API client for endpoint ‘chat.hostkey.ru’. 1 API clients left.
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: Removing API client for endpoint ‘chat.hostkey.ru’. 1 API clients left.
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: API client disconnected for identity ‘27785.hostkey.com_dev.hostkey.com’
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: API client disconnected for identity ‘27785.hostkey.com_dev.hostkey.com’
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: Removing API client for endpoint ‘27785.hostkey.com_dev.hostkey.com’. 1 API clients left.
Aug 20 10:46:42 master2.icinga2.hostkey.ru icinga2[22449]: Removing API client for endpoint ‘27785.hostkey.com_dev.hostkey.com’. 1 API clients left.
`
icinga2 --version icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.4-1)
icinga2 daemon -C [2019-08-20 10:49:04 +0300] information/cli: Icinga application loader (version: r2.10.4-1) [2019-08-20 10:49:04 +0300] information/cli: Loading configuration file(s). [2019-08-20 10:49:04 +0300] information/ConfigItem: Committing config item(s). [2019-08-20 10:49:04 +0300] information/ApiListener: My API identity: master2.icinga2.hostkey.ru [2019-08-20 10:49:05 +0300] warning/ApplyRule: Apply rule 'Sping4' (in /etc/icinga2/zones.d/satelliteHV2/services.conf: 1:0-1:21) for type 'Service' does not match anywhere! [2019-08-20 10:49:05 +0300] warning/ApplyRule: Apply rule 'Sdisk' (in /etc/icinga2/zones.d/satelliteHV2/services.conf: 6:1-6:21) for type 'Service' does not match anywhere! [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 ScheduledDowntime. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 825 Services. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 LivestatusListener. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 SyslogLogger. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 IcingaApplication. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 154 Hosts. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 FileLogger. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 2 NotificationCommands. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 855 Notifications. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 NotificationComponent. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 5 HostGroups. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 ApiListener. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 Downtime. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 15 Comments. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 CheckerComponent. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 113 Zones. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 StatusDataWriter. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 ExternalCommandListener. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 113 Endpoints. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 4 ApiUsers. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 User. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 215 CheckCommands. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 IdoPgsqlConnection. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 1 UserGroup. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 3 ServiceGroups. [2019-08-20 10:49:05 +0300] information/ConfigItem: Instantiated 3 TimePeriods. [2019-08-20 10:49:05 +0300] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' [2019-08-20 10:49:05 +0300] information/cli: Finished validating the configuration file(s).
Logs from master 1
(0) Handling new API client connection
[2019-08-20 15:24:32 +0300] information/ApiListener: New client connection for identity ‘ug.hostkey.nl’ from [158.255.6.219]:59822
[2019-08-20 15:24:32 +0300] warning/ApiListener: No data received on new API connection for identity ‘ug.hostkey.nl’. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
(0) Handling new API client connection
[2019-08-20 15:24:33 +0300] information/ApiListener: New client connection for identity ‘ovr1.hostkey.us’ from [139.60.160.147]:46858
[2019-08-20 15:24:33 +0300] information/ApiListener: New client connection for identity ‘logger’ from [91.210.104.91]:51050
[2019-08-20 15:24:33 +0300] warning/ApiListener: No data received on new API connection for identity ‘logger’. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
(0) Handling new API client connection
[2019-08-20 15:24:33 +0300] information/ApiListener: New client connection for identity ‘ovr2.hostkey.com’ from [5.39.217.38]:43474
[2019-08-20 15:24:33 +0300] warning/ApiListener: No data received on new API connection for identity ‘ovr1.hostkey.us’. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
(0) Handling new API client connection
[2019-08-20 15:24:33 +0300] information/ApiListener: New client connection for identity ‘cp.hostbuy.com’ from [5.39.219.205]:56552
[2019-08-20 15:24:33 +0300] warning/ApiListener: No data received on new API connection for identity ‘ovr2.hostkey.com’. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
(0) Handling new API client connection
[2019-08-20 15:24:33 +0300] warning/ApiListener: No data received on new API connection for identity ‘cp.hostbuy.com’. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
(0) Handling new API client connection
[2019-08-20 15:24:33 +0300] information/ApiListener: New client connection for identity ‘ntp.hostkey.us’ from [139.60.160.90]:36354
[2019-08-20 15:24:33 +0300] information/ApiListener: New client connection for identity ‘smtp.hostkey.ru’ from [158.255.3.75]:41238
[2019-08-20 15:24:33 +0300] warning/ApiListener: No data received on new API connection for identity ‘smtp.hostkey.ru’. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
(0) Handling new API client connection
Please help. Monitoring is very critical for me.