How to get to the bottom of certificate problem?

Icinga 2
1

I am trying to get passive checks (in python3) to work, but I’m getting the following error:

Max retries exceeded with url: /v1/objects/hosts/enlil (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)')))

I have seen the other questions about this type of error, but I’d like to understand in some depth what the problem is - like, how does icinga match the certificates, keys etc. I set this up some years ago, but now that I try to do it again, it somehow eludes me, even when I follow the instructions in painstaking detail.

On the master:

root@vogon:~# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.6-1)

Copyright (c) 2012-2023 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Debian GNU/Linux
  Platform version: 11 (bullseye)
  Kernel: Linux
  Kernel version: 5.10.0-13-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 10.2.1
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 1.1.1n  15 Mar 2022

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

root@vogon:~# icinga2 feature list
Disabled features: compatlog debuglog elasticsearch gelf graphite influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command icingadb mainlog notification

root@vogon:~# icinga2 daemon -C
[2023-03-15 12:42:29 +0000] information/cli: Icinga application loader (version: r2.13.6-1)
[2023-03-15 12:42:29 +0000] information/cli: Loading configuration file(s).
[2023-03-15 12:42:29 +0000] information/ConfigItem: Committing config item(s).
[2023-03-15 12:42:29 +0000] information/ApiListener: My API identity: vogon.comind.io
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 3 Hosts.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 1 IcingaDB.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 6 Zones.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 4 Endpoints.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 245 CheckCommands.
[2023-03-15 12:42:29 +0000] information/ConfigItem: Instantiated 4 Services.
[2023-03-15 12:42:29 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-03-15 12:42:29 +0000] information/cli: Finished validating the configuration file(s).

root@vogon:~# icinga2 object list --type Endpoint
Object 'vogon.comind.io' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:33
  * __name = "vogon.comind.io"
  * host = ""
  * log_duration = 86400
  * name = "vogon.comind.io"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 33
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "vogon.comind.io" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:33
  * type = "Endpoint"
  * zone = ""

Object 'Vogon' of type 'Endpoint':
  % declared in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 6:1-6:23
  * __name = "Vogon"
  * host = "vogon.comind.io"
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 7:5-7:28
  * log_duration = 0
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 8:5-8:21
  * name = "Vogon"
  * package = "director"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 23
    * last_line = 6
    * path = "/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf"
  * templates = [ "Vogon" ]
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 6:1-6:23
  * type = "Endpoint"
  * zone = "vogon"

Object 'wap1' of type 'Endpoint':
  % declared in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 11:1-11:22
  * __name = "wap1"
  * host = "wap.comind.io"
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 12:5-12:26
  * log_duration = 0
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 13:5-13:21
  * name = "wap1"
  * package = "director"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 22
    * last_line = 11
    * path = "/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf"
  * templates = [ "wap1" ]
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 11:1-11:22
  * type = "Endpoint"
  * zone = "vogon"

Object 'Enlil' of type 'Endpoint':
  % declared in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 1:0-1:22
  * __name = "Enlil"
  * host = "enlil.comind.io"
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 2:5-2:28
  * log_duration = 0
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 3:5-3:21
  * name = "Enlil"
  * package = "director"
  * port = "5665"
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 22
    * last_line = 1
    * path = "/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf"
  * templates = [ "Enlil" ]
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_endpoints.conf', lines 1:0-1:22
  * type = "Endpoint"
  * zone = "vogon"

root@vogon:~# icinga2 object list --type Zone
Object 'Enlil' of type 'Zone':
  % declared in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 1:0-1:18
  * __name = "Enlil"
  * endpoints = [ "Enlil" ]
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 3:5-3:27
  * global = false
  * name = "Enlil"
  * package = "director"
  * parent = "vogon"
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 2:5-2:20
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 18
    * last_line = 1
    * path = "/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf"
  * templates = [ "Enlil" ]
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 1:0-1:18
  * type = "Zone"
  * zone = "vogon"

Object 'director-global' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 17:1-17:29
  * __name = "director-global"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 18:2-18:14
  * name = "director-global"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 17
    * last_column = 29
    * last_line = 17
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "director-global" ]
    % = modified in '/etc/icinga2/zones.conf', lines 17:1-17:29
  * type = "Zone"
  * zone = ""

Object 'Vogon' of type 'Zone':
  % declared in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 6:1-6:19
  * __name = "Vogon"
  * endpoints = [ "Vogon" ]
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 8:5-8:27
  * global = false
  * name = "Vogon"
  * package = "director"
  * parent = "vogon"
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 7:5-7:20
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 19
    * last_line = 6
    * path = "/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf"
  * templates = [ "Vogon" ]
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 6:1-6:19
  * type = "Zone"
  * zone = "vogon"

Object 'vogon' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 9:1-9:19
  * __name = "vogon"
  * endpoints = [ "vogon.comind.io" ]
    % = modified in '/etc/icinga2/zones.conf', lines 10:2-10:34
  * global = false
  * name = "vogon"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 9
    * last_column = 19
    * last_line = 9
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "vogon" ]
    % = modified in '/etc/icinga2/zones.conf', lines 9:1-9:19
  * type = "Zone"
  * zone = ""

Object 'wap1' of type 'Zone':
  % declared in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 11:1-11:18
  * __name = "wap1"
  * endpoints = [ "wap1" ]
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 13:5-13:26
  * global = false
  * name = "wap1"
  * package = "director"
  * parent = "vogon"
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 12:5-12:20
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 18
    * last_line = 11
    * path = "/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf"
  * templates = [ "wap1" ]
    % = modified in '/var/lib/icinga2/api/packages/director/78e4ba30-b15e-435a-ae16-dabb327db8bb/zones.d/vogon/agent_zones.conf', lines 11:1-11:18
  * type = "Zone"
  * zone = "vogon"

Object 'global-templates' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 13:1-13:30
  * __name = "global-templates"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 14:2-14:14
  * name = "global-templates"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 13
    * last_column = 30
    * last_line = 13
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "global-templates" ]
    % = modified in '/etc/icinga2/zones.conf', lines 13:1-13:30
  * type = "Zone"
  * zone = ""

On the client:

root@enlil:~/jan/tools# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.2-1)

Copyright (c) 2012-2023 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 22.04.1 LTS (Jammy Jellyfish)
  Kernel: Linux
  Kernel version: 5.15.0-58-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 11.2.0
  Build host: lcy02-amd64-054
  OpenSSL version: OpenSSL 3.0.2 15 Mar 2022

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

root@enlil:~/jan/tools# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker mainlog

root@enlil:~/jan/tools# icinga2 daemon -C
[2023-03-15 12:47:13 +0000] information/cli: Icinga application loader (version: r2.13.2-1)
[2023-03-15 12:47:13 +0000] information/cli: Loading configuration file(s).
[2023-03-15 12:47:13 +0000] information/ConfigItem: Committing config item(s).
[2023-03-15 12:47:13 +0000] information/ApiListener: My API identity: enlil.comind.io
[2023-03-15 12:47:13 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-03-15 12:47:13 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2023-03-15 12:47:13 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-03-15 12:47:13 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2023-03-15 12:47:13 +0000] information/ConfigItem: Instantiated 4 Zones.
[2023-03-15 12:47:13 +0000] information/ConfigItem: Instantiated 2 Endpoints.
[2023-03-15 12:47:13 +0000] information/ConfigItem: Instantiated 244 CheckCommands.
[2023-03-15 12:47:13 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-03-15 12:47:13 +0000] information/cli: Finished validating the configuration file(s).

root@enlil:~/jan/tools# icinga2 object list --type Endpoint
Object 'enlil.comind.io' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 15:1-15:33
  * __name = "enlil.comind.io"
  * host = ""
  * log_duration = 86400
  * name = "enlil.comind.io"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 15
    * last_column = 33
    * last_line = 15
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "enlil.comind.io" ]
    % = modified in '/etc/icinga2/zones.conf', lines 15:1-15:33
  * type = "Endpoint"
  * zone = ""

Object 'vogon.comind.io' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:33
  * __name = "vogon.comind.io"
  * host = "vogon.comind.io"
    % = modified in '/etc/icinga2/zones.conf', lines 7:2-7:25
  * log_duration = 86400
  * name = "vogon.comind.io"
  * package = "_etc"
  * port = "5665"
    % = modified in '/etc/icinga2/zones.conf', lines 8:2-8:14
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 33
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "vogon.comind.io" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:33
  * type = "Endpoint"
  * zone = ""

root@enlil:~/jan/tools# icinga2 object list --type Zone
Object 'global-templates' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 23:1-23:30
  * __name = "global-templates"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 24:2-24:14
  * name = "global-templates"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 23
    * last_column = 30
    * last_line = 23
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "global-templates" ]
    % = modified in '/etc/icinga2/zones.conf', lines 23:1-23:30
  * type = "Zone"
  * zone = ""

Object 'director-global' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 27:1-27:29
  * __name = "director-global"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 28:2-28:14
  * name = "director-global"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 27
    * last_column = 29
    * last_line = 27
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "director-global" ]
    % = modified in '/etc/icinga2/zones.conf', lines 27:1-27:29
  * type = "Zone"
  * zone = ""

Object 'vogon' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 11:1-11:19
  * __name = "vogon"
  * endpoints = [ "vogon.comind.io" ]
    % = modified in '/etc/icinga2/zones.conf', lines 12:2-12:34
  * global = false
  * name = "vogon"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 19
    * last_line = 11
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "vogon" ]
    % = modified in '/etc/icinga2/zones.conf', lines 11:1-11:19
  * type = "Zone"
  * zone = ""

Object 'enlil.comind.io' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 18:1-18:29
  * __name = "enlil.comind.io"
  * endpoints = [ "enlil.comind.io" ]
    % = modified in '/etc/icinga2/zones.conf', lines 19:2-19:34
  * global = false
  * name = "enlil.comind.io"
  * package = "_etc"
  * parent = "vogon"
    % = modified in '/etc/icinga2/zones.conf', lines 20:2-20:17
  * source_location
    * first_column = 1
    * first_line = 18
    * last_column = 29
    * last_line = 18
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "enlil.comind.io" ]
    % = modified in '/etc/icinga2/zones.conf', lines 18:1-18:29
  * type = "Zone"
  * zone = ""
2

please provide your python code, you most likely need to disable ssl verfication in your python request as your client can not verify the certificate

3

Thanks for helping, Moreamazingnick!

This is the command I try to run, for reference:

host create enlil debug

The source for host:

# cat host
#!/usr/bin/env python3

import sys
sys.path.append('..')
import icck
import socket

argl=len(sys.argv)
if argl<3:
    print('Usage: host {create|delete|exist|ack} node')
    sys.exit()
else:
    cmd=sys.argv[1]
    node=socket.getfqdn(sys.argv[2])
    dbg='debug' in sys.argv

h=icck.ichost(node,debug=dbg)

try:
    if cmd=='create':
        h.create()
        print(('%s created' % node))
    elif cmd=='delete':
        h.delete()
        print(('%s deleted' % node))
    elif cmd=='exist':
        h.exist()
        print(('%s exists' % node))
    elif cmd=='ack':
        h.acknowledge('admin','Manual acknowledgement')
        print(('%s acknowledged' % node))
    else:
        print('Unknown command')
except Exception as arg:
    print('%s failed:'%cmd,arg)

And the source for icck.py:

# cat icck.py
"""
@author j.andersen@imperial.ac.uk
@brief Base classes for all icinga-checks
@details
Icinga checks fall in two broad categories: service checks and host checks, as reflected by in the class names
in the following. Implement a new check by inheriting from one of these base classes.

"""

import subprocess
import requests
import json
import logging
import logging.handlers
import socket
import time
import random
import os

import icconst
import icutil

"""
Global variables
"""

cluster=''
zone='vogon'

"""
@class IcingaException

I use exceptions as a lazy way of handling errors from the ichost class, and catch them in icservice.
The ichost isn't really designed to be used independently, although it is possible, if this behaviour
is kept in mind.
"""
class IcingaException(RuntimeError):
    def __init__(self,arg):
        self.args=arg
        super(IcingaException,self).__init__(arg)

"""
@class ichost

class ichost contains methods to check the existence of a host object,
create a host object and delete a host object via the REST interface.
"""

class ichost(object):
    """
    @fn __init__(self,node=None,debug=False) - constructor
    @param node: FQDN of the node
    @param debug: debug output is printed if this parameter is True
    """
    def __init__(self,node=None,debug=False):
        self.debug=debug
        if node is None:
            self.objnm=socket.getfqdn()
        else:
            self.objnm=node
        if self.debug:
            print("host.init node: %s"%self.objnm)
        handler=logging.handlers.SysLogHandler(address = '/dev/log')
        format=logging.Formatter('%(asctime)s - %(module)s - %(message)s')
        handler.setFormatter(format)
        self.logger=logging.getLogger(__name__)
        self.logger.setLevel(logging.INFO)
        self.usernm=icconst.zone_parm['vogon']['user']
        self.passwd=icconst.zone_parm['vogon']['pasw']
        self.url='https://%s:%s/v1/objects/hosts/%s' % (
            icconst.zone_parm['vogon']['svr'],
            icconst.zone_parm['vogon']['port'],
            self.objnm
        )
        self.headers={'Accept':'application/json'}

    """
    @fn exist(self) - check that the host object exists on the icinga server
    @return True if the host object exists, False otherwise
    @exception IcingaException
    """
    def exist(self,master=True):
        try:
            if master:
                usernm=self.usernm
                passwd=self.passwd
                url=self.url
            else:
                zone_parm=icconst.zone_parm[zone]
                usernm=zone_parm['user']
                passwd=zone_parm['pasw']
                url='https://%s:%s/v1/objects/hosts/%s' % (
                    zone_parm['svr'],
                    zone_parm['port'],
                    self.objnm
                )
            if self.debug:
                print("host.exist: %s, %s:%s" % (self.url,self.usernm,self.passwd))
            r=requests.get(
                url,
                headers=self.headers,
                auth=(usernm,passwd)
            )
        except Exception as arg:
            print("host.exist failed: %s"%arg)
            return False
        else:
            if self.debug:
                print("host.exist: %s" % json.dumps(r.json()))
        return ('error' not in r.json())

    """
    @fn create(self)
    @return True if host object was created on the icinga server, throw IcingaException otherwise
    @exception IcingaException
    """
    def create(self):
        data={
            "name":self.objnm,
            "templates":['generic-host'],
            "attrs":{
                'address':self.objnm,
                'check_command':'ssh',
                'vars.agent':'ssh',
                'vars.cluster':cluster,
                'vars.os':'Linux',
                'zone':zone
            }
        }
        try:
            if self.debug:
              print('url: ',self.url)
              print('data: ',json.dumps(data))
            r=requests.put(
                self.url,
                headers=self.headers,
                auth=(self.usernm,self.passwd),
                data=json.dumps(data)
            )
        except Exception as arg:
            raise IcingaException('ichost CREATE of %s failed\n%s' % (self.url,arg))

        if self.debug:
            print("host.create: %s" % r.json())
        if 'error' not in r.json():
            return True
        else:
            raise IcingaException(r.json())

    """
    @fn delete(self)
    @return True if the host object was deleted from the icinga server, False otherwise
    @exception IcingaException
    """
    def delete(self):
        url='%s?cascade=1' % self.url
        try:
            r=requests.delete(
                url,
                headers=self.headers,
                auth=(self.usernm,self.passwd)
            )
            if self.debug:
                print("host.create: %s" % r.json())
        except Exception as arg:
            raise IcingaException('DELETE of %s failed\n%s' % (self.url,arg))
        if 'error' not in r.json():
            return True
        else:
            raise IcingaException(r.json())

    """
    @fn acknowledge(self,author,comment)
    @param author: string identifying who acknowledged the issue
    @param comment: describes the actions taken etc
    @return True if the host object was acknowledged successfully, False otherwise
    """
    def acknowledge(self,author,comment):
        url='https://%s:%s/v1/actions/acknowledge-problem?type=Host&host=%s'%(
            icconst.zone_parm['vogon']['svr'],
            icconst.zone_parm['vogon']['port'],
            self.objnm
        )
        if self.debug:
            print('host.ack url %s'%url)
        data={
            'author':author,
            'comment':comment
        }
        if self.debug:
            print('host.ack data',data)
        try:
            r=requests.post(
                url,
                headers=self.headers,
                auth=(self.usernm,self.passwd),
                data=json.dumps(data)
            )
            if self.debug:
                print('host.ack result',r)
        except Exception as arg:
            print('Acknowledge %s failed\n%s'%(self.objnm,arg))
            return False
        return '[200]' in r.json()

"""
@class icservice

This class is meant to be inherited by the actual service classes. It contains the
basic constructor and the run() function raises an exception - this simulates the
pure virtual methods in C++ in that it must be overridden to do something meaningful.

The _run() function contains the code that is common to all service checks, notably 
it run the external check command, and creates the service object in icinga, if needed.

The correct way to use the icservice class is to create a child class and override
the run() function. The child class' run() function should call the _run() function, 
and carry out its own processing of the results before calling report().

A new service object can only be created if the hostname part of the object name:

the.hostname.com!the-service-name

already exists as a host object, so the host object may have to be created as part of 
creating the service object. Furthermore, host and service objects can only be created
by the master zone (vogon), hence the 'm'-versions of usernm, passwd, and url

"""

class icservice(object):
    """
    @fn __init__(self,cknm,icobj,data,node=None,cmd=None,debug=False)
    @param cknm: a string, the name of the check, will be displayed in the web page
    @param icobj: a string - the icinga object name
    @param data: a dictionary describing the service - as documented in the icinga docs
    @param node: a string containing the FQDN part of the service object
    @param cmd: a string, the command that should be run - this can be a complex pipeline
    @param delay: an integer, choose a random number n: 0 < n < delay and sleep that many mins
    @param ttl: an integer, 'time to live', sets the retry_interval and check_interval for the service
    @param debug: a Boolean - True turns on debugging output
    @param silent: if True, suppress messages to stdout
    """
    def __init__(self,cknm,icobj,data,node=None,cmd=None,delay='0',ttl=1800,debug=False,silent=False):
        self.debug=debug
        if self.debug:
            print("cknm:%s, icobj: %s"%(cknm,icobj))
        if node is None:
            self.node=socket.getfqdn()
        else:
            self.node=socket.getfqdn(node)
        self.silent=silent
        self.ttl=ttl
        self.hname=self.node.split('.')[0]
        if not self.silent:
            print("%s %s delay: %s min"%(self.hname,icobj,delay))
        self.cmd=cmd
        self.delay=int(delay)*60
        self.checkname=cknm
        self.icobj=icobj
        self.zone=zone
        self.cluster=cluster
        self.obj='%s!%s'%(self.node,icobj)

        handler=logging.handlers.SysLogHandler(address = '/dev/log')
        format=logging.Formatter('%(asctime)s - %(module)s - %(message)s')
        handler.setFormatter(format)
        self.logger=logging.getLogger(__name__)
        self.logger.setLevel(logging.INFO)

        self.usernm=icconst.zone_parm[zone]['user']
        self.passwd=icconst.zone_parm[zone]['pasw']
        self.url='https://%s:%s/v1/actions/process-check-result?service=%s' % (
            icconst.zone_parm[zone]['svr'],
            icconst.zone_parm[zone]['port'],
            self.obj
        )

        self.musernm=icconst.zone_parm['vogon']['user']
        self.mpasswd=icconst.zone_parm['vogon']['pasw']
        self.murl='https://%s:%s/v1/objects/services/%s' % (
            icconst.zone_parm['vogon']['svr'],
            icconst.zone_parm['vogon']['port'],
            self.obj
        )
        self.headers={'Accept':'application/json'}
        if self.debug:
            print("icck data: ",data)
        self.data=data
        self.data['attrs']['display_name']=self.checkname
        self.data['attrs']['host_name']=socket.getfqdn()
        self.data['attrs']['zone']=self.zone

    """
    @fn exist(self): Check for the existence of the service on the icinga server
    @return True if the service object exists, False otherwise
    @exception all exceptions are handled
    """
    def exist(self,master=True):
        try:
            if master:
                r=requests.get(
                    self.murl,
                    headers=self.headers,
                    auth=(self.musernm,self.mpasswd)
                )
            else:
                r=requests.get(
                    self.url,
                    headers=self.headers,
                    auth=(self.usernm,self.passwd)
                )
                
        except Exception as arg:
            print('GET of %s failed\n%s' % (self.murl,arg))
            return False
        else:
            if self.debug:
                print("service.exist: %s\n" % r.json())
            return 'error' not in r.json()

    """
    @fn create(self): Create a service object as defined by the parameters to the constructor
    @return True if the service was created, False otherwise
    @exception all exceptions are handled
    """
    def create(self):
        try:
            host=ichost(self.node,self.debug)
            if not host.exist():
                host.create()
            self.data['attrs']['retry_interval']=str(self.ttl)
            self.data['attrs']['check_interval']=str(self.ttl)
            if self.debug:
                print("icservice.create\n\turl: %s\n\tdata:%s\n"%(self.murl,json.dumps(self.data)))
            if self.debug:
                print("icck create data:",self.data)
            r=requests.put(
                self.murl,
                headers=self.headers,
                auth=(self.musernm,self.mpasswd),
                data=json.dumps(self.data)
            )
        except Exception as arg:
            print('icservice CREATE of %s failed\n%s' % (self.murl,arg))
            return False
        else:
            if self.debug:
                print("service.create: %s\n" % r.json())
            return 'error' not in r.json()

    """
    @fn delete(self): Delete the service object from the icinga server
    @return True if the service was deleted, False otherwise
    @exception all exceptions are handled
    """
    def delete(self):
        try:
            r=requests.delete(
                self.murl,
                headers=self.headers,
                auth=(self.musernm,self.mpasswd)
            )
            if self.debug:
                print("service.delete: %s\n" % r.json())
        except Exception as arg:
            print('DELETE of %s failed\n%s' % (self.murl,arg))
        else:
            return 'error' not in r.json()

    """
    @fn run(self): This is a placeholder for the real run method that must be implemented by a child class
    @exception RuntimeError

    The correct way to use the icservice class is to create a child class and override
    the run() function. The child class' run() function should call the _run() function, 
    and carry out its own processing of the results before calling report().
    """
    def run(self):
        raise RuntimeError('Running base class icservice is not defined')

    """
    @fn _run(self): call the external command and capture return values
    @return (status,output): numerical status code and string output from the command execution
    @exception all exceptions are handled
    """
    def _run(self,cmd=None):
        if not self.silent:
            print("%s %s %s: _run sleeping %d secs"%(self.hname,self.icobj,time.ctime(),self.delay))
        if self.delay>0:
            time.sleep(self.delay)
        if not self.silent:
            print("%s %s %s: _run finished sleeping"%(self.hname,self.icobj,time.ctime()))
        if not self.exist():
            if not self.create():
                return (-1,"create() of  failed"%self.murl)
        if (self.cmd is None) and (cmd is None):
            status=3
            output='Undefined command - check your code'
        else:
            if cmd is None:
                #if not self.silent:
                if False:
                    (status,output)=subprocess.getstatusoutput('ps -ef | grep %s | grep -v grep | grep -v %d' % (self.checkname,os.getpid()))
                    if self.checkname in output:
                        status=1<<8
                        output='%s is already running (hanging?)' % self.checkname
                else:
                    (status,output)=subprocess.getstatusoutput(self.cmd)
            else:
                (status,output)=subprocess.getstatusoutput(cmd)
            if self.debug:
                print('_run self check - %s' % output)
            status>>=8      #The return value from the exit call in the cmd script is in the 2nd byte
        if self.debug:
            print("service._run: status: %d, output: %s\n\n" % (status,output))
        return (status,output)

    """
    @fn report(): POST check results to icinga server
    @param status: an integer, should be 0 (OK), 1 (WARNING), 2 (CRITICAL) or 3 (UNKNOWN)
    @param alertstr: string, up to several lines of status text
    @param perfdat: list of strings: ["label=value[UOM];[warn];[crit];[min];[max]", ...]
    @return void
    """
    def report(self,status,alertstr,perfdat=None):
        headers={
            'Accept':'application/json'
        }
        if perfdat==None:
            data={
                'exit_status':status,
                'plugin_output':alertstr,
                'check_command':[self.cmd]
            }
        else:
            data={
                'exit_status':status,
                'plugin_output':alertstr,
                'check_command':[self.cmd],
                'performance_data':perfdat
            }
        if self.debug:
            print('%s result: %s'%(self.cmd,alertstr))
        #self.logger.info('%s result: %s'%(self.cmd,alertstr))
        try:
            #print "service.report: %s, %s/%s" % (self.url,self.usernm,self.passwd)
            r=requests.post(
                self.url,
                headers=headers,
                auth=(self.usernm,self.passwd),
                data=json.dumps(data)
            )
        except Exception as arg:
            print('POST to %s failed\n%s' % (self.url,arg))
        else:
            if not self.silent:
                print('%s - service.report: %s' % (self.checkname,r))
#        self.logger.info('Reply from icinga: %s' % r)

"""
## class extservice

This class implements an icinga check by calling a script and using the returned
output as the update to the web API. It should simply be instantiated in a script
with the appropriate parameters, no need to inherit.

This is the simplest implementation of a service check, which assumes that the external
command already returns a status code and an output string that are meaningful to icinga.
"""
class extservice(icservice):
    """
    @fn __init__(self,cknm,icobj,data,cmd=None,debug=False)
    @param cknm: a string, the name of the check, will be displayed in the web page
    @param data: a dictionary describing the service - as documented in the icinga docs
    @param cmd: a string, the command that should be run - this can be a complex pipeline
    @param delay: an integer, choose a random number n: 0 < n < delay and sleep that many secs
    @param debug: a Boolean - True turns on debugging output
    """
    def __init__(self,cknm,icobj,data,cmd,delay=0,debug=False):
        super(extservice,self).__init__(cknm,icobj,data,cmd=cmd,delay=delay,debug=debug)

    """
    @fn run(self): Runs the command, reports the results to icinga
    @return void
    """
    def run(self):
        #self.logger.info('Calling %s'%self.cmd)
        (status,output)=self._run()
        alertstr="%s: %s\n\n%s\n" % (icconst.statstr.get(status,'UNKNOWN'),self.checkname,output)
        self.report(status,alertstr)

And finally, icconst.py:

# cat icconst.py
"""
@author j.andersen@imperial.ac.uk
@brief Constants used by the Icinga checks

"""

zone_parm={
    'vogon':{
        'user':'root',
        'pasw':'123456789abcd',
        'svr':'vogon.comind.io',
        'port':5665
    },
    'admin':{}
}

statstr={0:'OK',1:'WARNING',2:'CRITICAL'}    #Translate return code to string

default_data={
    'attrs':{
        'check_command': 'dummy',
        'enable_active_checks': '1',
        'vars.dummy_text': 'No passive check result received',
        'vars.dummy_state': '3',
        'max_check_attempts': '1',
        'retry_interval': '1800',
        'check_interval': '1800'
    }
}
4

the code lacks a way to verify the selfsigned certificate.
If you add the icinga2-master node public key to the machines certificate store (the pc where you run the script) this code might work

otherwise you can change every r=requests.post, requests.get requests.put and so on to something like that and add a verify=False

r=requests.post(
                self.url,
                headers=headers,
                auth=(self.usernm,self.passwd),
                data=json.dumps(data),
                verify=False
            )```
5

Yes - that works! And reading a bit more, I can actually use verify='/var/lib/icinga2/certs/ca.crt'. Thank you for pointing me in the right direction.

Now I can work on resolving the other errors, like why I get a response like:

["Error: Object 'testhost' of type 'Host' re-defined: in /var/lib/icinga2/api/packages/_api/704c2040-f83a-4d2a-af1b-413769af2216/conf.d/hosts/testhost.conf: 1:0-1:21; previous definition: in /var/lib/icinga2/api/packages/_api/704c2040-f83a-4d2a-af1b-413769af2216/conf.d/hosts/testhost.conf: 1:0-1:21\nLocation: in /var/lib/icinga2/api/packages/_api/704c2040-f83a-4d2a-af1b-413769af2216/conf.d/hosts/testhost.conf: 1:0-1:21"], 'status': 'Object could not be created.'}]

But that’s for another question.

6

you cant create a host that is already there

7

No, that makes sense, of course; but I tried to create another one, and get this response:

url:  https://vogon.comind.io:5665/v1/objects/hosts/testhost2
data:  {"name": "testhost2", "templates": ["generic-host"], "attrs": {"address": "testhost2", "check_command": "ssh", "vars.agent": "ssh", "vars.cluster": "", "vars.os": "Linux", "zone": "vogon"}}
host.create: {'results': [{'code': 500, 'errors': ["Error: Import references unknown template: 'generic-host'\nLocation: in /var/lib/icinga2/api/packages/_api/704c2040-f83a-4d2a-af1b-413769af2216/conf.d/hosts/testhost2.conf: 2:2-2:22"], 'status': 'Object could not be created.'}]}
testhost2 created

Which has me confused - was it created or not? Also, I don’t see that host in the web interface, so I don’t think it was.

9

Did that solve your certificate problem? There is a solution button for the thread…