warning/TlsStream: TLS stream was disconnected

I am using the REST API to create host and service objects in a setup with a master zone and two child zones; this I do by communicating directly to the master, and I can see the objects in the web interface as well as when I query the master throught the API. However, when I try to update a service in a child zone, I get a 404 back, and when I query the child zone with curl, I get the same.

I created a debug log, and found:

warning/TlsStream: TLS stream was disconnected

I only found one reference to this issue on the net, and it was to a site that is now down for maintenance. Is this an issue I can fix in some way?

Can you provide a little more context around this log line? The previous and future log lines should likely provide more insights on why the TLS stream got disconnected, maybe due to a handshake error, or authentification mismatch, or just by a timer closing stale connections. Maybe even the client did not properly close the connection.

Btw, which version of Icinga 2 is involved here? Any specific headers set in the curl request?

Cheers,
Michael

Sorry, here’s a few lines from the log - I think they capture the whole event:

[2019-02-14 10:15:25 +0000] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'zenoss.hpc.imperial.ac.uk'
[2019-02-14 10:15:27 +0000] information/ApiListener: New client connection from [192.168.96.134]:51328 (no client certificate)
[2019-02-14 10:15:27 +0000] notice/ApiListener: New HTTP client
[2019-02-14 10:15:27 +0000] debug/HttpRequest: line: POST /v1/actions/process-check-result?service=/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check HTTP/1.1, tokens: 3
[2019-02-14 10:15:27 +0000] notice/WorkQueue: Spawning WorkQueue threads for 'HttpServerConnection'
[2019-02-14 10:15:27 +0000] information/HttpServerConnection: Request: POST /v1/actions/process-check-result?service=/cx1-106-1-1.cx1.hpc.ic.ac.uk%21cx1-mom-check (from [192.168.96.134]:51328, user: client-pki-ticket-cx1-admin)
[2019-02-14 10:15:27 +0000] warning/TlsStream: TLS stream was disconnected.
[2019-02-14 10:15:27 +0000] debug/HttpServerConnection: Http client disconnected
[2019-02-14 10:15:27 +0000] notice/WorkQueue: Stopped WorkQueue threads for 'HttpServerConnection'
[2019-02-14 10:15:29 +0000] notice/JsonRpcConnection: Received 'log::SetLogPosition' message from 'zenoss.hpc.imperial.ac.uk'
[2019-02-14 10:15:30 +0000] information/WorkQueue: #4 (ApiListener, RelayQueue) items: 0, rate: 0.133333/s (8/min 8/5min 8/15min);
[2019-02-14 10:15:30 +0000] information/WorkQueue: #5 (ApiListener, SyncQueue) items: 0, rate: 0.0166667/s (1/min 1/5min 1/15min);
[2019-02-14 10:15:30 +0000] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 15; Checks/s: 0
[2019-02-14 10:15:30 +0000] notice/ApiListener: Setting log position for identity 'zenoss.hpc.imperial.ac.uk': 2019/02/13 16:47:02
[2019-02-14 10:15:30 +0000] information/WorkQueue: #8 (JsonRpcConnection, #0) items: 0, rate: 0.1/s (6/min 6/5min 6/15min);
[2019-02-14 10:15:30 +0000] information/WorkQueue: #10 (JsonRpcConnection, #2) items: 0, rate:  0/s (0/min 0/5min 0/15min);

Icinga version:

[root@cx1-admin home]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.8.1-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-693.11.6.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: unknown

The curl request:

bash-4.2$ curl -k -s -u cx1adminroot:fgadgsdfhetyd 'https://admin.cx1.hpc.imperial.ac.uk:5665/v1/objects/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check' | python -m json.tool
{
    "error": 404.0,
    "status": "No objects found."
}

And the update:

    def report(self,status,alertstr):
        # authentication
        # https://stackoverflow.com/questions/26745462/basic-authentication-not-working-with-requests-library
        headers={
            'Accept':'application/json'
        }
        data={
            'exit_status':status,
            'plugin_output':alertstr,
            'check_command':[self.cmd]
        }
        #self.logger.info('%s result: %s'%(self.cmd,alertstr))
        try:
            print "service.report: %s, %s/%s" % (self.murl,self.musernm,self.mpasswd)
            r=requests.post(
                self.murl,
                headers=headers,
                auth=(self.musernm,self.mpasswd),
                data=json.dumps(data)
            )
        except Exception,arg:
            print 'POST to %s failed\n%s' % (self.url,arg)
        else:
            print 'service.report: %s' % r

The python code used to work in the past, when the services used to be defined in /etc/icinga2/zones.d/cx1-zone/services.conf, so I feel it ought to work still.

Is that ApIUser intentional? This reads like a PKI handling user which may be missing the appropriate permissions. Please share the output from icinga2 object list --type ApiUser --name client-pki-ticket-cx1-admin and remove the password value.

The curl request below shows the usage of cx1adminroot. If that’s the real password after the colon, I’d advise you to change it immediately as being exposed now.

Does Icinga 2 know about this service, e.g. verified with icinga2 object list --type Service --name 'cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check'? If not, verify that it’s parent directory is included in icinga2.conf.

Cheers,
Michael

Thanks for all your help, Michael, I really appreciate it.

I don’t know why the username has ‘pki’ in it, that was set up before my time, although I have been modifying things. When I finish my test work, I am going to change usernames and passwords, but thank you highlighting that I have been careless (I have changed that password, BTW).

client-pki-ticket-cx1-admin on the master:

[root@zenoss cx1-zone]# icinga2 object list --type ApiUser --name client-pki-ticket-cx1-admin
Object 'client-pki-ticket-cx1-admin' of type 'ApiUser':
  % declared in '/etc/icinga2/conf.d/api-users.conf', lines 15:1-15:44
  * __name = "client-pki-ticket-cx1-admin"
  * client_cn = ""
  * name = "client-pki-ticket-cx1-admin"
  * package = "_etc"
  * password_hash = .....
  * permissions = [ "actions/process-check-result" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 17:3-17:50
  * source_location
    * first_column = 1
    * first_line = 15
    * last_column = 44
    * last_line = 15
    * path = "/etc/icinga2/conf.d/api-users.conf"
  * templates = [ "client-pki-ticket-cx1-admin" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 15:1-15:44
  * type = "ApiUser"
  * zone = ""

same, on the child:

[root@cx1-admin home]# icinga2 object list --type ApiUser --name client-pki-ticket-cx1-admin
Object 'client-pki-ticket-cx1-admin' of type 'ApiUser':
  % declared in '/etc/icinga2/conf.d/api-users.conf', lines 6:1-6:44
  * __name = "client-pki-ticket-cx1-admin"
  * client_cn = ""
  * name = "client-pki-ticket-cx1-admin"
  * package = "_etc"
  * password = ......
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 7:3-7:55
  * permissions = [ "actions/process-check-result", "objects/create/host", "objects/delete/host", "objects/create/service", "objects/delete/service" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 8:3-8:148
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 44
    * last_line = 6
    * path = "/etc/icinga2/conf.d/api-users.conf"
  * templates = [ "client-pki-ticket-cx1-admin" ]
    % = modified in '/etc/icinga2/conf.d/api-users.conf', lines 6:1-6:44
  * type = "ApiUser"
  * zone = ""

The master knows the service object, but the child zone doesn’t, when checked with your command.

Hi,

doing my best, I have been offline providing a 2-day-GitLab training :slightly_smiling_face:

None of the above listed permissions include objects/query/Service for example. That’s why your GET request with curl fails in the first place. You’ll need to fix that to debug further.

How’s that initialized in your Python script?

Can you post the output on the master from that CLI command?

Cheers,
Michael

I’ll look into the permission issue in a moment.

The value of self.murl is

https://admin.cx1.hpc.imperial.ac.uk:5665/v1/actions/process-check-result?service=/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check

The output:

Object 'cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check' of type 'Service':
  % declared in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 1:0-1:29
  * __name = "cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check"
  * action_url = ""
  * check_command = "passive"
    % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 2:2-2:26
  * check_interval = 300
    % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 3:2-3:23
  * check_period = ""
  * check_timeout = null
  * command_endpoint = ""
  * display_name = "cx1-mom-check"
  * enable_active_checks = true
    % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 4:2-4:27
  * enable_event_handler = true
  * enable_flapping = false
  * enable_notifications = true
  * enable_passive_checks = true
  * enable_perfdata = true
  * event_command = ""
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * host_name = "cx1-106-1-1.cx1.hpc.ic.ac.uk"
    % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 5:2-5:43
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 1
    % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 6:2-6:25
  * name = "cx1-mom-check"
  * notes = ""
  * notes_url = ""
  * package = "_api"
  * retry_interval = 300
    % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 7:2-7:23
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 29
    * last_line = 1
    * path = "/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf"
  * templates = [ "cx1-mom-check" ]
    % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 1:0-1:29
  * type = "Service"
  * vars
    * dummt_text = "No passive check result received"
      % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 8:2-8:56
    * dummy_state = "3"
      % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 9:2-9:26
  * volatile = false
  * zone = "master"
    % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 11:2-11:16

(spotted a spelling mistake in the vars, BTW: dummt_text)

That’s a runtime API created object then, and its zone attribute is set to master. That’s why it is not synced to the satellite zone then thus not being available via API there. Therefore the error on the satellite for processing a check result.

Coming back to the original question, this likely is your problem :slight_smile:

Cheers,
Michael

I believe you. However, when I create the service, it ignores the value I give for the zone:

data={
    'attrs':{
        'check_command': 'passive',
        'enable_active_checks': '1',
        'vars.dummy_text': 'No passive check result received',
        'vars.dummy_state': '3',
        'max_check_attempts': '1',
        'retry_interval': '300',
        'check_interval': '300'
    }
}

svc=icck.icservice(
    'cx1-mom-check',
    'cx1-mom-check',
    data
)

and in the constructor:

class icservice(object):
    def __init__(self,cknm,icobj,data,cmd=None):
...
        self.headers={'Accept':'application/json'}
        self.data=data
        self.data['attrs']['display_name']=self.checkname
        self.data['attrs']['host_name']=socket.getfqdn()
        self.data['attrs']['zone']=self.zone

Here, self.zone has been set to cx1-zone. Isn’t this right?

Actually, when I dump the service object on the master, it mentions where it gets the data:

[root@zenoss cx1-zone]# icinga2 object list --type Service --name 'cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check' 
Object 'cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check' of type 'Service':
  % declared in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 1:0-1:29
...
  * volatile = false
  * zone = "master"
    % = modified in '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf', lines 11:2-11:16

and this is what is in that file:

[root@zenoss cx1-zone]# cat '/var/lib/icinga2/api/packages/_api/zenoss.hpc.ic.ac.uk-1520336949-1/conf.d/services/cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf'
object Service "cx1-mom-check" {
        check_command = "passive"
        check_interval = "300"
        display_name = "cx1-mom-check"
        enable_active_checks = "1"
        host_name = "cx1-106-1-1.cx1.hpc.ic.ac.uk"
        max_check_attempts = "1"
        retry_interval = "300"
        vars["dummy_state"] = "3"
        vars["dummy_text"] = "No passive check result received"
        version = 1550151210.674131
        zone = "cx1-zone"
}

That seems distinctly odd to me.

That’s likely a problem with caching. icinga2 object list reads from a file called icinga2.debug which is only updated on a) restart/reload b) config validation successful. It won’t be updated from runtime created objects as this information is only available when reading the entire configuration, compile it and extract the debug info from the config items.

That being said, in order to update the cache and use object list after creating an object at runtime, you’ll need to run icinga2 daemon -C up front once.

Btw if no zone attribute is provided (which is the default), Icinga 2 takes the current local zone if configured. That way objects created on master A are automatically synced to master B in case.

Cheers,
Michael

OK, the zone now shows up correctly, but the service is not visible in the child zone, at least not until I do a icinga2 daemon -C. Should this not propagate out to the zones automatically (or am I just too impatient)?

Can you hint how the zone hierarchy is built in your master’s zones.conf? Is cx1-zone a satellite zone beneath the master, or is it something else?

I’m a little confused about the distinction between satelite and child, but this is how it looks in zones.conf:

object Zone "master" {
  endpoints = [ "zenoss.hpc.imperial.ac.uk" ]
}

object Zone "cx1-zone" {
  parent = "master"
  endpoints = [ "admin.cx1.hpc.imperial.ac.uk" ]
}

So, master and the child zone cx1-zone referred to as satellite role here in my definition :slight_smile:

Coming back to the analysis, the _api package says it has the correct zone, but object list says it hasn’t. The last resort is now … avoiding curl, but you need an ApiUser with full root permissions * now.

icinga2 console --connect 'https://root:password@admin.cx1.hpc.imperial.ac.uk:5665/'

get_object("cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf")

This should print the object’s attribute to the screen.

Cheers,
Michael

It didn’t seem too impressed :slight_smile: :

<1> => get_object("cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf")
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<1>: get_object("cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf")
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error while evaluating expression: Too few arguments for function.

Ah, still tired today. I’m using this function.

get_object(Service, "cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf")
<4> => get_object(Service,"cx1-106-1-1.cx1.hpc.ic.ac.uk!cx1-mom-check.conf")
null

Oh, actually, I changed the name this morning, but the result remains the same:

<5> => get_object(Service,"cx1-106-1-1.cx1.hpc.ic.ac.uk!MOM-check.conf")
null

But funny enough, now I can update that service.