Unexpected certificate common name : New issue deploying linux agent while deploying satellite

ctrmint · February 19, 2020, 3:44pm

While working through my deployment of numerous satellite servers I’ve found one clean Ubuntu build is refusing to work as a remote Icinga agent.

I’m basically following the instructions here to add my satellite services.
https://blog.sleeplessbeastie.eu/2018/02/05/how-to-setup-icinga2-master-satellite-client-using-director-module/

These have worked really well so far.

As mentioned I’m adding another.

I’ve added the new satellite host into Director and deployed the configuration.
Then installed icinga2 onto the new Ubuntu clone.
run through the icinga2 node wizard, using the settings shown.
On the master, i’ve accepted the ca request using icinga2 ca sign ZXXXXXX etc
Restarted icinga2 on the new satellite.

I would normally progress and perform the satellite specific configuration.
However weirdly this hasn’t worked properly and the master is complaining.

This was the configuration wizard on the client

icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]:

Starting the Client/Satellite setup routine...

Please specify the common name (CN) [cyllene2]:

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): cyllene

Do you want to establish a connection to the parent node from this node? [Y/n]:
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): 192.x.x.x
Master/Satellite endpoint port [5665]:

Add more master/satellite endpoints? [y/N]:
Parent certificate information:

 Subject:     CN = cyllene
 Issuer:      CN = Icinga CA
 Valid From:  Jan 15 08:04:28 2020 GMT
 Valid Until: Jan 11 08:04:28 2035 GMT
 Fingerprint: 5C FB AF 35 80 87 BC 71 25 A8 AC C1 F1 B9 85 71 F8 5D 01 CB

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'cyllene2'):

No ticket was specified. Please approve the certificate signing request manually
on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details).

Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.

Done.

Now restart your Icinga 2 daemon to finish the installation!

CA was signed.


Fingerprint                                                      | Timestamp                | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------
7c6db1ed455de081209ac5aa7574d5214157fdf770d8a749f701f752bcb09e1a | Feb 19 15:36:34 2020 GMT |        | CN = cyllene2
XXXX@cyllene:/var/lib/icinga2/api/log# icinga2 ca sign 7c6db1ed455de081209ac5aa7574d5214157fdf770d8a749f701f752bcb09e1a
information/cli: Signed certificate for 'CN = cyllene2'.

The Master keeps reporting


[2020-02-19 15:40:19 +0000] information/JsonRpcConnection: Received certificate request for CN 'cyllene2' signed by our CA.
[2020-02-19 15:40:19 +0000] information/JsonRpcConnection: The certificate for CN 'cyllene2' is valid and uptodate. Skipping automated renewal.
[2020-02-19 15:40:28 +0000] information/ApiListener: Reconnecting to endpoint 'cyllene2.fqdn.com' via host '192.168.0.202' and port '5665'
[2020-02-19 15:40:28 +0000] warning/ApiListener: Unexpected certificate common name while connecting to endpoint 'cyllene2.fqdn.com': got 'cyllene2'
[2020-02-19 15:40:28 +0000] information/ApiListener: Finished reconnecting to endpoint 'cyllene2.fqdn,com' via host '192.168.0.202' and port '5665'

And the new client/satellite


        (0) Handling new API client connection

[2020-02-19 15:42:28 +0000] information/ApiListener: New client connection for identity 'cyllene' from [192.168.0.201]:40630
[2020-02-19 15:42:28 +0000] warning/ApiListener: No data received on new API connection for identity 'cyllene'. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
        (0) Handling new API client connection

I’m just repeating the same build process so I’m stuck why this is happening.
I’ve tried remove the host from Director and restarting the process. but I still keep getting the same issue. I also tried adding and removing the fqdn.

log1c · February 19, 2020, 3:52pm

This should be the problem if I am not mistaken.
You used the non-fqdn hostname in the setup wizard, but the server “announces” it self with the fqdn hostname at the master server when connecting.

Please re-run the setup wizard on the satellite with the fqdn again and see if that changes anything.

ctrmint · February 19, 2020, 3:57pm

Following that suggestion.

Satellite

icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]:

Starting the Client/Satellite setup routine...

Please specify the common name (CN) [cyllene2]: cyllene2.fqdn.com

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): cyllene

Do you want to establish a connection to the parent node from this node? [Y/n]: y
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): 192.168.0.201
Master/Satellite endpoint port [5665]:

Add more master/satellite endpoints? [y/N]:
Parent certificate information:

 Subject:     CN = cyllene
 Issuer:      CN = Icinga CA
 Valid From:  Jan 15 08:04:28 2020 GMT
 Valid Until: Jan 11 08:04:28 2035 GMT
 Fingerprint: 5C FB AF 35 80 87 BC 71 25 A8 AC C1 F1 B9 85 71 F8 5D 01 CB

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'cyllene2.fqdn.com'):

No ticket was specified. Please approve the certificate signing request manually
on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details).

Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...

Done.

Now restart your Icinga 2 daemon to finish the installation!
xxxxxx@cyllene2:/var/log/icinga2# service icinga2 restart

ReSigning on the master

xxxxx@cyllene:/# icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------
c8ba630fdafce9962022446238ba435cbc13238b6790f108164a3d5081f5ce0a | Feb 19 15:50:30 2020 GMT |        | CN = cyllene2.fqdn.com
xxxxxx@cyllene:/var/lib/icinga2/api/log# icinga2 ca sign c8ba630fdafce9962022446238ba435cbc13238b6790f108164a3d5081f5ce0a
information/cli: Signed certificate for 'CN = cyllene2.fqdn.com'.

ctrmint · February 19, 2020, 3:58pm

hmmm,
Just running the wizard again and the service isn’t starting.

log1c · February 19, 2020, 4:01pm

Did you create the satellite inside the Director without the fqdn as well?
then you will need to change the Directors config as well…
Also take a look at /var/lib/icinga2/api/ on the satellite

If there is stuff in it, remove it rm -rf /var/lib/icinga2/api/

ctrmint · February 19, 2020, 4:04pm

thanks, going to pull all the config and start again with it. Will post back in 10mins. This is weird though, I’ve done this on 3 remotes, this is actually a local satellite to off load processes. Odd its being difficult.

ctrmint · February 19, 2020, 4:14pm

Removing Satellite, icinga install

apt purge icinga2    
apt purge icinga2-common
apt autoremove

/var/lib/icinga2 was removed from above.

Entries relating to the host removed from Director, config rendered and installed.

Re-install
Setup Host in Director.

Host Template = Dummy
Hostname = cyllene2.fqdn.com
Display name = cyllene
Host address = 192.168.0.202
Cluster Zone = cyllene
Icinga Agent = Yes
Establish Connection = Yes
Accepts config = Yes

Install agent on Satellite
apt-get install icinga2

ctrmint · February 19, 2020, 4:18pm

Here is the new node wizard

icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]: y

Starting the Client/Satellite setup routine...

Please specify the common name (CN) [cyllene2]: cyllene2.fqdn.com

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): cyllene

Do you want to establish a connection to the parent node from this node? [Y/n]: y
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): 192.168.0.201
Master/Satellite endpoint port [5665]:

Add more master/satellite endpoints? [y/N]:
Parent certificate information:

 Subject:     CN = cyllene
 Issuer:      CN = Icinga CA
 Valid From:  Jan 15 08:04:28 2020 GMT
 Valid Until: Jan 11 08:04:28 2035 GMT
 Fingerprint: 5C FB AF 35 80 87 BC 71 25 A8 AC C1 F1 B9 85 71 F8 5D 01 CB

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'cyllene2.fqdn.com'):

No ticket was specified. Please approve the certificate signing request manually
on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details).

Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.

Done.

Now restart your Icinga 2 daemon to finish the installation

CA Sign on the master

icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------
f019f2d17cf3e3ad50e4b2b7a32bb5e490ae24ae150a105a2fb888a6fca027e9 | Feb 19 16:15:28 2020 GMT |        | CN = cyllene2.fqdn.com
xxx@cyllene:/var/log/icinga2# icinga2 ca sign f019f2d17cf3e3ad50e4b2b7a32bb5e490ae24ae150a105a2fb888a6fca027e9
information/cli: Signed certificate for 'CN = cyllene2.fqdncom'.

ctrmint · February 19, 2020, 4:23pm

Satellite log looks better now,

[2020-02-19 16:21:34 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'cyllene' in zone 'master'.
[2020-02-19 16:21:34 +0000] information/ApiListener: Sending replay log for endpoint 'cyllene' in zone 'master'.
[2020-02-19 16:21:34 +0000] information/ApiListener: Replayed 15 messages.
[2020-02-19 16:21:34 +0000] information/ApiListener: Finished sending replay log for endpoint 'cyllene' in zone 'master'.
[2020-02-19 16:21:34 +0000] information/ApiListener: Finished syncing endpoint 'cyllene' in zone 'master'.
[2020-02-19 16:21:34 +0000] information/ApiListener: Applying config update from endpoint 'cyllene' of zone 'master'.
[2020-02-19 16:21:44 +0000] information/WorkQueue: #5 (ApiListener, SyncQueue) items: 0, rate: 0.0166667/s (1/min 1/5min 1/15min);
[2020-02-19 16:21:44 +0000] information/WorkQueue: #4 (ApiListener, RelayQueue) items: 0, rate: 0.266667/s (16/min 16/5min 16/15min);
[2020-02-19 16:21:44 +0000] information/WorkQueue: #8 (JsonRpcConnection, #0) items: 0, rate: 0.05/s (3/min 3/5min 3/15min);
[2020-02-19 16:21:44 +0000] information/WorkQueue: #9 (JsonRpcConnection, #1) items: 0, rate:  0/s (0/min 0/5min 0/15min);

ctrmint · February 19, 2020, 4:25pm

guess that’s a noob mistake, thanks for pointing that one out.

ctrmint · February 19, 2020, 4:38pm

I’ve completed the setup and found the satellite starts producing similar errors again. Are these correct.
I have set the satellite to be in it’s own cluster zone.
???


        (0) Handling new API client connection

[2020-02-19 16:35:35 +0000] information/ApiListener: New client connection for identity 'cyllene' from [192.168.0.201]:43202
[2020-02-19 16:35:35 +0000] warning/ApiListener: No data received on new API connection for identity 'cyllene'. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
        (0) Handling new API client connection

[2020-02-19 16:35:45 +0000] information/ApiListener: New client connection for identity 'cyllene' from [192.168.0.201]:43236
[2020-02-19 16:35:45 +0000] warning/ApiListener: No data received on new API connection for identity 'cyllene'. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
        (0) Handling new API client connection

[2020-02-19 16:35:55 +0000] information/ApiListener: New client connection for identity 'cyllene' from [192.168.0.201]:43266
[2020-02-19 16:35:55 +0000] warning/ApiListener: No data received on new API connection for identity 'cyllene'. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
        (0) Handling new API client connection

[2020-02-19 16:36:05 +0000] information/ApiListener: New client connection for identity 'cyllene' from [192.168.0.201]:43276
[2020-02-19 16:36:05 +0000] warning/ApiListener: No data received on new API connection for identity 'cyllene'. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
        (0) Handling new API client connection

[2020-02-19 16:36:15 +0000] information/ApiListener: New client connection for identity 'cyllene' from [192.168.0.201]:43294
[2020-02-19 16:36:15 +0000] warning/ApiListener: No data received on new API connection for identity 'cyllene'. Ensure that the remote endpoints are properly configured in a cluster setup.

log1c · February 20, 2020, 7:07am

Yes, the satellite need its own zone and endpoint object.
Also the master needs to know both objects.

Please show zones.conf from the master and the satellite.

ctrmint · February 20, 2020, 8:35am

zones.d/cyllene/zones.conf from the Master, sourced from Director Rendered file.

object Zone "dmarcyllene" {
    parent = "cyllene"
    endpoints = [ "dmarcyllene" ]
}

object Zone "dmapcyllene" {
    parent = "cyllene"
    endpoints = [ "dmapcyllene" ]
}

object Zone "manilacyllene" {
    parent = "cyllene"
    endpoints = [ "manilacyllene" ]
}

object Zone "cyllene2" {
    parent = "cyllene"
    endpoints = [ "cyllene2" ]
}

ctrmint · February 20, 2020, 8:38am

The is the /etc/icinga2/zones.conf from the satellite.

/etc/icinga2# more zones.conf
/*
 * Generated by Icinga 2 node setup commands
 * on 2020-02-19 16:15:37 +0000
 */

object Endpoint "cyllene" {
        host = "192.168.0.201"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "cyllene" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

object Endpoint NodeName {
}

object Zone ZoneName {
        endpoints = [ NodeName ]
        parent = "master"
}

ctrmint · February 20, 2020, 8:51am

It appears the Master is reporting similar errors again.


[2020-02-20 08:50:18 +0000] information/ApiListener: Reconnecting to endpoint 'cyllene2' via host '192.168.0.202' and port '5665'
[2020-02-20 08:50:18 +0000] warning/ApiListener: Unexpected certificate common name while connecting to endpoint 'cyllene2': got 'cyllene2.fqdn.com'
[2020-02-20 08:50:18 +0000] information/ApiListener: Finished reconnecting to endpoint 'cyllene2' via host '192.168.0.202' and port '5665'

I thought I’d resolved this from the steps above.

log1c · February 20, 2020, 8:56am

What is the content behind the constants NodeName and ZoneName?
Look at the constants.conf file. I suspect this will be the full fqdn now, so you have to edit the masters config to match that.
As a rule of thumb, always use the FQDN for the endpoint name. The zone can be named as you like (location or what ever)

Also I don’t do the zone/endpoint configuration via the Director.
I do them manually inside the /etc/icinga2/zones.conf file and than import that into the Director via the kickstart wizard. I think that is also the recommended way.

ctrmint · February 20, 2020, 9:03am

On the Master server., /etc/icinga2/constants.conf

/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
 * This should be the common name from the API certificate.
 */
const NodeName = "cyllene"

/* Our local zone name. */
const ZoneName = "cyllene"

On the satellite


/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
 * This should be the common name from the API certificate.
 */
const NodeName = "cyllene2.fqdn.com"

/* Our local zone name. */
const ZoneName = "cyllene2.fqdn.com"

log1c · February 20, 2020, 10:37am

And, did changing the zones.conf on the master, to reflect these names, help?

ctrmint · February 20, 2020, 10:50am

I’ve not changed the masters, I’m concerned it might break those which are currently working as they expect a CN of cyllene not cyllene.fqdn.com. Am I worrying about nothing?

Also the sites which are working, seem to be configured as above.

log1c · February 20, 2020, 11:53am

I didn’t mean changing the masters configuration it self.

What I meant is:

ctrmint:

On the satellite

/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
 * This should be the common name from the API certificate.
 */
const NodeName = "cyllene2.fqdn.com"

/* Our local zone name. */
const ZoneName = "cyllene2.fqdn.com"

You have this on the satellite.
This has to be reflected by the masters zones.conf so they connect correctly.

So change the zones.conf on the master to contain

instead of the non-fqdn names.