CSR auto-signing fails silently if no ticket_salt is set in the ApiListener feature configuration

Icinga 2
, ,
2

You need to add the generated ticketnumber to --ticket

3

Sorry, my mistake. That’s actually – and unfortunately – simply a copy-paste error. The ticket number is there, I updated the examples above accordingly.
To make sure I’m not just imagining that, I just launched the command again to verify. (And I just reviewed my initial post and saw that the passing of the ticket was already visible in the client output I had pasted).

The following excerpt is just the icinga2 node setup part of a longer (-vv) ansible playbook run (cf. tentwentyfour/ansible-icinga2):

(Original output, I only added line-breaks for readability and removed the stdout_lines property; please note that the ticket is different because the CN differs from the example above)

changed: [centos4] => {"changed": true, "cmd": ["icinga2", "node", "setup", "--cn", "centos4", "--ticket", 
"7ad8d6e4b8c80dd603468280449977e565eba7c8", "--endpoint", "centos1", "--trustedcert", 
"/var/lib/icinga2/certs/trusted_parent.crt", "--zone", "centos4", "--parent_zone", "master", 
"--parent_host", "centos1", "--accept-commands", "--accept-config", "--disable-confd"], 
"delta": "0:00:00.496865", "end": "2019-07-28 09:08:51.519353", "rc": 0, 
"start": "2019-07-28 09:08:51.022488", "stderr": "", "stderr_lines": [], 
"stdout": "information/cli: Requesting certificate with ticket '7ad8d6e4b8c80dd603468280449977e565eba7c8'.
information/cli: Verifying parent host connection information: host 'centos1', port '5665'.
information/cli: Using the following CN (defaults to FQDN): 'centos4'.
information/cli: Created backup file '/var/lib/icinga2/certs//centos4.key.orig'.
information/cli: Created backup file '/var/lib/icinga2/certs//centos4.crt.orig'.
information/base: Writing private key to '/var/lib/icinga2/certs//centos4.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/certs//centos4.crt'.
information/cli: Verifying trusted certificate file '/var/lib/icinga2/certs/trusted_parent.crt'.
information/cli: Requesting a signed certificate from the parent Icinga node.
information/cli: Writing CA certificate to file '/var/lib/icinga2/certs//ca.crt'.
information/cli: !!!!!!
information/cli: !!! Certificate request for CN 'centos4' is pending. Waiting for approval from the parent Icinga instance.
information/cli: !!!!!!
information/cli: Disabling the Notification feature.
warning/cli: Feature 'notification' already disabled.
information/cli: Updating the ApiListener feature.
warning/cli: Feature 'api' already enabled.
information/cli: Backup file '/etc/icinga2/features-available/api.conf.orig' already exists. Skipping backup.
information/cli: Generating zone and object configuration.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
information/cli: Backup file '/etc/icinga2/zones.conf.orig' already exists. Skipping backup.
information/cli: Updating 'NodeName' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
information/cli: Updating 'ZoneName' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
information/cli: Make sure to restart Icinga 2.
information/cli: Updating '\"conf.d\"' include in '/etc/icinga2/icinga2.conf'.
information/cli: Backup file '/etc/icinga2/icinga2.conf.orig' already exists. Skipping backup.
information/cli: Make sure to restart Icinga 2.", 
[…]
}
4

Did you do a manual request before for that node? Looks like there is only a request there that is never signed. Use ‘icinga2 ca list’ on the first master to check it.

5

I’m not sure I understand exactly what you mean.

I’ve done a lot of test runs, and I’ve for instance made sure that no pending CSR – for this or any other host – existed on the master before running the new-cert, save-cert node setup combo (Either by just rm -rf /var/lib/icinga2/certificate_requests/* or by using the new icinga2 ca remove $cert in 2.11).

6

This one make me think there is something, thats why you should run icinga2 ca liston the master to verify if it is really waiting for approval or not.

This steps are not needed:

7

I made sure that icinga2 ca list would not show any pending CSRs for that particular host (other CNs, yes). Maybe I can record my terminals to show that, if you have doubts?

About new-cert and save-cert:
I didn’t have the new-cert step at the beginning, then re-read the documentation and added it, hoping it would make auto-signing finally work. (So the documentation goes through all 3 steps: https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#node-setup-with-satellitesclients).

P.S. did I really post this in the icingaweb2 section? Could a mod please move it into the correct section then?

8

Please retry without steps 1 & 2 and remove

"--trustedcert", 
"/var/lib/icinga2/certs/trusted_parent.crt"

from your script.

9

It doesn’t like that:

[root@centos4 ~]# icinga2 node setup \
>       --cn centos4 \
>       --ticket 7ad8d6e4b8c80dd603468280449977e565eba7c8 \
>       --endpoint centos1 \
>       --zone centos4 \
>       --parent_zone master \
>       --parent_host centos1 \
>       --accept-commands \
>       --accept-config \
>       --disable-confd
information/cli: Requesting certificate with ticket '7ad8d6e4b8c80dd603468280449977e565eba7c8'.
information/cli: Verifying parent host connection information: host 'centos1', port '5665'.
information/cli: Using the following CN (defaults to FQDN): 'centos4'.
information/base: Writing private key to '/var/lib/icinga2/certs//centos4.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/certs//centos4.crt'.
critical/cli: Please pass the trusted cert retrieved from the parent node (master or satellite)
(Hint: 'icinga2 pki save-cert --host <masterhost> --port <5665> --key local.key --cert local.crt --trustedcert parent.crt').

See https://asciinema.org/a/fooKgyhoN19O66kFQReyd8HkQ for the full transcript.
I’m going to create another one where I will do steps 1-3.

10

Here’s the full cast of a manual session doing steps 1-3 with a prior clean-up: https://asciinema.org/a/8HdDDivJ80L9R8rDCAdzVTYb3

11

Do what the hint tells you :slight_smile: before

12

Would you like me to repeat the process with only the save-cert and setup node steps? It’s what I’ve been doing all along, with the same results as you can see in the full cast. But I’ll gladly do it again if it helps?

1 Like
13

i justed checked some ansible scripts and i think you missed the request part. All do these steps for certificate:

  1. Node: icinga2 pki new-cert --cn centos4 --key /var/lib/icinga2/certs/centos4.key --cert /var/lib/icinga2/certs/centos4.crt
  2. Node: icinga2 pki save-cert --key /var/lib/icinga2/certs/centos4.key --cert /var/lib/icinga2/certs/centos4.crt --trustedcert /var/lib/icinga2/certs/trusted-master.crt --host centos1
  3. Master: icinga2 pki ticket --cn centos4
  4. Node: icinga2 pki request --host centos1 --ticket TICKETNUMBER_FROM_STEP_3 --key /var/lib/icinga2/certs/centos4.key --cert /var/lib/icinga2/certs/centos4.crt --trustedcert /var/lib/icinga2/certs/trusted-master.crt --ca /var/lib/icinga2/certs/ca.crt
14

Interesting… this is not mentioned in the docs for manual node set-up though, and to me it looks like this is the node set-up step broken apart into a PKI part and then the remaining zones.conf configuration.
I’ll try it in a few minutes.

15

Unfortunately, the results are the very same:

[root@centos4 ~]# icinga2 pki new-cert --cn centos4 \
>       --key /var/lib/icinga2/certs/centos4.key \
>       --cert /var/lib/icinga2/certs/centos4.crt
information/base: Writing private key to '/var/lib/icinga2/certs/centos4.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/certs/centos4.crt'.
[root@centos4 ~]# icinga2 pki save-cert \
>       --key /var/lib/icinga2/certs/centos4.key \
>       --cert /var/lib/icinga2/certs/centos4.crt \
>       --trustedcert /var/lib/icinga2/certs/trusted_parent.crt \
>       --host centos1
information/cli: Retrieving X.509 certificate for 'centos1:5665'.

 Subject:     CN = centos1
 Issuer:      CN = Icinga CA
 Valid From:  Jul 12 13:34:29 2019 GMT
 Valid Until: Jul  8 13:34:29 2034 GMT
 Fingerprint: 72 7B 5A 67 1D 94 62 A0 EA 7F 5C A1 5F B0 13 DD 74 66 62 3D

***
*** You have to ensure that this certificate actually matches the parent
*** instance's certificate in order to avoid man-in-the-middle attacks.
***

information/pki: Writing certificate to file '/var/lib/icinga2/certs/trusted_parent.crt'.
[root@centos4 ~]# icinga2 pki request \
>     --host centos1 \
>     --ticket 7ad8d6e4b8c80dd603468280449977e565eba7c8 \
>     --key /var/lib/icinga2/certs/centos4.key \
>     --cert /var/lib/icinga2/certs/centos4.crt \
>     --trustedcert /var/lib/icinga2/certs/trusted_parent.crt \
>     --ca /var/lib/icinga2/certs/ca.crt
information/cli: Writing CA certificate to file '/var/lib/icinga2/certs/ca.crt'.
information/cli: !!!!!!
information/cli: !!! Certificate request for CN 'centos4' is pending. Waiting for approval from the parent Icinga instance.
information/cli: !!!!!!

https://asciinema.org/a/IO208vjfjWeM5byPIiu811Sy3

16

On the master (centos1) in debuglog, I’m seeing this:

[2019-07-29 12:15:40 +0200] information/ApiListener: New client connection for identity 'centos4' from [10.24.1.191]:46200 (certificate validation failed: code 18: self signed certificate)
[2019-07-29 12:15:40 +0200] notice/ApiListener: New JSON-RPC client
[2019-07-29 12:15:40 +0200] notice/JsonRpcConnection: Received 'pki::RequestCertificate' message from 'centos4'
[2019-07-29 12:15:40 +0200] information/JsonRpcConnection: Received certificate request for CN 'centos4' not signed by our CA.
[2019-07-29 12:15:40 +0200] information/JsonRpcConnection: Certificate request for CN 'centos4' is pending. Waiting for approval.
[2019-07-29 12:15:40 +0200] notice/ApiListener: Relaying 'pki::RequestCertificate' message
[2019-07-29 12:15:41 +0200] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'centos4': Error: End of file


        (0) icinga2: icinga::JsonRpc::ReadMessage(std::shared_ptr<icinga::AsioTlsStream> const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) (+0x95) [0x95a1b5]
        (1) icinga2: icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) (+0x176) [0x9d31a6]
        (2) /usr/lib64/icinga2/sbin/icinga2() [0x9d3923]
        (3) libboost_context.so.1.69.0: make_fcontext (+0x2f) [0x7f0f611d118f]



[2019-07-29 12:15:41 +0200] warning/JsonRpcConnection: API client disconnected for identity 'centos4

The boost related thing seems to be something new to version 2.11. (I should probably downgrade back to 2.10.5 to make sure I don’t get any 2.11 bugs interfering with what is already enough of a conundrum.) But it comes after the message that the CSR is waiting for approval. So IMHO, at this point, the master has already decided not to use/trust the ticket it received and not to auto-sign the request.

17

That boost error with 2.11.0 is maybe something for @dnsmichi

18

Hi,

the boost error is only visible in the debug log, still could you maybe file a bug in the issue tracker? I’m not sure if this is a wanted behavior or actually a bug.

I just ran your steps to reproduce in a docker test setup (master and client) based on Debian buster and Icinga 2 v2.11 RC1, I had no issues. Since I had no DNS service in my test environment I used the IP addresses for connections. Can you verify if your DNS resolves properly to the expected host and maybe try IP addresses.

My steps:

Agent/Client:

icinga2 pki new-cert --cn $HOSTNAME \
--key /var/lib/icinga2/certs/$HOSTNAME.key \
--cert /var/lib/icinga2/certs/$HOSTNAME.crt

icinga2 pki save-cert --key /var/lib/icinga2/certs/$HOSTNAME.key \
--cert /var/lib/icinga2/certs/$HOSTNAME.crt \
--trustedcert /var/lib/icinga2/certs/trusted_parent.crt \
--host 172.17.0.2

Master:

icinga2 pki ticket --cn deb10-icinga2-client

Agent/Client:

icinga2 node setup \
--cn $HOSTNAME \
--ticket f1d062d85861a459f42325e6f546f5fc04f3afd0 \
--endpoint deb-icinga2-master \
--zone $HOSTNAME \
--parent_zone master \
--parent_host 172.17.0.2 \
--trustedcert /var/lib/icinga2/certs/trusted_parent.crt \
--accept-commands --accept-config --disable-confd

Best regards
Michael

19

Another thought:

How did you setup the Icinga 2 API on your master? Please verify and ensure that you set a TicketSalt in the API feature:

# vim /etc/icinga2/features-enabled/api.conf

/**
 * The API listener is used for distributed monitoring setups.
 */
object ApiListener "api" {

 ticket_salt = TicketSalt
}

Also ensure and verify the constant TicketSalt is set:

# vim /etc/icinga2/constants.conf

[...]
 
/* Secret key for remote node tickets */
const TicketSalt = "0f466570dd96c1c0a446937b55b7a7ad"

If no TicketSalt is set Icinga 2 is not able to use the auto signing feature.

Cheers
Michael

1 Like
20

Well I’ll be damned, that was it! :man_facepalming:

I was convinced I’d previously experienced that the icinga2 pki ticket command would fail if a TicketSalt wasn’t properly set. Turns out it was only half-set. As a constant, yes, it was set:

/**
 * This file defines global constants which can be used in
 * other configuration files.
 */

[…]
const NodeName = "centos1"
const ZoneName = "master"
const TicketSalt = "IitsWtw2geK895xmY8HXwnsOGwXltbE8QxMhCxFPEX2PxqK2smshm2yBiDMUh6DJ"

And I’m guessing that’s enough to satisfy the pki ticket command…
However, it wasn’t passed into the ApiListener feature configuration at all.
So I added it and… voilà, now auto-signing works :open_mouth:
I kinda also assumed for some reason that the ApiListener feature would pick it up automatically from the constant… don’t ask me why.

I was almost sure it must’ve been a PEBKAC error, but haven’t been able to figure it out for the life of it.

Now it’s always easy to say that, but IMHO there could have been two things to make debugging this less of a headache:

  1. Mentioning that the ApiListener configuration needs a ticket_salt set on https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#csr-auto-signing (I can perhaps offer a pull-request for the docs here?)
  2. Adding a note to the debug output on the master that the CSR couldn’t be signed because of X (it probably doesn’t even have to be very specific, just mentioning that it failed to sign the CSR because not all requirements are met might have pushed me in the right direction.)
21

Agreed on the docs, in terms of skipping the signing this is a little more complicated in this scope. If you’re interested, the code parts are located in lib/remote/jsonrpcconnection-pki.cpp.

Cheers,
Michael