Own CA for Icinga Cluster/API communication?

Icinga 2
, , ,
1

Every now and then you may be required to answer the question on “why not use the company’s CA certificates” or “The REST API is not trusted in my browser, fix it”.

Here’s some ideas and pros and cons. Please discuss your own findings below :slight_smile:

Matter of Trust

Icinga as an application with masters, satellites and agents should be used in its own CA environment. Once you open this up by e.g. re-using the Puppet CA, you’ll end up with two applications and security layers (Icinga and Puppet) opening the trust levels.

Some environments also have the idea or requirement to re-use the company CA and open this up for any installation and application in the local (or Internet) network.

What’s your experience with that, how often does this requirement occur, how do you solve?

Own CA and certificates

Keep in mind that the ApiListener object is used for two mechanisms:

  • Cluster API with JSON-RPC messages and TLS handshakes for endpoints
  • REST API where HTTP clients/browsers may connect against

By default, the CLI wizard tools will create a custom CA with self-signed certificates. Meaning to say, when an HTTP client connects, the CA is not trusted by the root CA and as such your browser marks the website as “not trusted” when connecting via https protocol.

Own Company CA

You can use your company’s CA for example when there’s a deployment mechanism for all hosts to add them to the trusted store.

Certificate Requirements

CA

  • Version 3
  • Extensions
    • x509v3 Basic Contraints: CA:TRUE

A CA generated from the Icinga CLI commands looks like this:

$ openssl x509 -in ca.crt -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            bb:d0:6d:ad:d9:b0:d3:96:f1:c4:25:6c:d7:2d:33:e0:77:f8:bd:93
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Icinga CA
        Validity
            Not Before: Jan  4 14:32:57 2019 GMT
            Not After : Dec 31 14:32:57 2033 GMT
        Subject: CN=Icinga CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:cc:7d:ec:37:83:b9:7d:07:98:3e:53:05:3f:11:
                    .....

                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         22:0c:d6:38:ea:6a:7c:62:8a:65:a6:dd:cb:4d:d3:9c:db:87:
         .......

Signed Certificates

  • Version 3
  • Subject: CN=<FQDN>
    • This is mandatory and will be used after TLS handshake for CN == endpoint object security.
  • Extensions
    • x509v3 Basic Contraints: CA:FALSE
    • x509v3 Subject Alternative Name: DNS:<FQDN>

A signed certificate from the CLI commands and auto-signing within Icinga (CA proxy, CSR autosigning) looks like this:

openssl x509 -in mbpmif.int.netways.de.crt -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            80:5c:70:7e:cf:e5:dc:04:2c:66:1b:e6:c8:e2:1d:f9:e2:38:18:46
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Icinga CA
        Validity
            Not Before: Jan  4 14:38:12 2019 GMT
            Not After : Dec 31 14:38:12 2033 GMT
        Subject: CN=mbpmif.int.netways.de
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:da:94:48:be:b0:f5:00:73:0f:70:fb:06:d4:cd:
                    ....
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name:
                DNS:mbpmif.int.netways.de
    Signature Algorithm: sha256WithRSAEncryption
         1d:4f:4c:5c:2c:d0:ea:8b:d9:74:5b:e9:0c:17:18:fa:fc:c8:
         ....

How do you generate/use such certificates, does it work for you?

Any parts you’d like to see improved for Icinga?

Cheers,
Michael

1 Like
2

TL:DR: You can replace the CA with your own if you know, what you’re doing, but don’t do it. For your own good.

The detailed version:

From a partners support engineer point of view I want to add the following: Don’t use another CA than the one Icinga brings.

We had a lot of tickets with hard to debug problems which came down to someone messing up the certificate issueing process. While it’s technically true that you can issue your own certificates in a way Icinga can use them, it’s rather easy (and common) to create ones which might not work.

All in all you’ll have a lot of extra work to do for (almost) no other benefit than ticking off a checkbox in your policy papers. Normally there are only three kinds of connection to the API:

  • other Icinga instances which play well with the internal CA
  • external tools/scripts which would need the CAs certificate once and are then good to go
  • users using tools like curl which should be “technically-minded” enough to understand and be able to deal with certificates signed by the Icinga CA

All of them can use the Icinga CA. Normally the only benefit you get from using your companys CA is that users already have that CAs certificate imported into their browsers certificate store. This prevents your browser from showing an “insecure” warning if you visit such a site. But every user who might ever connect to Icingas API should know about certificates and that these sites are not “insecure”, they merely use a certificate issued by a CA your browser doesn’t know yet.

Sorry if I’m sounding rude but this is an issue which brought us quite some hard times debugging, comes again and again and is brought up most of the time just because someone doesn’t want to dig into the topic how certificates and CAs work.

3 Likes
3

This is not a new thread but still very relevant. What lacks that I can see is clear documentation on what icinga commands are needed to use my own CA , or for anyone. I have spent too much time searching and that is just wrong. Using their own CA is best practice these days for security for a while now and having another CA as a one-off in the corporate, government , or educational/public internal networks is not best practice. I can agree that troubleshooting can be difficult but no more so than the same amount incurred in all the virtualization we have these days.
Just the Icinga2 node using the API internally for comments/notifications on hosts or services need this basic item, without being distributed.
As such the users need that document. How to use your own CA for the API.
I am sure the rest of us can add our own CA and certs to the HTTP interface in apache/nginx as that is pretty par for the course these days.
At this point I am left to guess about your CLI for this and that is not good, and I am very good at CA’s/Certs and all the other layers.
Please make a guide for that.

icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)

$ icinga2 pki
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)

Usage:
icinga2 []

Supported commands:

  • pki new-ca (sets up a new CA)
  • pki new-cert (creates a new CSR)
  • pki request (requests a certificate)
  • pki save-cert (saves another Icinga 2 instance’s certificate)
  • pki sign-csr (signs a CSR)
  • pki ticket (generates a ticket)

Global options:
-h [ --help ] show this help message
-V [ --version ] show version information
–color use VT100 color codes even when stdout is not a
terminal
-D [ --define ] arg define a constant
-I [ --include ] arg add include search directory
-x [ --log-level ] arg specify the log level for the console log.
The valid value is either debug, notice,
information (default), warning, or critical
-X [ --script-debugger ] whether to enable the script debugger

4

I agree how certificates and CA’s work is not trivial nor is troubleshooting for the non-CA oriented tech support person (which few are). But after the recent past SolarWinds fiasco, companies are more and more locking up keys and certs, correctly. On top of virtualization, its an added layer of complexity sure. Its become table stakes now though. Just look at how the browsers and OS’s are getting locked up tighter, for the better. The government and all suppliers/contractors are now required to have a great deal more IT security as well, non-optional. All medical and finance companies are getting pushed up too. This is the world we live and work in today in 2021.

5

I can just tell you what I see in lots of customers. Be it enterprise size customers, government agencies even high security environments. Not a single one I know uses certificates other than from the internal certification authority of Icinga.

In fact, that “internal” certificates are at least as secure as are those issued by some company CA. The only benefit you get from a company CA is that users that connect to the Icinga API don’t have to accept the Icinga CA root certificate as valid. But since no user should ever connect to the API this benefit is more or less non existent.

So, there’s no technical or security-related reason to switch from the CA Icinga builds for you. The only reasons I get are, pardon, from stubborn or ignorant leads that lack technical knowledge.

Please don’t take this as rude. I just want to make totally clear that there’s no security benefit in exchanging the certificates with those from another certification authority. And it’s extremly uncommon for users to replace them. Because the only thing you get is more work and more potential for problems.

If you see a potential increase in security by changing how the certificates are created, please let us know. Of course, there’s always the possibility of bugs or place for improvement in how CAs and certificates are created. But this should not be mitigated by replacing all certificates with company CA certificates. It’s way better to file a bug report or request for enhancement so all certificates are fixed in all setups.

6

Just my 2 cents… it depends for the use case…

If it is purely about backend communication I am completely fine with internal CA certs.
If it is user or interface with other tools related I think „simply add the internal icinga CA to your trusted ones“ is not increasing security. It is simply the opposite…making users ignoring encryption errors or warning and simply adding some cert. and with users I also mean admins…or how many have really (so really and always) checked fingerprints before quickly adding some cert to trusted?

Sure, unfortunately internal CAs are often insecure as well, but just because both is true the other approach isn’t simply false… :wink:

7

Hm, I can fully agree. An internal CA doesn’t improve security but it doesn’t reduce security as well. It’s just as secure as a CA signed by a commonly accepted master CA.

And that’s my point: You get no benefit from using certificates signed by a company CA. The only thing you get are potential problems because the certificates might be built with different options.

I’m purely talking about backend communication and that’s what the Icinga internal CA is for. You don’t use it for Webinterfaces, only for the API.

If you want to use internal certificates for Icinga Web 2 (what I wouldn’t suggest) you definitely should make sure to have the CA certificate put into users trust stores automatically. I’m totally on your side that telling users to accept the CA certificate is very bad practice.

Side note: I’ve seen people checking every single fingerprint before accepting a connection. At customer sites I’m not allowed to talk about. :slight_smile: But it really does happen.

8

Whilst I get your argument in general for internal set ups I tend to disagree on the greater scheme. In a multi cloud environment, which is more and more common, you might have to deal with icinga outside your infrastructe, as I currently have to do in a customer environment. Decently configured firewalls and proxys will deny those self signed certificates, importing externally managed internal CAs is a trust problem on its on, if the setup even allows you to do it, so having a service provider providing monitoring via icinga externally creates a load of troubleshooting issues of the very kind you mentioned to advise against using your own certificates. I know very few services that would not allow me in a proper way to integrate own certificates, so the issue of troubleshooting could also be seen as an unwiullingness to provide a proper infrastructure to handle those in a decent way (and thus avoid the problems). To quote your words ‘that might sound rude’, but seen from a security perspective and what is possible with other products - I can hardly see it differently.

1 Like
9

Ok, so our experiences differ in this case. :slight_smile: I guess, the best solution would be if both ways would be easily possible. I still prefer to have your own CA for Icinga and strictly use these certificates. Using others should be possible, too - as long as the people creating the certificates what they are doing.

Let’s agree on that we should have both options, ok?

10

I’m not sure why one would want a program such as Icinga issuing certificates, so I’m doing nothing in that area. But I do need secure and trusted communication with the API using a proper certificate. This doesn’t seem like an unusual request.

For the record, all that’s needed is to replace the certificates found in /var/lib/icinga2/certs/ with ones issued by your company CA.

11
12

As an update on this subject, the files in /var/lib/icinga2/ca/ need to be removed or else Icinga will “helpfully” overwrite your cert with one from its own CA. Not sure when this behaviour changed, but it was recently.

It would be nice if there were configuration options to just specify what certificate you wanted to use, and then have it used without complications. (Like literally every other program that provides a TLS interface.)

13

If you (don’t) want to use cert X, you simply (don’t) put it at Icinga 2 standard path. Meanwhile we have documented this topic: