Adding host via Saltstack bash script fails

Icinga Director
1

Hi,

I use the Icingaweb2 Director icinga2-agent-kickstart.bash script.
I made a small adjustment:

ICINGA2_CA_TICKET=$(icinga2 pki ticket --cn {{ grains.fqdn }} --salt "{{ salt['pillar.get']('ticketsalt') }}")

This works perfectly fine when I execute it from my LXC container:

# bash icinga2-agent-kickstart.bash
INFO: This should be a Debian system
check: icinga2 installed - OK: 2.12.3-1
INFO: Using new SSL directory: /var/lib/icinga2/certs
information/base: Writing private key to '/var/lib/icinga2/certs/librenms.example.com.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/certs/librenms.example.com.crt'.
information/base: Writing certificate signing request to '/var/lib/icinga2/certs/librenms.example.com.csr'.
information/cli: Retrieving TLS certificate for 'icinga.example.com:5665'.

 Version:             3
 Subject:             CN = icinga.example.com
 Issuer:              CN = Icinga CA
 Valid From:          Jan  3 21:43:55 2021 GMT
 Valid Until:         Dec 31 21:43:55 2035 GMT
 Serial:              d4:28:33:35:09:73:f3:7f:bc:0e:3c:c2:28:65:ff:b7:70:2f:55:c8

 Signature Algorithm: sha256WithRSAEncryption
 Subject Alt Names:   icinga.example.com
 Fingerprint:         CA A0 92 B7 4D DE 74 76 AE 21 CD F7 D1 CC 9C AB 11 71 AC AD 5C C3 49 E3 0A 1C F2 5B 22 91 7C 8B

***
*** You have to ensure that this certificate actually matches the parent
*** instance's certificate in order to avoid man-in-the-middle attacks.
***

information/pki: Writing certificate to file '/var/lib/icinga2/certs/trusted-master.crt'.
64f4ef0eeb1cba5061ce4a9a94f9c537ee97062a
information/cli: Writing CA certificate to file '/var/lib/icinga2/certs/ca.crt'.
information/cli: Writing signed certificate to file '/var/lib/icinga2/certs/librenms.example.com.crt'.
Writing config to /etc/icinga2/icinga2.conf
Writing config to /etc/icinga2/zones.conf
Writing config to /etc/icinga2/features-available/api.conf
warning/cli: Feature 'api' already enabled.
[2021-01-17 17:29:08 +0100] information/cli: Icinga application loader (version: r2.12.3-1)
[2021-01-17 17:29:08 +0100] information/cli: Loading configuration file(s).
[2021-01-17 17:29:08 +0100] information/ConfigItem: Committing config item(s).
[2021-01-17 17:29:08 +0100] information/ApiListener: My API identity: librenms.example.com
[2021-01-17 17:29:08 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2021-01-17 17:29:08 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-01-17 17:29:08 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-01-17 17:29:08 +0100] information/ConfigItem: Instantiated 3 Zones.
[2021-01-17 17:29:08 +0100] information/ConfigItem: Instantiated 2 Endpoints.
[2021-01-17 17:29:08 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2021-01-17 17:29:08 +0100] information/ConfigItem: Instantiated 235 CheckCommands.
[2021-01-17 17:29:08 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-01-17 17:29:08 +0100] information/cli: Finished validating the configuration file(s).
Please restart icinga2:
  systemctl restart icinga2

Notice the echo of the variable ICINGA2_CA_TICKET: 64f4ef0eeb1cba5061ce4a9a94f9c537ee97062a

When I execute from Saltstack:

----------
          ID: gen_certificate
    Function: cmd.run
        Name: bash /root/icinga2-agent-kickstart.bash
      Result: False
     Comment: Command "bash /root/icinga2-agent-kickstart.bash" run
     Started: 16:28:35.880497
    Duration: 1547.181 ms
     Changes:
              ----------
              pid:
                  130306
              retcode:
                  1
              stderr:
                  INFO: This should be a Debian system
                  INFO: Using new SSL directory: /var/lib/icinga2/certs
                  ERROR: Could not retrieve final certificate from host icinga.example.com
              stdout:
                  check: icinga2 installed - OK: 2.12.3-1
                  [2021-01-17 17:28:36 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  [2021-01-17 17:28:36 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  information/base: Writing private key to '/var/lib/icinga2/certs/librenms.example.com.key'.
                  information/base: Writing X509 certificate to '/var/lib/icinga2/certs/librenms.example.com.crt'.
                  information/base: Writing certificate signing request to '/var/lib/icinga2/certs/librenms.example.com.csr'.
                  [2021-01-17 17:28:37 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  [2021-01-17 17:28:37 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  information/cli: Retrieving TLS certificate for 'icinga.example.com:5665'.

                   Version:             3
                   Subject:             CN = icinga.example.com
                   Issuer:              CN = Icinga CA
                   Valid From:          Jan  3 21:43:55 2021 GMT
                   Valid Until:         Dec 31 21:43:55 2035 GMT
                   Serial:              d4:28:33:35:09:73:f3:7f:bc:0e:3c:c2:28:65:ff:b7:70:2f:55:c8

                   Signature Algorithm: sha256WithRSAEncryption
                   Subject Alt Names:   icinga.example.com
                   Fingerprint:         CA A0 92 B7 4D DE 74 76 AE 21 CD F7 D1 CC 9C AB 11 71 AC AD 5C C3 49 E3 0A 1C F2 5B 22 91 7C 8B

                  ***
                  *** You have to ensure that this certificate actually matches the parent
                  *** instance's certificate in order to avoid man-in-the-middle attacks.
                  ***

                  information/pki: Writing certificate to file '/var/lib/icinga2/certs/trusted-master.crt'.
                  [2021-01-17 17:28:35 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  [2021-01-17 17:28:35 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  64f4ef0eeb1cba5061ce4a9a94f9c537ee97062a
                  [2021-01-17 17:28:37 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  [2021-01-17 17:28:37 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  information/cli: Writing CA certificate to file '/var/lib/icinga2/certs/ca.crt'.
                  critical/cli: !!! Invalid ticket for CN 'librenms.example.com'.

Again please notice the echo of the ticket: 64f4ef0eeb1cba5061ce4a9a94f9c537ee97062a.
But it gives the error critical/cli: !!! Invalid ticket for CN 'librenms.klapwijk.it'.
What am I doing wrong?
It is the EXACT same script, but one is executed from Saltstack, the other directly from Bash.

What am I missing, or is this a bug?

Thank you!

2

Hi,

if you are using the SaltStack: Why you run the commands not directly with Salt instead of running an external script?
In the docs you’ll find an extra chapter for automatinon tools: https://icinga.com/docs/icinga-2/latest/doc/06-distributed-monitoring/#automation
There are also some code examples you could run with “cmd.run” in SaltStack. This works fine for us.

3

That’s what I had, but for whatever reason it doesn’t work (for me):

----------
          ID: gen_certificate
    Function: cmd.run
        Name: icinga2 pki new-cert --cn librenms.example.com --key /etc/icinga2/pki/librenms.example.com.key --cert /etc/icinga2/pki/librenms.example.com.crt &&
icinga2 pki save-cert \
--trustedcert /etc/icinga2/pki/trusted-parent.crt \
--host icinga.example.com &&
ticket=$(icinga2 pki ticket --cn librenms.example.com --salt "a+uXX3LMrqMUuutfjo9IuSKyrpqZYEoT9IV1USyr") &&
icinga2 node setup --ticket ${ticket} \
--cn librenms.example.com \
--endpoint icinga.example.com \
--zone librenms.example.com \
--parent_zone master \
--parent_host icinga.example.com \
--trustedcert /etc/icinga2/pki/trusted-parent.crt \
--accept-commands --accept-config \
--disable-conf

      Result: False
     Comment: Command "icinga2 pki new-cert --cn librenms.example.com --key /etc/icinga2/pki/librenms.example.com.key --cert /etc/icinga2/pki/librenms.example.com.crt &&
              icinga2 pki save-cert \
              --trustedcert /etc/icinga2/pki/trusted-parent.crt \
              --host icinga.example.com &&
              ticket=$(icinga2 pki ticket --cn librenms.example.com --salt "a+uXX3LMrqMUuutfjo9IuSKyrpqZYEoT9IV1USyr") &&
              icinga2 node setup --ticket ${ticket} \
              --cn librenms.example.com \
              --endpoint icinga.example.com \
              --zone librenms.example.com \
              --parent_zone master \
              --parent_host icinga.example.com \
              --trustedcert /etc/icinga2/pki/trusted-parent.crt \
              --accept-commands --accept-config \
              --disable-conf
              " run
     Started: 19:18:01.378184
    Duration: 695.586 ms
     Changes:
              ----------
              pid:
                  251252
              retcode:
                  1
              stderr:
              stdout:
                  [2021-01-18 20:18:01 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  [2021-01-18 20:18:01 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  information/base: Writing private key to '/etc/icinga2/pki/librenms.example.com.key'.
                  information/base: Writing X509 certificate to '/etc/icinga2/pki/librenms.example.com.crt'.
                  [2021-01-18 20:18:01 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  [2021-01-18 20:18:01 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  information/cli: Retrieving TLS certificate for 'icinga.example.com:5665'.

                   Version:             3
                   Subject:             CN = icinga.example.com
                   Issuer:              CN = Icinga CA
                   Valid From:          Jan  3 21:43:55 2021 GMT
                   Valid Until:         Dec 31 21:43:55 2035 GMT
                   Serial:              d4:28:33:35:09:73:f3:7f:bc:0e:3c:c2:28:65:ff:b7:70:2f:55:c8

                   Signature Algorithm: sha256WithRSAEncryption
                   Subject Alt Names:   icinga.example.com
                   Fingerprint:         CA A0 92 B7 4D DE 74 76 AE 21 CD F7 D1 CC 9C AB 11 71 AC AD 5C C3 49 E3 0A 1C F2 5B 22 91 7C 8B

                  ***
                  *** You have to ensure that this certificate actually matches the parent
                  *** instance's certificate in order to avoid man-in-the-middle attacks.
                  ***

                  information/pki: Writing certificate to file '/etc/icinga2/pki/trusted-parent.crt'.
                  [2021-01-18 20:18:02 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  [2021-01-18 20:18:02 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
                  critical/cli: Too many arguments. At most 0 arguments may be specified.

I can execute this 100 times by hand, but with Saltstack it just gives Too many arguments. At most 0 may be specified.. What the heck is up with that?

EDIT: Ran the same thing on a physical host, and works fine, seems related to LXC somehow…
EDIT2: Running the same commands in a privileged container works. So I guess it’s a bug related to Icinga, and caused by the error:
warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"

4

Hi,
Which strategy did you choose for the certificates? auto-sign? you want to sign the certifcate. requests?

This works fine for us:

icinga_csr_save_{{ host_name }}:
   cmd.run:
     - name: icinga2 pki save-cert --trustedcert /var/lib/icinga2/certs/trusted-master.crt --host {{ config_master }}
     - cwd: /var/lib/icinga2/certs/
     - creates:
        - /var/lib/icinga2/certs/trusted-master.crt

icinga_csr_create_{{ host_name }}:
   cmd.run:
     - name: icinga2 pki new-cert --cn {{ host_name }} --cert /var/lib/icinga2/certs/{{ host_name }}.crt --key /var/lib/icinga2/certs/{{ host_name }}.key
     - cwd: /var/lib/icinga2/certs/
     - creates:
       - /var/lib/icinga2/certs/{{ host_name }}.key
       - /var/lib/icinga2/certs/{{ host_name }}.crt

icinga_cert_request_{{ host_name }}:
   cmd.run:
     - name: icinga2 pki request --host {{ config_master }} --cert /var/lib/icinga2/certs/{{ host_name }}.crt --key /var/lib/icinga2/certs/{{ host_name }}.key --trustedcert /var/lib/icinga2/certs/trusted-master.crt --ca /var/lib/icinga2/certs/ca.crt 
     - cwd: /var/lib/icinga2/certs/
     - creates:
        - /var/lib/icinga2/certs/ca.crt

This is only a part from our salt. Between are some other steps like chance the permissions etc.

5

Hi Stevie,

It works fine on a privileged LXC container for me, however it fails on an unprivileged container.
Are you using this on LXC as well?
By hand it works as well, so I’m guessing this is a problem related to Saltstack, however the issue only happens in icinga2 cli tool, so I’m not sure where to look now.

I auto-sign the certificates, that is correct.

6

Hi sorry we don’t use LXC containers

7

So the issue is caused because of unprivileged containers.
This gives a warning message. How it’s caused or the exact details I’m not sure, I am not a LXC expert unfortunately.
However after further testing I discovered that when I use ticket=$(icinga2 pki ticket --cn librenms.example.com --salt "a+uXX3LMrqMUuutfjo9IuSKyrpqZYEoT9IV1USyr") it assigns the following value to ticket:

[2021-01-19 11:36:09 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted" [2021-01-19 11:36:09 +0100] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted" 64f4ef0eeb1cba5061ce4a9a94f9c537ee97062a

So it also includes the warning message in the variable!
My fix was a bit ugly, but works:

ticket=$(ticket=$(icinga2 pki ticket --cn librenms.example.com --salt "a+uXX3LMrqMUuutfjo9IuSKyrpqZYEoT9IV1USyr") | grep -E "[a-f0-9]{40}")

I just filter the UUID, problem solved!
If someone from Icinga still wants to take a look at it, because I don’t think it should happen in an unprivileged container.

8

Hi,

if you think it’s a bug and a developer should look into it, maybe you create a issue on Git? But good to read you found a workaround to fix it