I’m trying to automate the deployment of our debian hosts in Icinga.
My goal: It should be possible for a host to register itself with the icinga-master and (ideally) also activate defined services. Agent-based services such as CPU and storage should be monitored.
My attempts so far:
“icinga2 node wizard”; walk through the setup and approve the certificate on the master.
However, this requires a icinga-master administrator to assist manually. Also the wizard on the host is a manually step and I could not find a solution to do this in a script.
Register the host via Directors REST API:
Created an agent based host template to get an API key.
curl -k --json ‘{“address”: “”,“display_name”: “\<HOSTNAME\>”}’ ‘\<URL MASTER\>/director/self-service/register-host?name=\<HOSTNAME\>&key=\<TEMPLATE API KEY\>’
The host is created and can be monitored. However, when I add agent-based services here, I get the message “Remote Icinga instance ‘\<HOSTNAME\>’ is not connected to ‘\<MASTER\>’”.
I also have no request to sign a host certificate on the Master like the wizard does.
Is there a way to add a host to the icinga-master as an agent-based host via script directly from the host?
but there is also a selfservice api endpoint for the ticket: /icingaweb2/director/self-service/register-host returns the HOSTKEY if successfully registered
you can use this key to ask icinga director selfservice api for the ticket
I use the self service API of the director in first run scripts for new Windows and Linux hosts.
The one for Windows is directly offered and the one for Debian I build my self but it wasn’t to hard.
First activate the feature and then enable the token on a host template.
Put the token in your script.
It’s maybe a bit overkill as we have 2 nodes in the master zone, an internal package mirror and internal only certificates. The apt mark hold stuff could also be more problem then it’s worth in your setup.
Maybe rip out, what you don’t need and replacing stuff on lines with XXX and example.com with your values and you should be good.
#!/usr/bin/env bash
#set -x
echo "Setup Icinga2 agent and the Linuxfabrik's monitoring-plugins"
# config vars
DIRECTOR_KEY='XXXXXXXXXXXXXXXXXXXX' # tpl-debian-base-img
MIRROR='https://deb.example.com/'
APT_KEYS_DIR='/etc/apt/trusted.gpg.d'
ICINGA_MASTER1='icinga1.example.com'
ICINGA_MASTER2='icinga2.example.com'
ICINGA_URL='https://icinga.example.com'
APT_SOURCES_ICINGA='/etc/apt/sources.list.d/packages_icinga_com_debian.list'
APT_SOURCES_LINUXFABRIK='/etc/apt/sources.list.d/linuxfabrik-monitoring-plugins.list'
MONITORING_PLUGINS_VERSION='2.2.1-1'
PACKAGES=(
curl
icinga2
"linuxfabrik-monitoring-plugins=${MONITORING_PLUGINS_VERSION}"
)
HOST_KEY_FILE="/var/lib/icinga2/certs/host_key"
# Define color codes
RED='\033[0;31m'
NC='\033[0m' # No Color
# taken from /var/lib/icinga2/certs/ca.crt on icinga1.example.com
ICINGA_CA=$(cat <<'EOF'
-----BEGIN CERTIFICATE-----
XXXXXXX
-----END CERTIFICATE-----
EOF
)
EXAMPLE_CA_2021A=$(cat <<'EOF'
-----BEGIN CERTIFICATE-----
XXXXXXXXXXXXX
-----END CERTIFICATE-----
EOF
)
EXAMPLE_ROOT_CA_2021=$(cat <<'EOF'
-----BEGIN CERTIFICATE-----
XXXXXXXXXXXXXX
-----END CERTIFICATE-----
EOF
)
# https://packages.icinga.com/icinga.key
ICINGA_PGP_KEY=$(cat <<'EOF'
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGZMb30BEAC6c5P5lo5cLN2wX9+jA7TEEJ/NiiOM9VxBwB/c2PFd6AjdGBbe
28VcXWmFdETg1N3Woq08yNVXdxS1tMslyl9apmmyCiSC2OPMmTOveLzZ196IljYR
DeZMF8C+rdzNKXZzn7+nEp9xRy34QUZRfx6pEnugMd0VK0d/ZKgMbcq2IvcRQwap
60+9t8ppesXhgaRBsAzvrj1twngqXP90JwzKGaR+iaGzrvvJn6cgXkw3MyXhskKY
4J0c7TV6DmTOIfL6RmBp8+SSco8xXD/O/YIpG8LWe+sbMqSaq7jFvKCINWgK4RAt
7mBRHvx81Y8IwV6B2wch/lSyYxKXTbE7uMefy3vyP9A9IFhMbFpc0EJA/4tHYEL4
qPZyR44mizsxa+1h6AXO258ERtzL+FoksXnWTcQqBKjd6SHhLwN4BLsjrlWsJ6lD
VaSKsekEwMFTLvZiLxYXBLPU04dvGNgX7nbkFMEK6RxHqfMu+m6+0jPXzQ+ejuae
xoBBT61O7v5PPTqbZFBKnVzQPf7fBIHW5/AGAc+qAI459viwcCSlJ21RCzirFYc0
/KDuSoo61yyNcq4G271lbT5SNeMZNlDxKkiHjbCpIU6iEF7uK828F1ZGKOMRztok
bzE7j1IDIfDQ3P/zfq73Rr2S9FfHlXvEmLIuj5G4PO7p0IwUlCD1a9oY+QARAQAB
tCxJY2luZ2EgR21iSCAoQnVpbGQgc2VydmVyKSA8aW5mb0BpY2luZ2EuY29tPokC
TgQTAQoAOBYhBN069hmO0AC0wLc5VswRb1WqfyOCBQJmTG99AhsDBQsJCAcCBhUK
CQgLAgQWAgMBAh4BAheAAAoJEMwRb1WqfyOCGrIP/i/4fYEkdCi4nhQGMzSP0Eyh
UhJjsUP9mEqSQRqOAplvjYa1yBbrSPLfkRE0oAL/o+4eUKcAQFeDQtDXJ/D4xl3Q
J5MehRJYzklrSs5XkEscb73HoDBUfFSgCVM2zK+JkCX0CPJ4ZLWtZGJ+8pCLpnkH
nCPonbGc6sS+m2JsPRwxyxAhdXxWSAesXd8dUSW3MOQz9JlC4/idQcCFs03fdhuZ
4jGMry08OihWVudTDK8nkwRZLzNoOivAQ3mIeaTcRMmgPJfYN4k0o90lXJWAbG+2
j8p7Pyjv71OctI8KUbS4+f2H8i6r5Pc4M4hlUQh6QAN9o1oPJrXxurdp0EXgQXSy
rVH2MeguqprFJxGjdlTCSTYgQEmEXMixRAGzteEgCf/Qk9mPXoxFTNyNg4/Lkglb
Nj6dY6or6w+IsbdrcePqDAs+j9t5B97vU7Ldquloj85myQjkWPP8kjlsOlsXBkQ/
C+mD+5iW2AiWh+yCasf6mOZwUfINZF+VDpmfIsZZbWpcMgp1f32fpRFZ3ietnsnR
+luNb19hUHKyyDDHMe/YM7H9P5vtX9BGz6O9kNpo1LAnigkSQSFBZlK3Po3Yk9eg
XPbDT5HsU3TMyS5ZnSDRRPPJwsyGPXz+0pCADae9H9hCc2C2LZIrrtwlOFPWuViA
ifY/dQmUP37n5XgMADRc
=O0zm
-----END PGP PUBLIC KEY BLOCK-----
EOF
)
# https://repo.linuxfabrik.ch/linuxfabrik.key
LINUXFABRIK_PGP_KEY=$(cat <<'EOF'
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGM9do4BEADaLrP/SnIFkk4BwC5VA8JYhLmA8IRRazZAi05hzPv+2mnCUkJ+
/ro6fopAAB7k+70ioqSNd8tHg0vbEwpmN+YHVHzyZuX399EWhT+LuibBdnRvqTog
9ujw5dXRHnppQ8qNUMj/+mq0wZN4EzQWu1fBnrnXL4TYp2MGZTlshJi3lFGtktjb
Zg/7ZmG3Kd8uPyGEmyIyTJUDluq7nqk9NRXZmHxRt4Cs5P7ZpUkJlB+dEpkP3t6v
M//6xdsLy7d37M+t09VyboLEq7hw0GD5YhitnVoI92ZTQTlXj3iDvhGpdNXBa6v7
4Gnv/9jGhmdZxjKJKfyS0DEa+Yat5nF/uKmWRdUBzVUognE2SDQ9P3C7Pcd8vZ/C
2UgxAk8rRhkdzypBYiSdN2KltZNbr2fcFf+Ngiepu3cRVztvn4gzj9xHh7L0LnIK
jmkBLO3QiOhGnMsDijwz/uOt56Ug0/JwEJG1cvdbYhCd9NcaUiSZhkDByggKbGGk
6/kAPGRG/DoObHhjfW7j5QvEygdoc811/0UR9ClPdQxAEqT7SBq0c7C9u6unE+vC
clyIePDXDQUWJsI+dASUUp592j3QlxbB56mTI3vjsSSmBMxiuPM4syW8KdVIbCvE
DQ9tSdqyQmE3zSxQanPahqCcFSvk21O2MNw0FoOFYynCL6gv7uLYW8GBMwARAQAB
tCxMaW51eGZhYnJpayAoUGFja2FnZXIpIDxpbmZvQGxpbnV4ZmFicmlrLmNoPokC
UQQTAQgAOxYhBBAXN3/dk102nM0MWfJ5iOw50Xp4BQJjPXaOAhsvBQsJCAcCAiIC
BhUKCQgLAgQWAgMBAh4HAheAAAoJEPJ5iOw50Xp4pYoP/1ZFzCMCAoopWzip3vb3
J8qSsJUptmYKggED0XXjfND14UCqGhBQlZcPF4Qbo59487PEB5IscRj7gWuJiMQ9
52/slYcnON4SMJJacmaYMcMTO8rGB0AeyaOeMYX+DG1iOI5pbLPxl5nfpLl50Ej6
JNMfNDS18m4C0XdjPGRvPQ7U9Jo2+LwD3Hbi4ROiO65NkAfgFoZaSK0/x58CObG+
wWJW4sbLLc7QW5KOtlYHfZHCXP2BBntvcFQXQNFPBLBRObacCYnnaVTQlNpPGpZ+
lmZl7PdOPM6Fym6rUxwW44D7nkClZvrs4EGyjyfsIIbt42jvhrZCUiyPozWRTe4J
vUbOaw/f9iHeLQleLHshbx5W4Tx5my4YWNbpG+yzVXymx0zemCQukvAentADGf+H
plfc9anejA0CjEypZQuDdmXT+3U9WbAebYdYw2LL8shCKDa/4rWGFv/AEzzUeGn7
eDr8ugnJSsTp0/RMKI1pw/bEDLpcRLjj8LLM7/hwPL0JfVQfh4ovYrLRftBdxaIl
si2eZ++UFXA3/Ia5WQKPDovwRnO6FrdAPTtH9gy/kwvqLfiYn6dcPIh1EtCezSzP
At8Vh7+jgna4skTL1BUUuJCR6E3dAYjcGCFRxCUhZSf9HUynm8OHGgNO4dtismug
g4kWv3M/8nYy+hgGkeGwVU7C
=r/Ck
-----END PGP PUBLIC KEY BLOCK-----
EOF
)
# system vars
. /etc/os-release
DISPLAY_NAME=$(hostname -s)
FQDN=$(hostname -f)
IP=$(getent hosts "$FQDN" | awk '{ print $1 }' | head -n1)
TICKET_FILE="/var/lib/icinga2/certs/ticket"
echo "1. setup repos unsing mirror"
# add repo GPG keys
install --owner=root \
--group=root \
--mode=755 \
--directory $APT_KEYS_DIR
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed to create apt keys directory.${NC}"
exit 1
fi
echo "$ICINGA_PGP_KEY" > ${APT_KEYS_DIR}/icinga.asc
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed to write Icinga PGP key file.${NC}"
exit 1
fi
echo "$LINUXFABRIK_PGP_KEY" > ${APT_KEYS_DIR}/linuxfabrik.asc
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed to write Linuxfabrik PGP key file.${NC}"
exit 1
fi
# add CA for TLS certs used by the internal mirror
echo "$EXAMPLE_CA_2021A" > /usr/local/share/ca-certificates/EXAMPLE_CA_2021A.crt
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed to write EXAMPLE_CA_2021A file.${NC}"
exit 1
fi
echo "$EXAMPLE_ROOT_CA_2021" > /usr/local/share/ca-certificates/EXAMPLE_ROOT_CA_2021.crt
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed to write EXAMPLE_ROOT_CA_2021 file.${NC}"
exit 1
fi
update-ca-certificates
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed to update CA certificates.${NC}"
exit 1
fi
# store the icinga ca for certificate verification
echo "$ICINGA_CA" > /tmp/icinga-ca.crt
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed to write /tmp/icinga-ca.crt file.${NC}"
exit 1
fi
# construct sources.list files
cat > $APT_SOURCES_ICINGA << EOF
deb [signed-by=${APT_KEYS_DIR}/icinga.asc arch=amd64] ${MIRROR}icinga icinga-${VERSION_CODENAME} main
deb-src [signed-by=${APT_KEYS_DIR}/icinga.asc arch=amd64] ${MIRROR}icinga icinga-${VERSION_CODENAME} main
EOF
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed write Icinga2 sources file.${NC}"
exit 1
fi
cat > $APT_SOURCES_LINUXFABRIK << EOF
deb [signed-by=${APT_KEYS_DIR}/linuxfabrik.asc arch=amd64] ${MIRROR}linuxfabrik/monitoring-plugins/debian/ ${VERSION_CODENAME}-release main
EOF
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed write monitoring-plugins sources file.${NC}"
exit 1
fi
echo "2. install Icinga2 and monitoring-plugins"
# refresh repos
echo "Updating package lists..."
if ! apt-get update; then
echo -e "${RED}Failed to update package lists. Exiting.${NC}"
exit 1
fi
# make packages installable
apt-mark unhold icinga2
apt-mark unhold linuxfabrik-monitoring-plugins
# install packages
echo "Installing packages..."
if ! apt-get install -y "${PACKAGES[@]}"; then
echo -e "${RED}Package installation failed. Exiting.${NC}"
exit 1
fi
# prevent updates to incompatible versions
apt-mark hold linuxfabrik-monitoring-plugins
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed to mark monitoring-plugins as on hold.${NC}"
exit 1
fi
echo "3. get key from director"
# use HOST Template API Key to create host and get self-service Host API Key
HOST_KEY=$(curl --silent --show-error --fail --request POST \
--url "$ICINGA_URL/icingaweb2/director/self-service/register-host?name=$FQDN&key=$DIRECTOR_KEY" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '{
"display_name": "'"$DISPLAY_NAME"'",
"address": "'"$IP"'"
}' 2>&1) # Capture errors as well
# Check if the request was successful
if [[ $? -ne 0 || -z "$HOST_KEY" ]]; then
echo -e "${RED}Error: Failed to register the host or received an empty response."
echo -e "Curl Output: $HOST_KEY${NC}"
exit 1
fi
echo "Host registered successfully. Received key: $HOST_KEY"
# remove surrounding doulbe quotes
HOST_KEY="${HOST_KEY%\"}"
HOST_KEY="${HOST_KEY#\"}"
# if error in HOST_KEY try to find key in /var/lib/icinga2/certs/ticket
if [[ $HOST_KEY == *"error"* ]]; then
if [ -f "${HOST_KEY_FILE}" ]; then
HOST_KEY=$(<"${HOST_KEY_FILE}")
else
echo -e "${RED}Error: could not create or load host key for self service ticket API to get Cert signed by Icinga PKI!"
echo -e "Remove host from director or check self service token of diector host-template.${NC}"
exit 1
fi
fi
# use Host API Key to get Icinga2 PKI ticket
ICINGA_TICKET=$(curl --request GET \
--url $ICINGA_URL'/icingaweb2/director/self-service/ticket?key='$HOST_KEY \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
)
# remove surrounding doulbe quotes
ICINGA_TICKET="${ICINGA_TICKET%\"}"
ICINGA_TICKET="${ICINGA_TICKET#\"}"
# if error in HOST_KEY try to find key in /var/lib/icinga2/certs/ticket
if [[ $ICINGA_TICKET == *"error"* ]]; then
if [ -f "${TICKET_FILE}" ]; then
ICINGA_TICKET=$(<"${TICKET_FILE}")
else
echo -e "${RED}Error: could not create or load ticket to get Cert signed by Icinga PKI!"
echo -e "Remove host from director or check self service token of diector host-template.${NC}"
exit 1
fi
fi
echo "4. Setup Icinga2 Agent"
#make sure the directory /var/lib/icinga2/certs exists and has the correct permissions
install --owner=nagios \
--group=nagios \
--mode=700 \
--directory /var/lib/icinga2/certs
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed to create certs directory.${NC}"
exit 1
fi
#save host key
echo "$HOST_KEY" > $HOST_KEY_FILE
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Failed write host key.${NC}"
exit 1
fi
#save PKI cert from Icinga config master while still
#fd3 will preserve output to stdout while capturing stdout for later checking of fingerprint
{ PKI_OUTPUT=$(icinga2 pki save-cert \
--host $ICINGA_MASTER1 \
--port 5665 \
--key local.key \
--cert local.crt \
--trustedcert /var/lib/icinga2/certs/master.crt | tee /dev/fd/3 ); } 3>&1
if [[ $? -ne 0 || -z "$PKI_OUTPUT" ]]; then
echo -e "${RED}Error: Failed to get certificate or received an empty response."
echo -e "Icinga2 pki save-cert Output: $PKI_OUTPUT${NC}"
exit 1
fi
if openssl verify -CAfile /tmp/icinga-ca.crt /var/lib/icinga2/certs/master.crt ; then
echo "Icinga PKI: certificate verification successful"
else
echo -e "${RED}Error: Icinga PKI certificate could not be verified! Man in the middle attack?${NC}"
exit 1
fi
# execute icinga2 node setup
icinga2 node setup --zone $ICINGA_MASTER1 \
--endpoint ${ICINGA_MASTER1},${ICINGA_MASTER1},5665 \
--endpoint ${ICINGA_MASTER2},${ICINGA_MASTER2},5665 \
--parent_host ${ICINGA_MASTER1},5665 \
--parent_zone master \
--cn $FQDN \
--accept-config \
--accept-commands \
--disable-confd \
--trustedcert /var/lib/icinga2/certs/master.crt \
--ticket $ICINGA_TICKET
# Check if the request was successful
if [[ $? -ne 0 ]]; then
echo -e "${RED}Error: Icinga2 node setup failed.${NC}"
exit 1
fi
# enable the icinga2 service
systemctl enable --now icinga2.service
# restart the icinga2 service to ensure new configuration is enabled
systemctl restart icinga2.service