Before I start, a story… the folks at our office do not allow our servers to have direct access to the internet. We have an internal yum server from which we must perform all our installs. I am the one doing monitoring so, on a separate instance that was connected to the internet, I ran all the downloads with yum downloader
and then copied all the rpms to the yum server. I wrote a script so I can do it faster next time. I did a few manually, but this gets most of what was needed for Icinga2 and Icingaweb2. Perhaps it could be provided as a utility for others who face this predicament.
#!/bin/bash
# downloads icinga2 and dependent packages
yum -y install epel-release yum-utils
yum -y install http://rpms.remirepo.net/enterprise/remi-release-7.rpm
yum-config-manager --disable remi-php56
yum-config-manager --enable remi-php73
pkgList=(
centos-release-scl
icinga2
nagios-plugins-all
icinga2-selinux
vim-icinga2
mysql-server
mysql
icinga2-ido-mysql
httpd
icingaweb2
icingacli
rh-php73-php-common
rh-php73-php-mysqlnd
)
for i in "${pkgList[@]}"
do
sudo yumdownloader --resolve $i
done
The one thing the above list provides is a list (perhaps incomplete) of what the heck I need to install the base components.
My new dilemma
But now with Icinga2 and Icingaweb2 installed, I am only monitoring the Icinga server.
I added 4 remote servers to the hosts.conf file so now I can see the extra 4 servers with 2 default services each; ping and ssh. Good start. Small simple accomplishment.
Next task - I must decide how I will run all the checks on remote servers.
I see the Configuration section and that all looks approachable. All the config files in the documentation look more or less like what I see in the /etc/icinga2/
directory.
Next I see a Chapter on Services and Plugins but this seems too detailed and I will read it later. I already saw and understand services.conf from the previous chapter.
Next up, Distributed Monitoring. Oh my. It’s a long chapter; and to my quick look it is 120 web screens long. I need a web server (Master?) with Icinga2/Icingaweb2 and I have 4 “satellites”. I am starting to wonder how much work this will be… and will I be done in time to meet my friends at the beer garden in a few hours.
So now I am desperate to know the fastest, best-practice, recommended approach to just get this done.
I just looked up in the Distributed Monitoring chapter and there are 26 “you want” references. IOW, if you want to, whether you want to, in case you want to, etc. There are too many choices and I hear my friends laughing and having fun.
I skip the Distributed Monitoring for now in the hopes that there is a more simple approach.
Ah. Here is Agent Based Monitoring. Now it feels like we get to the heart of monitoring… but wait; it says, Prior to installing and configuration an agent service, evaluate possible options based on these requirements (I count 8 things I must evaluate). Another decision about things upon which I will have to do google research.
The first agent described is Icinga as agent (I think it is just Icinga2) and here are the kind of words I’ve been looking for… “For the most common setups on Linux/Unix and Windows, we recommend to setup the Icinga agent in a distributed environment.” But now I’m back to Distributed Monitoring.
Next I see a list of key benefits of using the Icinga Agent - do I need convincing? The document says “For the most common setups on Linux” and we recommend - let’s get to it.
Then to learn how to Install the Icinga Agent, a link takes me back to the Distributed Monitoring chapter where it is titled: Agent/Satellite Setup and in the first sentence of that section it informs me that I must first run the master setup.
So it seems the most common and recommended approach is Distributed Monitoring with Icinga as agent. OK. Fine. Why didn’t the docs just say that in the beginning.
I can tell by now, I will not be meeting my friends for beer, or finishing this within the next few days.
If anyone has a TL;DR version or an outline for the least number of steps to set up a master/satellite configuration with ssh keys, I would appreciate any references.
I love Icinga. I’m trying to keep it light and a bit funny but also making a point as I did in another post that it feels like the documentation is getting in the way of just monitoring. It is a vitally critical function within any operation, but for me, it’s one of many pieces I manage. I don’t have the bandwidth to become an expert in the different ways to install and integrate this monitoring tool. I need the developers of Icinga to be the experts. I think my expertise comes near the end of all the configuring when it is time to make sense of the data flowing through graphite/grafana and how to make sure that only truly critical issues get sent as alerts.
One other point, if you please. I see references of using vim as an editor over and over when we just need to see the results of the config files. It seems to me, if we are actually logging into hosts to edit configs instead of deploying locally edited templated configs with tools like ansible, we’re dealing with too much tedium. In general if we edited it once, we’ll likely edit again and again, so we might as well deploy in a reliable repeatable manner. And I recommend ansible because I think it hits the right level between simplicity and power.
I’ll be impressed if you get here without thinking to yourself, TL;DR. But I appreciate if you can understand what I’m trying to say.
Now it’s late, but I’ll have a beer anyway. Cheers.