Making and integrating graphs into icinga with rrdtool

rrdtool module setup Almalinux 9

Download module from icingaweb2-module-rrdtool v1.5.1.zip

Unzip and move to

/usr/share/icingaweb2/modules/rrdtool

Set same permissions as other modules and fix ownership.
Also

restorecon -RFv /usr/share/icingaweb2/modules/rrdtool

Install dependencies for almalinux 8 or 9.

dnf install -y rrdtool perl-rrdtool perl-Time-HiRes php-gd php-xml php-pecl-rrd

perl-Time-HiRes php-gd php-xml are most likely already installed if your icinga is working.

create datadir and fix ownership. Also restorecon

/var/lib/icingaweb2/rrdtool/
chown root:icingaweb2 /var/lib/icingaweb2/rrdtool/
restorecon -RFv /var/lib/icingaweb2/rrdtool/

enable perfdata incase not enabled. Restart is required if wasn’t previously enabled.

icinga2 feature enable perfdata
systemctl restart icinga2.service

Enable module via UI. And set the following settings.

Path to RRD and XML files:/var/lib/icingaweb2/rrdtool/
Path is readable: Should be a green check mark here.
Path to templates:/usr/share/icingaweb2/modules/rrdtool/templates/

Path to PerfdataWriter output:/var/spool/icinga2/perfdata
Checks with multiple RRDs: Leave empty
Enable perfdata processing: Set this on
Enable logging: Set this on if you want logs
Verbose logging: Set this on if you want verbose logs

PHP RRD Extension: Should be a green check mark here.
Path to rrdtool binary: rrdtool
Override max thumbnails: Leave this empty.

SAVE CHANGES!

Now running :

icingacli rrdtool process

Should result in some rrd and xml files under your datadir

ls -larth /var/lib/icingaweb2/rrdtool/your_host/

Check log for possible errors:

/var/lib/icingaweb2/rrdtool/rrdtool.log

Services that have been in unknown state will not produce data. Logs will tell you if this is the case.

Set a cron job so that rrd are regurlary updated.

#cat /etc/cron.d/process_rrd
#process data
*/5 * * * * root /usr/bin/icingacli rrdtool process

Creating graphs for ssh response time last hour:

rrdtool graph ssh_graph_1.png --start now-3600s --end now DEF:1=/var/lib/icingaweb2/rrdtool/your_host/ssh.rrd:1:AVERAGE LINE1:1#FF0000:“SSH response time”

This is where i put my SELinux back on to see if it would break the installation. It did this is because php-fpm is not able to read the data directory. Easy fix is to make a module which allows it to read it. Or you can also fix the context to httpd_t for the directory (recommended).
Or to be a noob and not use SELinux alltogether.(But what would Dan Walsh think of you then? https://stopdisablingselinux.com/)

Now this is as far as I managed to do by myself in a short time period.

If i do this part which is mentioned in the “installation” guide.

When you use checkcommands like nrpe, that actually call other commands, you can add a customvar like rrdtool to your services and append it to the checkcommand in your perfdata to allow differentiation:

object PerfdataWriter “perfdata” {
rotation_interval = 10s
service_format_template = “DATATYPE::SERVICEPERFDATA\tTIMET::$service.last_check$\tHOSTNAME::$host.name$\tSERVICEDESC::$service.name$\tSERVICEPERFDATA::$service.perfdata$\tSERVICECHECKCOMMAND::$service.check_command$$rrdtool$\tHOSTSTATE::$host.state$\tHOSTSTATETYPE::$host.state_type$\tSERVICESTATE::$service.state$\tSERVICESTATETYPE::$service.state_type$”
}

Results as missing macro in logs:

[2023-10-11 15:44:12 +0300] warning/MacroProcessor: Macro ‘rrdtool’ is not defined.
Context:
(0) Resolving macros for string ‘DATATYPE::SERVICEPERFDATA TIMET::$service.last_check$ HOSTNAME::$host.name$ SERVICEDESC::$service.name$ SERVICEPERFDATA::$service.perfdata$ SERVICECHECKCOMMAND::$service.check_command$$rrdtool$ HOSTSTATE::$host.state$ HOSTSTATETYPE::$host.state_type$ SERVICESTATE::$service.state$ SERVICESTATETYPE::$service.state_type$’
(1) Writing performance data for object ‘icinga!ssh’

If I add a variable

vars.rrdtool = “rrdtool”

to the template Service “generic-service” it goes away since now we have it defined.

But I really don’t know how to visualize and integrate the rrd files into icinga as the projects images suggest that they’re able to do. I’m obviously lacking some configuration on icinga. What should be done in order to have the same visualization as in the module front page suggests? rrdtool :: Icinga Exchange

TBH the documentation is lacking for this project. I think each module project should have clear information on how to actually install it and achieve the status which the images on the project show.
Not everyone has the time/knowledge to do this. Projects like this should be more accessible.

Much help would be appreciated.

Also shoutout to https://freedom-for-icinga.com/ for offering a RHEL 9 repo for free.

Oh boy oh boy.

I managed to get this working. More or less what I did on a fresh install.

Install php rrd extension

dnf install php-pecl-rrd
Move the module data into corret folder
mv icingaweb2-module-rrdtool-master /usr/share/icingaweb2/modules/rrdtool
Set selinux off for now
setenforce 0

Enable perfdata module and restart

icinga2 feature enable perfdata
systemctl restart icinga2.service

Find out templates etc. location

ls -larth /var/spool/icinga2/perfdata/
ls -larth /usr/share/icingaweb2/modules/rrdtool/templates/pnp4nagios/

Make a directory for rrd data

mkdir /var/lib/php/rrdtool
chown icinga:apache /var/lib/php/rrdtool

Configure your module

Path to RRD and XML files:/var/lib/php/rrdtool
Path is readable: Verify that you have a green check mark here.
Path to templates:/usr/share/icingaweb2/modules/rrdtool/templates/pnp4nagios/
Path to PerfdataWriter output:/var/spool/icinga2/perfdata/
Enable perfdata processing: check this
Enable logging: check this
Verbose logging: check this if you want verbose logging
PHP RRD Extension: Verify that you have a green check mark here.
Path to rrdtool binary:/usr/share/bash-completion/completions/rrdtool

Wait a few minutes in order to have some perfdata to process and then process it.

icingacli rrdtool process

Check your data directory if it has successfully processed performance data.

ls -larth /var/lib/php/rrdtool

Create a cron job to process perfdata every 5 mins or so.

vim /etc/cron.d/create_graphs_rrd

Check logs for errors etc.

less /var/lib/php/rrdtool/rrdtool.log

After finding out it works is time to do some SELinux magic and do some restorecons on new directories etc. and then fixing the rest of the problem. When no sealerts are present you should be able to turn selinux back on without breaking your graphs.

Also remember if you have a lot of rrd data or performance data it might be smart to move data out of var to a separate disk for example.