Migrating checks from NRPE to Icinga Agent using Icinga Director

Service Monitoring
1

We are attempting to migrate from Icinga 1 to Icinga 2 and a part of that process is moving from using nrpe to Icinga agent. We have approximately 300 checks in our nrpe.cfg file of which approximately 200 call to home-built scripts. I believe this is the correct process for transitioning from nrpe to Icinga Agent using Icinga Director, however I wanted to post here for feedback and suggestions with the process. Our short term goal is to get all our checks running via Icinga Agent, and look to replace scripts with ITL checks on a scheduled basis. Time is not on our side in this migration, so getting checks functioning is currently the main goal, and improving checks is unfortunately going to be further down the line.

Our Environment

We currently are running a 3 tiered environment. We have 2 high-availability masters sitting in a master zone that send check commands to satellites. The IDO database is on an separate host. One of the masters has Icinga Web and Icinga Director installed with the Icinga Director database being local on the master (for the time being at least). We have zones for each of our n data centers, each zone having 2 satellites to load-balance checks. Each data center has between 1000-2000 hosts and approximately 25 services per host. Each host has icinga2 installed and has a signed CA from the master. Independently of Icinga, we have an RPM that ensures all of our scripts are in either /usr/lib64/nagios/plugins or /usr/lib64/nagios/plugins/trip/ accordingly for all hosts.

Migrating from NRPE to Icinga Agent.

I would like to run through 2 examples. One being a simple check with no arguments, and the other being a more complicated case.

Case 1

Our line in npre.cfg for this check is as follows:
command[check_ta_bios]=sudo /usr/lib64/nagios/plugins/trip/check_ta_bios

  1. Create a custom CheckCommand (assuming PluginDir is set correctly) :
    1. Navigate to Icinga Director => Commands => Add
    2. Command type is Icinga Check Command
    3. Command name is “ta_bios”
    4. Command is “check_ta_bios”
  2. Create an Agent Check via Command Endpoint:
    1. Create a new apply rule called “TA Bios”
    2. check_command = “ta_bios”
    3. command_endpoint = “host.name”
    4. Give an apply rule for the check.
  3. Check is applied to hosts that meet apply rules.
  4. Done (?)

Case 2

This case is more complicated and where I have more questions. This covers 6 checks that execute the same script but have different variables.

command[check_openmanage_storage]=sudo /usr/lib64/nagios/plugins/trip/check_openmanage storage
command[check_openmanage_memory]=sudo /usr/lib64/nagios/plugins/trip/check_openmanage memory
command[check_openmanage_misc]=sudo /usr/lib64/nagios/plugins/trip/check_openmanage misc
command[check_openmanage_storage_non_redundant_power]=sudo /usr/lib64/nagios/plugins/trip/check_openmanage storage --non-redundant-power
command[check_openmanage_memory_non_redundant_power]=sudo /usr/lib64/nagios/plugins/trip/check_openmanage memory --non-redundant-power
command[check_openmanage_misc_non_redundant_power]=sudo /usr/lib64/nagios/plugins/trip/check_openmanage misc --non-redundant-power
  1. Create custom CheckCommand:
    1. Navigate to Icinga Director => Commmands => Add
    2. Command Type is Icinga Check Command
    3. Command is “openmanage”
    4. Command is “check_openmanage”
    5. Add arguments:
      1. config
        • Description: Which configuration to run: storage, memory, misc
        • Value: $config
        • Position: 0
        • skip key: true
        • required: true
      2. --non-redundant-power
        • Description: Description
        • No Value
        • Not Required.
        • Don’t skip key
        • Define set_if condition (for us: based on host.vars)
  2. Create new apply services:
    1. I think there is something we can do with for loops here to define one apply Service and have 3 checks applied to each host: storage, memory, and misc
  3. Make sure hosts get check applied.

I know an alternative option is to write a file in global-templates, but I want to make sure I know the steps to define complicated things in Director because we are hoping to allow other teams to submit checks and services to be applied to their hosts and I would like for them to operate solely through Director. So I’m trying to get the process for both a simple and complicated case cleared up for me.