Trying to reduce Noise from Icinga2 - Conditional Check - on Highly available Windows servers

Hi Team,

Hope you are doing good…

I have been thinking through the conditional windows service check. Suppose we have windows service like IIS Admin service on Server A & Server B. Icinga2 should only notify only if IIS admin goes down on both servers A & B. Icinga2 should not notify if IIS goes down either Server A or Server B.

And Another scenario.

There are few windows services & those should not run on both servers. Here, Icinga2 should notify when IIS admin service running both servers A & B. It should not notify when IIS admin service runs either A or B.

The reason why i am trying this is… We are running our applications in AWS… All of our windows servers are highly available with 2 servers for every application… we should not worry if any service down in any one of the server. We are getting so much noice & false alerts for every service on both servers…

i have tried with business process & somehow this is not worked…

Can someone please help on this?

Many thanks,
Srimani

Hello @mani.vajrala!

What exactly didn’t work with Business Process?

Have you tried check_multi?

Best,
AK

I have not tried with check_multi …

You want me try this post?

With Business Process: Appropriate configuration changes have been done… & still didn’t seems works.

Step1:

Create ClusterAmazonSSMAgent service block for AmazonSSMAgent service

apply Service “ClusterAmazonSSMAgent” {
check_command = “service-windows”
enable_notifications = false
vars.service_win_service = “AmazonSSMAgent” //access the host custom vars
command_endpoint = host.vars.client_endpoint

assign where host.vars.SSMAgent == “AmazonSSMAgent” && host.vars.client_endpoint
}

Step2:

Updated both host config files with var attributes

object Host “******-01.#######ANDCO.AWS” {
check_command = “hostalive”
address = “########”
vars.SSMAgent = “AmazonSSMAgent”

object Host “******-02.#######ANDCO.AWS” {
check_command = “hostalive”
address = “########”
vars.SSMAgent = “AmazonSSMAgent”
}

So that i can see the AmazonSSMAgent up & running on icinga2 with notifications disabled…

----------------------Adding BP---------------------

Step3:

Adding the BP Service block on same service config file in step 1

Apply Service “BP CluserService” {

check_command = “icingacli-businessprocess”
enable_notifications = true

check_interval = 1s
retry_interval = 30s

command_endpoint = “<Icinga2 Master(Linux box)>” // BP is a Icingweb2 module and the command should run where icingaweb2 is installed.

vars.icingacli_businessprocess_config = “ClusterAmazonSSMAgent” // Put in here the filename of your BP config without the extension.
vars.icingacli_businessprocess_process = “ClusterAmazonSSMAgent”
assign where host.vars.bpssm == “AmazonSSMAgent”
}

Step4: Added another dummy hostfile with different name as shown below

object Host “*****-cluster.#######ANDCO.AWS” {
check_command = “dummy”
check_interval = 24h
retry_interval = 24h
enable_notifications = true
vars.bpssm = “AmazonSSMAgent”
vars.dummy_text = “Cluster dummy host”
vars.dummy_state = 0
}

Step5: Added BP config file on Icinga2 Master

/etc/icingaweb2/modules/businessprocess/processes# cat ClusterAmazonSSMAgent.conf

ClusterAmazonSSMAgent = ******-01.#####ANDCO.AWS;ClusterAmazonSSMAgent | ******-02.#######ANDCO.AWS;ClusterAmazonSSMAgent
display 1;ClusterAmazonSSMAgent;ClusterAmazonSSMAgent

Restarted icinga2 & Tested the scenarios like shutdown the both the services & i have not seen any email notification.

My expectation is notification should only send when both the services are shutdown… If any one of the service shudown we should not get any email notification…

Here the BP Screenshots


You may also try putting notifications on a dummy service like this one: