Icinga2 Snmptrap/Notifications do not work at all

Hello everyone

The Problem described in the title is something I’m working on for 2 months now without having found a solution to it, there could be similar problems to mine already, but my Situation stated below is something I haven’t seen yet.
For a background Info: I’m totally new to Icinga and also new to this community forum, also I started directly with Icinga2 so I have no experience with Icinga1 whatsoever.
My Situation is as follows: I had to install Icinga2 from scratch (as rpm Packages with all the dependencies etc., because there is no internet connection for this host and there was no chance to get one) this Icinga2 host will take over for an older Icinga1 host (those are two different hosts). The person before me had all the knowledge of Icinga1 & 2 and constructed all the needed configurationfiles for Icinga2 (based on the old Icinga1 Instances configs) before departing, so the next person in charge wouldn’t have to change or do that much, sadly the aforementioned lack of an Internet connection, the person in question not working here anymore, the config files being not complete etc. made it all a lot more complicated than it might be.
The thing in question is: Getting Notfications and most importantly of all snmptraps (which are configurated as Notifications) to work.
So here are some Informations about the host and the installation in general:

  • Icinga2 Version: 2.12.2-1
  • Icingaweb2 Version: 2.8.2
  • OS: Centos7
  • Dependencies: Python 2.7.5 and PHP 7.3.11 (I know those aren’t all, but I don’t know which dependencies are helpful/needed for your knowledge so if you need something specific, please just ask me and I will try to deliver everything I can.)
  • What have I tried so far: I looked up the official Documentation and Trobuleshooting sections for everything that had remotely (directly or indirectly) to do with notifications + I searched up a lot of icinga community posts about snmptraps and notifications, but then again, I’m a total rookie to Icinga2 so I can’t really understand what is missing (or not) or what I actually have to do here, so bare with me if for you the solution might be crystal clear in the first place, for me it is not and I really need a bit of guidance when it comes to this.
  • I also have included two screenshots below, one for the daemon and one for the featurelist.

Output icinga2 daemon -C:

Output featurelist:

At this point I thank you in advance for your support, if you need anything more, please just ask ahead and I wish you a pleasent rest of the week.

Cheers,
Andy

Hi and welcome,

could you tell us how you configured the snmptrap functionallity? Which 3rd party tools or icingaweb modules do you use? How does the service check look like etc.

Icinga2 itself can’t handle snmptraps by the way. You need tools like net-snmp, logstash with the modules snmptrap-input-plugin and logstash-output-icinga plugin, icingaweb module like trapdirector etc.

Also would be interesting how you cofigured the notification in icinga. Does ist work with other checks?

Hi Stevie, thank you for answering!

Sure, so I read that there should be something extra, that handles the snmptraps, but I don’t know how to do it and what to use, because for me it looked like the old Icinga1 Instance had it working without all of this, but I might be wrong, I don’t have the correct information about that and don’t know what to look for, so I can’t compare it to the new Instance.

I installed some rpm packages that had to do with net-snmp (like utils etc.), but that’s about it.

There is something that I forgot to mention before: I send snmptrap heartbeats to the same destination hosts and those work perfectly, but those aren’t handled as notifications but rather as cronjobs.

Now to the important stuff, here are the requested informations and some more. The IPs, descriptions etc. are all dummies (security reasons) so if you see only a single letter or number don’t panic, I just wrote something else in the place of what is there:

notifications.conf

object NotificationCommand "a-snmptrap-host-notification-command" {
	command = [ PluginDir + "snmp.sh", "$host.vars.i_infos.p$", "$host.last_state_change$", "$host.name$", "$host.vars.i_infos.id$", "$host.vars.i_infos.moc$", "$host.i_infos.moi$", "$host.state$", "$host.i_infos.pcc$", "$host.output$" ]
}

object NotificationCommand "a-snmptrap-service-notification-command" {
	command = [ PluginDir + "snmp.sh", "$host.vars.i_infos.p$", "$service.last_state_change$" ,"$host.name$", "$service.i_infos.id$", "$service.vars.i_infos.moc$", "$service.vars.i_infos.moi$", "$service.state$", "$service.vars.i_infos.pcc$", "$service.output$" ]
}

apply Notification "a-snmptrap-host-notification" to Host {
	if (len(host.vars.notification.snmptrap.users) == 0 && len(host.vars.notification.snmptrap.user_groups) == 0) {
		log(LogWarning, "config", "Host '" + host.name + "' does not specify required user/user_groups configuration attributes for notification '" + name + "'.")
	}
	
	//vars.notification_logtosyslog = true
	
	import "generic-host-notification-template"
	command = "a-snmptrap-host-notification-command"
	
	users = host.vars.notification.snmptrap.users
	user_groups = host.vars.notification.snmptrap.groups

	assign where host.vars.notification.snmptrap && typeof(host.vars.notification.snmptrap) == Dictionary
}

apply Notification "a-snmptrap-service-notification" to Service {
	if (len(service.vars.notification.snmptrap.users) == 0 && len(service.vars.notification.snmptrap.user_groups) == 0) {
		log(LogWarning, "config", "Service '" + service.name + "' does not specify required user/user_groups configuration attributes for notification '" + name + "'.")
	}
	
	//vars.notification_logtosyslog = true
	
	//log(LogWarning, "config", "users: '" + service.vars.notification.snmptrap.users + "' groups: '" + service.vars.notification.snmptrap.groups + "' assign: '" + service.vars.notification.snmptrap)
	
	import "generic-service-notification-template"
	command = "a-snmptrap-service-notification-command"
	
	users = service.vars.notification.snmptrap.users
	user_groups = service.vars.notification.snmptrap.groups

	assign where service.vars.notification.snmptrap && typeof(service.vars.notification.snmptrap) == Dictionary
}

templates.conf

template Notification "generic-host-notification-template" {
	interval = 24h
	period = "7x24"
	types = [ Problem, Recovery ] // DowntimeStart, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart, FlappingEnd
	states = [ Up, Down ] 
}
	
template Notification "generic-service-notification-template" {
	interval = 24h
	period = "7x24"
	types = [ Problem, Recovery ] // DowntimeStart, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart, FlappingEnd
	states = [ OK, Warning, Critical, Unknown ] 
}

template Host "linux-host-template" {
	import "generic-host-template"
		
	vars.os_type = "Linux"
}

template Host "generic-host-template" {
	check_command = "hostalive"
	max_check_attempts = 4
	check_period = "7x24"
	check_timeout = 20s
	check_interval = 5m
	retry_interval = 1m
	enable_notifications = true
	enable_active_checks = true
	enable_passive_checks = false
	enable_event_handler = false
	enable_flapping = false
	enable_perfdata = true
		
	vars.os_type = "Unknown"
	vars.location = "Unknown"
	vars.project = "G"
	vars.i_infos = { p = "G", id = "G1", moc = "Node", moi = "Node", pcc = "123456" }
	vars.notification["mail"] = { users = [ "a" ]}
	vars.notification["snmptrap"] = { users = [ "i" ]}
}

template User "generic-user-template" {
	enable_notifications = true
	period = "7x24"
	types = [ Problem, Recovery ]
	states = [ OK, Warning, Critical, Unknown ]
}

hosts.conf

#That's what a Host would look like in this conf
object Host "test-host-01" { 
    	import "linux-host-template"
    	address = "15.15.15.15"
    	notes = "sampletext"
    	vars.i_infos = { p = "G", id = "G1", address = "15.15.15.15" }
    }

hosts File linux

10.10.10.10 1.TEST.SERVER.HOST
20.20.20.20 2.TEST.SERVER.HOST

/usr/lib64/nagios/plugins service being used snmp.sh

#!/bin/bash

echo "Running $0 ($#)..."

#
# initialize varibles from arguments
#
S_P_CODE=${1}
I_ALARM_RAISED_TIME=${2}
S_NODE=${3}
S_E_ID=${4}
S_M_O_C=${5}
S_M_O_I=${6}
S_STATUS=${7}
I_P_C_C=${8}
S_ADDITIONAL_TEXT=${9}

# test 10.10.10.10 20.20.20.20

#send snmp trap
/usr/bin/snmptrap -v 2c -c public 1.TEST.SERVER.HOST "" MIB::gEventNotification sPCode s "$S_P_CODE" iAlarmRaisedTime i $I_ALARM_RAISED_TIME sNode s "$S_NODE" sEId s "$S_E_ID" sMOC s "$S_M_O_C" sMOI s "$S_M_O_I" sNagiosState s "$S_STATUS" iPCC i $I_P_C_C sAdditionalText s "$S_ADDITIONAL_TEXT"

users.conf

object User "a" {
	import "generic-user-template"
	email = "testmail@test.com"
}

#The user in question not getting the traps Notification
object User "i" {
	import "generic-user-template"
}

RPM Packages installed with snmp

perl-Net-SNMP-6.0.1-7.el7.noarch.rpm
net-snmp-5.7.2-49.el7.x86_64.rpm
net-snmp-agent-libs-5.7.2-49.el7.x86_64.rpm 
net-snmp-utils-5.7.2-49.el7.x86_64.rpm

Used Icingaweb2 Modules

Only Monitoring is enabled

Also, is there a way to know, how I can check which program or service is used on the old instance for sendning snmptraps/mails notification? That would actually help a lot too.

Thanks in Advance.

Cheers,
Andy

Hi,

ok, as I wrote Icinga2 has no trap receiver included. But what Icinga can handle is passive check results (look here Advanced Topics - Icinga 2)
What you have to do now is to choose a way to handle the traps and push it into icinga. As I wrote above we choosed logstash. Others are using the new icingaweb2 module mentioned above.

With a little scripting could also use net-snmp whose daemon listens to traps and processed the messages for icinga.
So there are a lot possibilities and you should choose the way and programm you like. If the trap messages are shipped to icinga, also the notification rules should work

By the way, Im curious about this statement from you

How this works? Because if this works with icinga2, the rest should also work. I saw your script. Or do you only mean that the destination is receiving the snmp trap which is sent by net-snmp commands independent of Icinga?

Hi Stevie,

Sorry for the late response on my behalf.
Yeah, so the script that sends the heartbeats looks like this:

#!/bin/bash

#
# this script checks the status of icinga2 and sends a heartbeat message to imon
#
# the following line has to be added to the crontab with the command crontab -e:
# * * * * * /usr/lib64/nagios/plugins/a_send_heartbeat.sh

# check status of nagios/icinga server service
systemctl status icinga2 | grep 'active (running)' > /dev/null
if [ $? -eq '0' ]
then
	#send heartbeat trap to hosts
	/usr/bin/snmptrap -v 2c -c public 1.TEST.SERVER.HOST "" MIB::gGHeartBeatNotification sPCode s "G"
	/usr/bin/snmptrap -v 2c -c public 2.TEST.SERVER.HOST "" MIB::gGHeartBeatNotification sPCode s "G"
	#/usr/bin/snmptrap -v 2c -c public 1.PROD.SERVER.HOST "" MIB::gGHeartBeatNotification sPCode s "G"
	#/usr/bin/snmptrap -v 2c -c public 2.PROD.SERVER.HOST "" MIB::gGHeartBeatNotification sPCode s "G"
fi

And the comment in this script regarding the crontab is what I set up and those work (even the for now commented ones), but I must be honest, I don’t know why those work and the others don’t. So I thought that it might be something with the notifications.conf that doesn’t work or maybe there is something I’m missing. I will definitely try to set up the trapdirector module for icingaweb2.

Edit: I found something right now, on the old instance (Icinga1) there is a mib2c folder with a bunch of files and in the /usr/share/snmp directory there are also a bunch of mib2c .conf files and 2 perl scripts apparently named snmp_perl.pl & snmp_perl_trapd.pl which are all missing in the new instance.

Also: How are the configs I send you? Do you see something off/missing or is everyting fine with them?

By calling your script via cronjob is this independent from icinga.

What I miss here is the service definition in Icinga. And how it looks like in the web, if you say this is working. In your post above we only see the notification and host definition.

Maybe this is a good idea. For this module there still exists some threads with some information, if you need further informations: Search results for 'trapdirector category:9' - Icinga Community

Just to get this straigt, are you trying to RECEIVE SNMP-Traps from switches or something like that?
Currently it looks like you are doing the alarming part (notifications) with snmptraps.
Is that on purpose?

Hi Lorenz

Yes, that’s on purpose, because that’s what I’m looking for.
Sending SNMPTrap Notifications based on the state of the host/services Icinga2 monitors.

Hi Stevie,

Can you please specify a bit more which one you mean? I don’t really have any specific services for those notifications in the services.conf files. Or are you asking me to see the content of the services.conf files I’m using, because in that case here are some examples:

object Service "HOST_NETWORK_BONDING" {
    import "active-service-template"
    host_name = "g1"
    check_command = "check_snmp_bond_windows"
    notes = "Sample Text"
    vars.notification.users = [ "i","a","c" ]
    vars.imon_infos = { id="G-2", moc="Network", moi="Team0", pcc="1" }
    vars.check_arg1="7,9,10"
    vars.check_arg2="9,10"
    vars.check_arg3="7"
}

object Service "HOST_CPU_LOAD" {
    import "active-service-template"
    host_name = "g1"
    check_command = "check_snmp_cpuload_windows"
    notes = "Sample Text"
    vars.notification.users = [ "i","a","c" ]
    vars.imon_infos = { id="G-3", moc="CPU", moi="Node", pcc="2" }
    vars.check_arg1="70"
    vars.check_arg2="90"
}

Also what do you think about this, is it needed?:

Ok, because of your answer to @lorenz it looks like I misunderstood you completly. Sorry! I thought icinga should receive snmp trap and the notification doesn’t work after the trap is proccessed by icinga. So this wasn’t clear to me.

If I understand you correctly, based on your recent posts, icinga should send notification (like mails) if something is in a non-ok state, right? And if so and you write, running the script manually works, you should test the permisions.
That means run the notification script under the icinga user. Is this working? If you are using something like SELinux, maybe you see something in the logs, why icinga can’t trigger the notification script. Maybe the icinga user isn’t allowed to use the net-snmp library/commands or access the snmp port.

the two mentioned perl script I don’t know

Ou I thought I made it clear, sorry for that.

Yes you understand it right, if a Service is in a for example non-ok state, it should send an snmptrap notification to a specific user (here in the scripts mentioned as “i”). But yeah, I will check the permissions thanks for the suggestion!

Also, based on the fact that you missunderstood me does that mean, that the trapdirector isn’t needed in this instance?

Yes, the trapdirector is a icingaweb2 module if icinga should be able to receive and process snmp traps (e.g. from switches like @lorenz asked)

Got it, thanks.

I will look up on the permissions.

Edit: So, I checked the permissions and the Icinga User has the permission to execute/read/write the script which sends the traps. SELinux is disabled so far. I tested this with the linux “test” command like so:

sudo -u icinga test -r /usr/lib64/nagios/plugins/snmp.sh; echo Yes

using -w or -x instead of -r checks for writability and executability

And I logged in as Icinga User, tried the same manual trap command from the console (the same seen on the snmp.sh file I posted) and monitored it with tcpdump and it was sent successfully everytime.

The file in question is owned by root (as user and group) and has rwxr-xr-x (755) as permissions set.
(Same applies to the mail notification script and that one works normally)

How do I check this?

An update to the situation:

I got the notifications (Traps and Mails) working now.
I don’t know what of it really made it, but I will write here what I changed until it worked:

  • There were “/” missing in the commands for the notifications directing to the script, so Icinga was getting a path like this: /this/is/the/path/to/the/scriptscript.sh. Also some typos like the wrong variable name and a variable not being in the .vars Macros were other problems that affected it and I think that’s the main reason it was solved.

  • I also added the Icinga User to the sudoers file granting it a ALL=(secure) access to the snmp.sh script specifcly even though like I said, that User had the permissions to do everything with it.

So all in all the main problem why it wasn’t working, were typos and missconfigurations from my ex-colleague leaving and I was here basically just debugging everything when in the end it just was a “you forgot a ; / wrote helo instead of hello” type of thing.

For others that will see this post: Check your parameters/variables and how they are written or in which sequence they get passed, that will save you a lot of trouble and if everything is ok, check the permissions/SELinux/Firewalls/Ports and other stuff we discussed in here.

And of course thank you Stevie, I appreciated your help a lot.

2 Likes

I am happy to read that it is now working. Often it is really only the “little things” that have a big impact like typings. I know this things also.

Because you asked above how to check the maybe missing for net-snmp: That would be the case if you had activated SELinux. Because you would see something like “denied” in the SELinux-Logs.

Ok, thanks for the information, that might be something to look out for, if I decide to enable SELinux at some point.

Thanks again for your help!

Cheers

1 Like