# Telegram notifications not getting triggered from Icinga2

Icinga 2
1

Documentation used : url Monitoring-Notification-Via-Telegram

OS : CentOS Linux 8
Icinga2 version : 2.12.1-1
Installation source / method :

dnf install -y https://packages.icinga.com/epel/icinga-rpm-release-8-latest.noarch.rpm
dnf install icinga2

This is a testbed machine, and I am only monitoring one host (my laptop) at the moment, I am a first time Icinga user and want to check all functionalities required in my case on this testbed machine before replicating anything onto production.
At the moment, the monitoring part works well, the email notifications are getting triggered, I am receiving them on my Gmail. What is not working is the ‘Telegram’ notification, I followed the instructions given in the link above. I verified the Chat and Token ID from the terminal and those test notification are coming in, but through the application, it’s not getting triggered.

I also tried the python script ‘icinga2telegram’ option by following the link url Icinga2Telegram but even that did not work. Email notification continue to come in both cases, but not the Telegram. I’ve now reverted back to how it was earlier.

@dnsmichi Attaching configuration of my basic setup.

…/conf.d/users.conf

object User "teleadmin" {
  	import "generic-user"
  	display_name = "Telegram Admin"
  	enable_notifications = true
  	vars.telegram_chat_id = "-00000000000000"
}

…/conf.d/templates.conf

template Notification "telegram-host-notification" {
  command = "telegram-host-notification"
  period = "24x7"
}

…/conf.d/notifications.conf

apply Notification "telegram-notify" to Host {

        import "telegram-host-notification"
        assign where host.address
        users = host.vars.notification.telegram.users
}

…/conf.d/hosts.conf

object Host "CA" {
        address = "192.168.1.10"
        display_name = "CALLISTO"
        check_command = "ssh"
        vars.opsys = "CentOS"
        vars.notification["mail"] = {
                groups = [ "icingaadmins" ]
                users = [ "icingaadmin" ]
        }
        vars.notification["telegram"] = {
                users = [ "teleadmin" ]
        }
}

…/icinga2/constants.conf

const TelegramBotToken = "${11111111111111111111111111111AAAAAAAAAAAAAAAAAAAA}$"

Thanks, Best

2

Bringing this to the notice of veterans @twidhalm @unic @gschoenberger @phil-or

3

Is this your actual Telegram Bot Token or did you replace it with a personal version? This looks like some example code.

And did you check /var/log/icinga2/icinga2.log for errors or at least Telegram Notifications being sent?

4

Hi, Thomas, I replaced my token and chat_id. The rest of the code is just the way it is on the machine. Also I checked the log file, but it has no reference or errors relating to ‘Telegram’ or ‘notification’. There is just one error and it’s related to ‘Exception during Graphite operation: Can’t connect to Graphite on host 127.0.0.1 port 2003’

Thanks, Best

5

If you don’t see the notification being fired, then it’s not about the Telegram integration, it’s about your notification not being triggered. Double check your notification configuration and compare the ones for e-mail and telegram.

6

Hi @twidhalm
There are quite a few differences between the mail and telegram syntax, but unless I know what I am looking for, I don’t know which of those differences are responsible for this problem.
I had documented my steps while doing the installation, I’ll list a few those few differences I have noticed in my case.

The link I followed, talked about creating a const TelegramBotToken = "${TOKEN}" in the constants.conf file.
But the first time, I did that, it threw an error critical/config: Error: Validation failed for object
‘telegram-service-notification’ of type ‘NotificationCommand’; Attribute ‘env’ -> ‘TELEGRAM_BOT_TOKEN’: Closing $ not found in macro format string ‘${11111111111111111111111111111AAAAAAAAAAAAAAAAAAAA}’.

Therefore I changed that to const TelegramBotToken = "${11111111111111111111111111111AAAAAAAAAAAAAAAAAAAA}$"

2: In the commands.conf file, I noticed, the command which calls the mail script used ‘ConfigDir’, but the corresponding command in telegram used ‘SysconfDir’. Frankly, I don’t know what the difference is, I tried changing that statement to 'ConfigDir’, but that made no difference.

3: The templates.conf file had “states” and “types” defined for mail notification, but the link I followed had just the “command” and “period” statements for telegram. In the process of trying to replicate the mail notification, I tried putting in the “states” and “types” for the telegram template as well, yet again, the results made no difference.

template Notification "telegram-host-notification" {
  command = "telegram-host-notification"
  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  period = "24x7"
}

template Notification "telegram-service-notification" {
  command = "telegram-service-notification"
  states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  period = "24x7"
}

One last things I tried, is substituting the values of “token” and “chat_id” directly in the scripts /icinga2/scripts/telegram-host-notification.sh and /icinga2/scripts/telegram-service-notification.sh. When executed on their own, the scripts are able to send out blank notifications to telegram, but I can’t understand what’s preventing them from getting triggered from the application.

Thanks, Best

7

Thanks for the detailed information. That’s always a good thing to get more insight.

From what you’re saying, I think it’s more of a problem with Icinga not triggering the notification, not the actual notification being sent.

Maybe the following gives some hints what’s going wrong:

  • Enable debuglog feature
  • Have a Host or a Service that has both types of notification: E-Mail and Telegram with Custom type of notification enabled
  • Trigger both types of notifications via Icinga Web 2
  • grep the debug.log for the Name of the Host or Service

You should see whether Icinga 2 even tries to send the notification and if not, why. You can paste the lines here, if you’re not sure and we all should have a look.

1 Like
8

Thank you Thomas, @twidhalm for leading in the right direction, I enabled the debuglog

icinga2 feature enable debuglog

I have only one host on this test machine with both ‘Mail’ and ‘Telegram’ notifications enabled. The extracts from the debug log exactly pointed out where the problem was.

tail -f /var/log/icinga2/debug.log | grep CA

[2020-11-19 16:03:19 +0530] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'CA!telegram-notify'.

[2020-11-19 16:03:19 +0530] warning/PluginNotificationTask: Notification command for object 'CA' (PID: 11139, arguments: '/etc/icinga2/icinga2/scripts/telegram-host-notification.sh') terminated with exit code 128, output: execvpe(/etc/icinga2/icinga2/scripts/telegram-host-notification.sh) failed: No such file or directory

Having realised the command statement is causing the issue, I removed the leading /icinga2 from the path.

//command = [ ConfigDir + "/icinga2/scripts/telegram-host-notification.sh" ]
command = [ ConfigDir + "/scripts/telegram-host-notification.sh" ]

Although it resolved the earlier error, a new one has cropped up now

[2020-11-19 16:38:46 +0530] warning/PluginNotificationTask: Notification command for object 'CA' (PID: 12391, arguments: '/etc/icinga2/scripts/telegram-host-notification.sh') terminated with exit code 128, output: execvpe(/etc/icinga2/scripts/telegram-host-notification.sh) failed: Permission denied

A quick check shows, the ownership and execute permission on the scripts, are exactly the way they are for the mail scripts.

Thanks, Best

9

Hey, great! Congrats. :tada:

10

Would you kindly hit the “Solution” checkbox when one of the posts lead you on the correct path? This way you not only give kudos but also show others that you’re no longer needing any help.

11

Thank you Thomas, but the notifications are still not coming in, the error has changed from invalid path to permission denied. I checked the permissions on the script and files, but can’t find anything unusual.
Is there anything else I can check…?

12

Oh sorry, I didn’t read your reply thoroughly enough.

Maybe a problem with SELinux?

Always use the same user Icinga is running with (icinga on RedHat, nagios on Debian) when testing Plugins - no matter if it’s Monitoring or Notification. Did you do that? This should shed some light why Icinga is not allowed.

1 Like
13

You were absolutely right, Selinux context type was causing the permission denied issue.

The default mail scripts had the context type nagios_notification_plugin_exec_t and the telegram scripts had context type icinga2_etc_t
I changed the Telegram scripts context type to reflect the same as Mail scripts and bang on! Telegram notifications have started coming in.

semanage fcontext -a -e ./mail-host-notification.sh ./telegram-host-notification.sh
semanage fcontext -a -e ./mail-host-notification.sh ./telegram-service-notification.sh

restorecon -vRF ./*

tail -f /var/log/icinga2/debug.log | grep CA

[2020-11-19 19:48:30 +0530] information/Notification: Completed sending 'Problem' notification 'CA!telegram-notify' for checkable 'CA' and user 'teleadmin' using command 'telegram-host-notification'.

Thank you once again, Thomas. It was great learning!

1 Like