hi, the log file is /var/log/icinga2/icinga2.log.
There is no log file there
Here are the features list
Disabled features: compatlog debuglog elasticsearch gelf graphite livestatus opentsdb perfdata syslog
Enabled features: api checker command ido-mysql influxdb mainlog notification statusdataNormally the Directory should look like this:
You could try to turn on the debuglog Feature. Then you’ll get also a file “debug.log” in this directory with much more Information. But don’t forget to turn this feature off after you finish debugging. Because this file is growing very fast, depending on your configuration.
Hey Stevie,
I found the log files(was originally looking in /etc not /var)
The log file was enormous so I erased the entire thing and then put my vm in critical state and still no notification and no additional logs
I also turned on the debuglog and also got nothing in there
Thanks Joseph
is your host reaching the hard state before you expect a notification? If i’m correctly with check_interval und retry_interval your host should arrive the hard state after 30-35 min! So after this time Icinga sends with your configuration the notification if the hosts arrives that state. The Default value for max_check_attempt is 3. For testing you can set the max_check_attempt option to 1 and the retry interval to 1m. So should get an the notification earlier.
if you have the icinga2 book, there is a very good Explanation about notification incl. a graph.
1 Like
Hey Stevie,
I changed the max_check_attempt to 1 and retested and still no notification being sent
Thanks Joseph
Look into your Icingaweb2 --> System --> Monitoring Health:
Did you enable the Notifications?
1 Like
What do you mean by this?
Are there literally no lines at all in both logs?
Check the log around the time the notification should be sent for lines like
information/Checkable: Checking for configured notifications for object
-and/or-
notice/Notification: Attempting to send notifications for notification object ....
-and/or-
notice/Notification: Not sending notifications for notification object....
You can also check some variables on the service object via the API:
https://localhost:5665/v1/objects/notifications?filter=match("<service_name>", service.name)
Output looks something like this:
{
"attrs":
{
"__name":"<hostname>!<service_name>!<notification_name>",
"active":true,
"command":"alarm-service_autoticket",
"command_endpoint":"",
"ha_mode":0.0,
"host_name":"<hostname>",
"interval":0.0,
"last_notification":1513774196.1616489887,
"last_problem_notification":1513774196.1616489887,
"name":"<notification_name>",
"next_notification":1515498467.2694029808,
"no_more_notifications":true,
"notification_number":0.0,
"notified_problem_users":["Autoticket-Intern"],
"original_attributes":null,
"package":"director",
"paused":false,
"period":"Always",
"service_name":"<service_name>",
"source_location":{"first_column":1.0,
"first_line":485.0,
"last_column":79.0,
"last_line":485.0,
"path":"/var/lib/icinga2/api/packages/director/something-1515418089-16/zones.d/something/notification_apply.conf"},
"states":["Critical"],
"templates":["<notification_name>",
"generic-service-alarm_autoticket",
"generic-service-alarm"],
"times":{"begin":3600.0},
"type":"Notification",
"types":["Problem"],
"user_groups":null,
"users":["Autoticket-Intern"],
"vars":null,
"version":0.0,
"zone":"something"
},
"joins":{},
"meta":{},
"name":"<hostname>!<service_name>!<notification_name>",
"type":"Notification"
}
]
}
Check the variables last_notification, last_problem_notification, next_notification, no_more_notifications, notification_number, notified_problem_users
1 Like
Yea Notifications are on
If I delete the file and then put my vm in crtitical mode(over 80% cpu usage) there are no additional logs that are put in icinga2.log the only time I see it atually put new logs is on restart
I took again a look to your Notification rule. I don’t see a Notification Command. You could also check this again. Also the Definition of your command. Here you can check if you can manually run the notification script.
If this is correct, i think it’s time for the debug log.
And maybe you should also ask your mail admin if he see something in his logs, if the mail server is receiving something from your icinga server.
1 Like
No need to bother the mail admin, on CentOS with the default postfix mta each sent mail is recorded in /var/log/maillog 
Another possible source of failure if you have a HA-setup: make sure that both master servers have the notification feature enabled. I had that problem a while ago, master1 didn’t care about notifications for a specific service and master2 had them turned off -> nothing was sent out.
I checked the maillog and I can entries hit the maillog if I send the notification manually. But if I am waiting for it to send automatically it never hits the maillog
Also for testing I enabled the notification feature on the vm
And this is not a HA setup it is a master server with several client servers
@rsturm thanks for the info, I didn’t know the log. Maybe there was never a Need to know it. Because of our big infrastructure and its confiuration.
@flaminggenius If you see log entries after sending it manually (i guess with the root-user/permissions) and not if icinga is sending it, i would say the icinga user misses some permissions for that (file permisisons, SELinux etc.).
1 Like
Hey Stevie,
When I say that I am sending it manually I mean I go to a service and hit send notification.
Also I do have selinux disabled
If there was a permissions issue then it wouldnt allow me to send it manually would it? Even manually it still within icinga so I know that icinga is capable of sending them but its not sending them automatically when a service enters a critical or warning state.
Thanks Joseph
hmm. ok if from the side of your OS is no permission problem and you can send it from the GUI than there must something wrong in your notification rule. But I don’t see an error.
Sorry, in this case I’ve run out of ideas. I hope other community members have more ideas for you.
1 Like
Thank you for trying to help hopefully someone else can provide some more clarity.
Thanks Joseph
Have you checked your service via the API for these variables? You can query the API with the URL via the browser or using curl.
1 Like
That would make no difference from what i can icinga is not being triggered to send a notification
This would make a difference, as you get information that is not available in the DB (afaik) and thus can maybe see why the notification is not triggered at all.