Config Validation Error after Notifications Apply Rule

[bak0523 (Gabriel Laszlo)]

It’s me again,

now, I never had a problem with Notifications even if its a bit tricky to set up.
I have recently installed a different Icinga Instance with Distributed Monitoring running. (currently 1 Satellite).

Everything is working as it should, but now when I apply a Notification Rule the config validation in the Icinga Director Deployment doesn’t work.

[2022-07-26 09:36:48 +0200] information/cli: Icinga application loader (version: r2.13.3-1)
[2022-07-26 09:36:48 +0200] information/cli: Loading configuration file(s).
[2022-07-26 09:36:48 +0200] information/ConfigItem: Committing config item(s).
[2022-07-26 09:36:48 +0200] information/ApiListener: My API identity: Icinga
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'mx10-32-60-177!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'bkwend!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'FEIG Terminal Sanitär!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'Fridge 1!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'MX-BR304-65884!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'mx10-32-58-208!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'FEIG Terminal Fridge 2!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'Touch-Panel 2 / Ausgabe!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'FEIG Terminal Fridge 1!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'Fridge 2!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'Server Hausautomatisation!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'mx10-32-108-205!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'bk-0001!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'Coffeebar Eversys Cameo!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'mx10-32-57-185!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'Touch-Panel 1 / Tür!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: Error: Validation failed for object 'FEIG Terminal Coffeebar!Host Notifications' of type 'Notification'; Attribute 'command': Object 'mail-host-notification' of type 'NotificationCommand' does not exist.
Location: in [stage]/zones.d/bkwend/notification_templates.conf: 2:5-2:38
[stage]/zones.d/bkwend/notification_templates.conf(1): template Notification "Host Notification Template - Email" {
[stage]/zones.d/bkwend/notification_templates.conf(2):     command = "mail-host-notification"
                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/bkwend/notification_templates.conf(3):     interval = 0s
[stage]/zones.d/bkwend/notification_templates.conf(4):     period = "always"
[2022-07-26 09:36:48 +0200] critical/config: 17 errors
[2022-07-26 09:36:48 +0200] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.

Now it clearly says “Object ‘mail-host-notification’ of type ‘NotificationCommand’ does not exist”.

Which is kinda wierd because on both Master and Staellite the Command exists and so does the script in /etc/icinga2/scripts/

What’s also wierd, on both Master and Satellite the icinga2 daemon -C returns a succesful config validation.

Also the notifcation feature is enabled on both…

Am I missing something?

Kind regards,
Gabriel

• Director version (System - About): 1.9.1
• Icinga Web 2 version and modules (System - About): 2.10.1
• Icinga 2 version (icinga2 --version): r2.13.3-1
• Operating System and version: Debian 11 LTS
• Webserver, PHP versions: PHP 2.4.46, 7.4.28

[bak0523 (Gabriel Laszlo)]

Update:

so, I decided to make new commands using the same script and now it works…

new

old

any Idea why the original commands were not working?

[log1c]

My guess is that the config file containing the command definitions by default (/etc/icigna2/conf.d/commands.conf) wasn’t included in the config.
The happens if the icinga2 node wizard is run with the default settings, which disabled the inclusion of the entire conf.d directory apart from the api-users.conf

As they were listed in the Director I assume you had run the Directors kickstart wizard prior to the icinga2 node wizard command, but didn’t use the commands until now.
The “old” commands will vanish from the Director when you run the Director kickstart again.

[bak0523 (Gabriel Laszlo)]

the icinga2 node wizard did run with default settings.

after the icinga2 node wizard i ran the kickstard wizard and in my changes i saw it deleted the mail-host-notification but i restored the objects because i obviously wanted notifications…

what I am trying to figure out right now is why it doesn’t send any emails… even if the .log looks good

[2022-07-26 13:19:19 +0200] information/ApiListener: New client connection from [::1]:35532 (no client certificate)
[2022-07-26 13:19:19 +0200] information/Checkable: Checkable 'bk-0001' has 1 notification(s). Checking filters for type 'Custom', sends will be logged.
[2022-07-26 13:19:19 +0200] information/Notification: Sending 'Custom' notification 'bk-0001!Host Notifications' for user 'monitoring@domain'
[2022-07-26 13:19:19 +0200] information/HttpServerConnection: Request: POST /v1/actions/send-custom-notification (from [::1]:35532), user: bkwapi00, agent: , status: OK).
[2022-07-26 13:19:19 +0200] information/HttpServerConnection: HTTP client disconnected (from [::1]:35532)
[2022-07-26 13:19:19 +0200] information/Notification: Completed sending 'Custom' notification 'bk-0001!Host Notifications' for checkable 'bk-0001' and user 'monitoring@domain' using command 'email-host-notification'.

It should work since everything has ben set up correctly.

is it a good idea to do no include the conf.d directory?

should I run the icinga2 node wizard again and if yes with what settings?

[log1c]

No, I don’t think that is necessary.

The conf.d directory holds only an example configuration. So it genreally is not recommended to include it.
I, personally, include the commands.conf out of convenience to have the notification commands in the Director after a kickstart without having to manually create them.

@bak0523

what I am trying to figure out right now is why it doesn’t send any emails… even if the .log looks good

Have a look at the mail log as well.

[bak0523 (Gabriel Laszlo)]

@log1c

I, personally, include the commands.conf out of convenience to have the notification commands in the Director after a kickstart without having to manually create them.

that makes a lot of sense… you have taught me something new!

@log1c

Have a look at the mail log as well.

an edit in the /etc/hosts did it.