Icinga Director cant add service-template to service-set

Apparently i somehow destroyed my icinga-director Service sets.

First things first:
I created a service-set and added 3 service templates to it and assigned it to a host-template.
So far so good, all 3 services are beeing executed as they should.
I then moved forward and created a 4th service-template with the check command iostat.
I tried adding it, but the service didnt show up in the overview.
After trying to readd, i got the follow error, because director thought that the service-template is already there.
→ Trying to recreate icinga_service (“{“service_set_id”:“2”,“object_name”:”

If i click on services within the serviceset, my 3 services are showing like they should.
When i click on service within serviceset and add the service-template, i can only see the iostat template within services. I then have to go out of director tab and switch back in. Then i see my 3 templates that already have been there, but iostat is not there. I cant find where it is, and i dont undertand why it shows me that is there when i create it (AND it doesnt show the others) but when i reopen it, vice versa.

Funny enough, now when i click on any of my 3 service-templates, they state a “not found error” in the director.
I have no clue why this occured, because i was following the documentation and it did work. Except when i started to add iostat.

Does anybody have a clue why that could have happened? If you need any information / Log input, feel free to ask me for it.

I will try to investigate it on my own, but its kind of frustrating as i dont know where to start.

Service Set - preview shows all templates except iostat. (Makes it even more confusing that director shows not found error.+

This happens when i click on any of the service-templates within services in the serviceset:

All 3 services showing, and they are beeing executed, they are also showing in the preview. But somehow i cant open them. Therefore im not even able to delete them. I could remove and add the whole service set, but thats my last step, as i want to understand the problem first.

Any help is kindly appreciated.
Best regards,
Nicolas Kumhofer

Update:

I changed the name of the serviceset, before problems occured. Therefore i could restore the former object, which left me with 2 service sets.
Configured the “old” one accordingly and now it is somehow working??
Atleast when i click on any service template within the services tab of the serviceset, i still get the error described above.

I mean it works, so the problem isnt that big of a deal, would still prefer having a monitoring solution that doesnt throw random errors by the time.

I would leave the topic open for now, as it is not really closed for me.

BTW: when i click the link to the service within the preview tab of the serviceset it redirects me to the correct service-template. So it only affects the services tab within the serviceset.

Hi,

is there any update from developers or Netways regarding this issue? We are expiriencing it and for us is pretty much big deal cause it is potential “messy” bomb.

Our workaround solution is to clone problematic service set and this is working. But it is pretty much messy for those which containts apply rules

We have in our test env, updated to newest icinga2 version and issue is still same.

r2.13.7-1

For us problem starts with version: r2.13.3-1 and following version of components

Error:

Error in plain text:

Not found

#0 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(507): Icinga\Module\Director\Controllers\ServiceController->loadObject()

#1 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(100): Icinga\Module\Director\Web\Controller\ObjectController->loadOptionalObject()

#2 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(69): Icinga\Module\Director\Web\Controller\ObjectController->initializeWebRequest()

#3 /usr/share/icingaweb2/modules/director/application/controllers/ServiceController.php(50): Icinga\Module\Director\Web\Controller\ObjectController->init()

#4 /usr/share/php/Icinga/Web/Controller/ActionController.php(170): Icinga\Module\Director\Controllers\ServiceController->init()

#5 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(59): Icinga\Web\Controller\ActionController->__construct()

#6 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()

#7 /usr/share/php/Icinga/Application/Web.php(290): Zend_Controller_Front->dispatch()

#8 /usr/share/php/Icinga/Application/webrouter.php(105): Icinga\Application\Web->dispatch()

#9 /usr/share/icingaweb2/public/index.php(4): require_once(String)

#10 {main}

Any new additional info about it? Sorry in advance if my googling skills missed already existen topic/topics

Hi everyone,

We’re still experiencing an issue and actively searching for a solution. Has anyone else encountered the same problem?

I have open issue for couple months ago and had no luck to solve it.

Thank you in advance for sharing any insights.

Kind regards,
J.

can you prepare a basket with command, service-template, serviceset that explicitly reproduces that issue?

sure will try to prepare and get back with more info … we didnt used so far basket and will need go trough that I dont share some sensitive information couse some templates/checks contain such info.

Hi,

sorry for waiting. Below is one of the service set’s affected with issue.

{
    "ServiceSet": {
        "set_win_nrpe_2022_mind_5665": {
            "object_name": "set_win_nrpe_2022_mind_5665",
            "object_type": "template",
            "services": {
                "svc_windows_nrpe_cpu": {
                    "enable_notifications": true,
                    "fields": [],
                    "imports": [
                        "tp_windows_nrpe"
                    ],
                    "max_check_attempts": "10",
                    "object_name": "svc_windows_nrpe_cpu",
                    "object_type": "object",
                    "vars": {
                        "nrpe_arguments": [
                            "crit=99",
                            "time=5m"
                        ],
                        "nrpe_command": "check_cpu",
                        "nrpe_port": "5665",
                        "nrpe_version_2": true
                    }
                },
                "svc_windows_nrpe_disk_C": {
                    "enable_notifications": true,
                    "enable_perfdata": true,
                    "fields": [],
                    "imports": [
                        "tp_windows_nrpe"
                    ],
                    "max_check_attempts": "10",
                    "object_name": "svc_windows_nrpe_disk_C",
                    "object_type": "object",
                    "vars": {
                        "nrpe_arguments": [
                            "ShowAll",
                            "MaxCritUsed=95%",
                            "Drive=C",
                            "MaxWarnUsed=90%"
                        ],
                        "nrpe_command": "CheckDriveSize",
                        "nrpe_port": "5665",
                        "nrpe_version_2": true
                    }
                },
                "svc_windows_nrpe_eventlog_6008": {
                    "fields": [],
                    "imports": [
                        "tp_windows_nrpe"
                    ],
                    "max_check_attempts": "10",
                    "object_name": "svc_windows_nrpe_eventlog_6008",
                    "object_type": "object",
                    "vars": {
                        "nrpe_arguments": [
                            "file=system",
                            "MaxWarn=1",
                            "MaxCrit=1",
                            "filter=generated gt -2h AND id IN (6008)",
                            "truncate=1023"
                        ],
                        "nrpe_command": "CheckEventLog",
                        "nrpe_port": "5665",
                        "nrpe_version_2": true
                    }
                },
                "svc_windows_nrpe_mem": {
                    "enable_notifications": true,
                    "fields": [],
                    "imports": [
                        "tp_windows_nrpe"
                    ],
                    "max_check_attempts": "10",
                    "object_name": "svc_windows_nrpe_mem",
                    "object_type": "object",
                    "vars": {
                        "nrpe_arguments": [
                            "ShowAll",
                            "MaxCrit=99%",
                            "type=physical",
                            "MaxWarn=98%"
                        ],
                        "nrpe_command": "checkmem",
                        "nrpe_port": "5665",
                        "nrpe_version_2": true
                    }
                },
                "svc_windows_nrpe_services_automatic": {
                    "fields": [],
                    "imports": [
                        "tp_windows_nrpe"
                    ],
                    "max_check_attempts": "10",
                    "object_name": "svc_windows_nrpe_services_automatic",
                    "object_type": "object",
                    "vars": {
                        "nrpe_arguments": [
                            "exclude=SysmonLog",
                            "exclude=WbioSrvc",
                            "CheckAll",
                            "exclude=wuauserv",
                            "exclude=mfehidin_svc",
                            "exclude=ManageMyPhone",
                            "exclude=tiledatamodelsvc",
                            "exclude=DsmSvc",
                            "exclude=RemoteRegistry",
                            "exclude=ShellHWDetection",
                            "exclude=sppsvc",
                            "exclude=MSExchangeNotificationsBroker",
                            "exclude=TrustedInstaller",
                            "exclude=clr_optimization_v4.0.30319_32",
                            "exclude=edgeupdate",
                            "exclude=Citrix.Xip.ClientService",
                            "exclude=stisvc",
                            "exclude=gupdate",
                            "exclude=MapsBroker",
                            "exclude=clr_optimization_v4.0.30319_64",
                            "exclude=AeLookupSvc",
                            "exclude=xdrhealth"
                        ],
                        "nrpe_command": "CheckServiceState",
                        "nrpe_port": "5665",
                        "nrpe_version_2": true
                    }
                },
                "svc_windows_nrpe_uptime": {
                    "enable_notifications": false,
                    "fields": [],
                    "imports": [
                        "tp_windows_nrpe"
                    ],
                    "max_check_attempts": "10",
                    "object_name": "svc_windows_nrpe_uptime",
                    "object_type": "object",
                    "vars": {
                        "nrpe_arguments": [
                            "crit= uptime > 350d",
                            "warn=uptime > 400d"
                        ],
                        "nrpe_command": "check_uptime",
                        "nrpe_port": "5665",
                        "nrpe_version_2": true
                    }
                }
            }
        }
    }
}

Following is the error when we want to edit service check from inside of service set from above

Error
 
Not found

#0 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(507): Icinga\Module\Director\Controllers\ServiceController->loadObject()
#1 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(100): Icinga\Module\Director\Web\Controller\ObjectController->loadOptionalObject()
#2 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(69): Icinga\Module\Director\Web\Controller\ObjectController->initializeWebRequest()
#3 /usr/share/icingaweb2/modules/director/application/controllers/ServiceController.php(50): Icinga\Module\Director\Web\Controller\ObjectController->init()
#4 /usr/share/php/Icinga/Web/Controller/ActionController.php(170): Icinga\Module\Director\Controllers\ServiceController->init()
#5 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(59): Icinga\Web\Controller\ActionController->__construct()
#6 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()
#7 /usr/share/php/Icinga/Application/Web.php(290): Zend_Controller_Front->dispatch()
#8 /usr/share/php/Icinga/Application/webrouter.php(105): Icinga\Application\Web->dispatch()
#9 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#10 {main}

Thank you,
J.

these things are missing, so i cant import that to my test environment

… appologies, took me a time to find right one … our enviroment is massive …

following is missing cmd conf:

{
    "ExternalCommand": {
        "nrpe": {
            "arguments": {
                "-6": {
                    "description": "Use IPv6 connection",
                    "set_if": "$nrpe_ipv6$"
                },
                "-4": {
                    "description": "Use IPv4 connection",
                    "set_if": "$nrpe_ipv4$"
                },
                "-2": {
                    "description": "Use this if you want to connect to NRPE v2",
                    "set_if": "$nrpe_version_2$"
                },
                "-A": {
                    "description": "The CA file to use for PKI",
                    "value": "$nrpe_ca$"
                },
                "-C": {
                    "description": "The cert file to use for PKI",
                    "value": "$nrpe_cert$"
                },
                "-H": {
                    "description": "The address of the host running the NRPE daemon",
                    "value": "$nrpe_address$"
                },
                "-K": {
                    "description": "The key file to use for PKI",
                    "value": "$nrpe_key$"
                },
                "-L": {
                    "description": "The list of SSL ciphers to use",
                    "value": "$nrpe_cipher_list$"
                },
                "-S": {
                    "description": "The SSL/TLS version to use",
                    "value": "$nrpe_ssl_version$"
                },
                "-a": {
                    "repeat_key": false,
                    "value": "$nrpe_arguments$",
                    "order": "1"
                },
                "-c": "$nrpe_command$",
                "-d": {
                    "description": "Anonymous Diffie Hellman use: 0 = deny, 1 = allow, 2 = force",
                    "value": "$nrpe_dh_opt$"
                },
                "-n": {
                    "description": "Do not use SSL",
                    "set_if": "$nrpe_no_ssl$"
                },
                "-p": "$nrpe_port$",
                "-t": {
                    "description": "<interval>:<state> = <Number of seconds before connection times out>:<Check state to exit with in the event of a timeout (default=CRITICAL)>",
                    "value": "$nrpe_timeout$"
                },
                "-u": {
                    "description": "Make socket timeouts return an UNKNOWN state instead of CRITICAL",
                    "set_if": "$nrpe_timeout_unknown$"
                }
            },
            "command": "check_nrpe",
            "fields": [
                {
                    "datafield_id": 123,
                    "is_required": "n",
                    "var_filter": null
                },
                {
                    "datafield_id": 76,
                    "is_required": "n",
                    "var_filter": null
                },
                {
                    "datafield_id": 77,
                    "is_required": "n",
                    "var_filter": null
                },
                {
                    "datafield_id": 124,
                    "is_required": "n",
                    "var_filter": null
                },
                {
                    "datafield_id": 78,
                    "is_required": "n",
                    "var_filter": null
                },
                {
                    "datafield_id": 95,
                    "is_required": "n",
                    "var_filter": null
                },
                {
                    "datafield_id": 392,
                    "is_required": "n",
                    "var_filter": null
                },
                {
                    "datafield_id": 393,
                    "is_required": "n",
                    "var_filter": null
                }
            ],
            "methods_execute": "PluginCheck",
            "object_name": "nrpe",
            "object_type": "external_object",
            "timeout": "300",
            "vars": {
                "check_address": {
                    "arguments": [],
                    "deprecated": false,
                    "name": "<anonymous>",
                    "side_effect_free": false,
                    "type": "Function"
                },
                "check_ipv4": "$nrpe_ipv4$",
                "check_ipv6": "$nrpe_ipv6$",
                "nrpe_address": "$check_address$",
                "nrpe_no_ssl": false,
                "nrpe_timeout_unknown": false,
                "nrpe_version_2": false
            }
        }
    },
    "Datafield": {
        "123": {
            "varname": "nrpe_address",
            "caption": "nrpe_address",
            "description": "The address of the host running the NRPE daemon",
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeString",
            "format": null,
            "originalId": "123",
            "settings": {},
            "category": null
        },
        "76": {
            "varname": "nrpe_arguments",
            "caption": "nrpe_arguments",
            "description": null,
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeArray",
            "format": null,
            "originalId": "76",
            "settings": {},
            "category": null
        },
        "77": {
            "varname": "nrpe_command",
            "caption": "nrpe_command",
            "description": null,
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeString",
            "format": null,
            "originalId": "77",
            "settings": {},
            "category": null
        },
        "124": {
            "varname": "nrpe_no_ssl",
            "caption": "nrpe_no_ssl",
            "description": "no SSL handshake to use",
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeBoolean",
            "format": null,
            "originalId": "124",
            "settings": {
                "rename_vars": "y"
            },
            "category": null
        },
        "78": {
            "varname": "nrpe_port",
            "caption": "nrpe_port",
            "description": null,
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeString",
            "format": null,
            "originalId": "78",
            "settings": {},
            "category": null
        },
        "95": {
            "varname": "nrpe_timeout",
            "caption": "nrpe_timeout",
            "description": "<interval>:<state> = <Number of seconds before connection times out>:<Check state to exit with in the event of a timeout (default=CRITICAL)>",
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeString",
            "format": null,
            "originalId": "95",
            "settings": {},
            "category": null
        },
        "392": {
            "varname": "nrpe_timeout_unknown",
            "caption": "nrpe_timeout_unknown",
            "description": null,
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeBoolean",
            "format": null,
            "originalId": "392",
            "settings": {},
            "category": null
        },
        "393": {
            "varname": "nrpe_version_2",
            "caption": "nrpe_version_2",
            "description": null,
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeBoolean",
            "format": null,
            "originalId": "393",
            "settings": {},
            "category": null
        }
    }
}

service template:

{
    "ServiceTemplate": {
        "tp_windows_nrpe": {
            "check_command": "nrpe",
            "check_interval": "300",
            "check_period": "notify_24",
            "check_timeout": "30",
            "enable_active_checks": true,
            "enable_event_handler": true,
            "enable_flapping": false,
            "enable_notifications": true,
            "enable_passive_checks": true,
            "enable_perfdata": true,
            "fields": [
                {
                    "datafield_id": 77,
                    "is_required": "n",
                    "var_filter": null
                },
                {
                    "datafield_id": 95,
                    "is_required": "n",
                    "var_filter": null
                }
            ],
            "max_check_attempts": "7",
            "object_name": "tp_windows_nrpe",
            "object_type": "template",
            "retry_interval": "30",
            "vars": {
                "nrpe_address": "$address$",
                "nrpe_timeout": "30",
                "nrpe_version_2": true
            }
        }
    },
    "Datafield": {
        "77": {
            "varname": "nrpe_command",
            "caption": "nrpe_command",
            "description": null,
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeString",
            "format": null,
            "originalId": "77",
            "settings": {},
            "category": null
        },
        "95": {
            "varname": "nrpe_timeout",
            "caption": "nrpe_timeout",
            "description": "<interval>:<state> = <Number of seconds before connection times out>:<Check state to exit with in the event of a timeout (default=CRITICAL)>",
            "datatype": "Icinga\\Module\\Director\\DataType\\DataTypeString",
            "format": null,
            "originalId": "95",
            "settings": {},
            "category": null
        }
    }
}

Hope this is all.

remove react and ipl it’s deprecated.
update icinga/icinga-php-library to 0.12.0

as far from that I still can’ reproduce this issue.

  • I changed the check_period manually to my own
  • I can create everything via cli.
  • I can click on the service set


Hi,

didnt help,

I have restarted icinga2 and director service and tested in our test env which should be replica from our prod

and still same error is present

Did I miss maybe some step? Have followed instructions on git and have just replaced files and old one removed

Did additional log check (messages, icingaweb2 and icinga2) nothing related found

J.

Hi,

I just wanted to provide an update on what I have found. It’s a very unusual situation.

I removed a problematic service set apply rule, and after doing so, the service set didn’t apply to any host objects. Suddenly, it appears to be functioning normally, and I can now modify and change service checks within this service set.

Below is an example where we manually removed the service set from multiple host objects that were previously added manually:

Now, we are able to change parameters within the service set:

We have also re-applied the same service set to the same host objects from which I had previously deleted it:

And the error no longer exists.

This behavior is the same whether it’s an apply rule set or manually adding a set of service check.

I believe we may have encountered a BUG here.

what was the problematic rule?

In the example above, it didn’t contain an apply rule.

In the case of the service set that has the same issue and contains an apply rule, it is based on conditions: host.groups == "some host group" another set which has same issue it was based on apply rule host.vars.customer_abbreviation == "some customer id" … also I think have encounter on various combiations two or more rules in one service set …

only thing what is common to all affected service set’s, is that issue appeard after we updated icinga to version 2.13.4. and director on 1.10.0 … if I am not mistaken … then we have updated to 2.13.7 and director on 1.10.2 … and it is still existen … and not all of them are affected after upgrading … but most of them …

Hi,

I just want to push post a bit further. What I described in the previous post is not an acceptable solution. We cannot simply move and apply rules that affect approximately 2,000 service checks, as it could potentially cause chaos with the alerting system.

Any advice or insights from someone more knowledgeable than me would be greatly appreciated.

J.

Hi just wanna give update regarding this issue. With newest director version 1.11.0 bug is not existen any more. It is working as expected. Got finally time to test it.

Cheers!