Unable to reschedule service check via icingaweb2

Describe the bug

Service check reschedule is not working. It does nothing.

To Reproduce

Provide a link to a live example, or an unambiguous set of steps to reproduce this issue. Include configuration, logs, etc. to reproduce, if relevant.

  1. Select a service
  2. Click on “Reschedule” button and set next execution
  3. Check if “Next check” time changed

Expected behavior

Expectation is to be executed when scheduled but nothing changes.

Screenshots

Check next check time:
image

Reschedule, in this example 2 hours later:
image

Check next check time:
image

Hi,

very disappointing to see that you’ve just copied the GitHub issue here.

So, how about involved core and web versions, logs with traces, etc?

Regards,
Michael

Hi,

Sorry for that but is the same issue and my first post here.
We have IcingaWeb2 v2.6.3 (installed on a web server) and two cluster nodes running r2.10.5-1. Then we have various satellites, one per site, all running same version like the cluster nodes.

Please help me collect the logs with traces and any other data you need.

Thanks

Hi,

consider following the FAQ and always provide as much detail as one would need to understand your problem. Each distributed environment is different, and without configuration bits, it’s hard to guess.

Imagine that we don’t know about this, and also, the provided service/host being late after re-schedule, which zone does it belong to? Providing an overview into your distributed environment is key here, including the zones.conf file(s).

Whenever an API action fires the reschedule-check event, this generates a log entry on the master. The master knows which zone is responsible for checking the host/service, and forwards the request to the corresponding satellite. The satellite receives the cluster message and will itself force a re-check. This re-check can be seen in the logs too, as well as locally on the satellite via REST API and last_check and last_check_result attributes. Then the satellite sends back the check result via cluster message, the master(s) receive that message, process it, update last_check/last_check_result and optionally trigger notifications. Once a check result is processed, the IDO database backend is updated with *status and statehistory tables.

That’s the trace you need to analyse. Portions are only visible in the debug logs, so you’ll need to enable this for all involved endpoints.

Regards,
Michael

Hi,

I enabled debuglog feature on master and site sattelite and restarted icinga2 service. Now, having tail -f on debug.log of master I can see these lines when i click on “Reschedule”:

[2019-07-03 11:46:52 +0200] debug/HttpRequest: line: POST /v1/actions/reschedule-check HTTP/1.1, tokens: 3
[2019-07-03 11:46:52 +0200] information/HttpServerConnection: Request: POST /v1/actions/reschedule-check (from [ip]:44942), user: icingaweb2)
[2019-07-03 11:46:52 +0200] debug/HttpUtility: Request body: '{"next_check":1562150830,"force":true,"service":"host!APP_SP_Nintex_Workflow_Maintenance"}'
[2019-07-03 11:46:52 +0200] notice/ApiActionHandler: Running action reschedule-check

The thing is I can’t see anything in the satellite debug.log regarding this service… I looked for service name (APP_SP_Nintex_Workflow_Maintenance).

Inspect the state of the service on the master and the service via API.

curl -k -s -u root:icinga `https://localhost:5665/v1/objects/services/host!APP_SP_Nintex_Workflow_Maintenance'

Also, the zones.conf configuration is still missing to get an idea about your distributed setup.

This is the zones.conf file:

/*
 * Generated by Icinga 2 node setup commands
 * on 2017-01-16 11:20:52 +0100
 */

object Endpoint "mqde01icinga2cl01" {
        host = "10.10.2.63"
}

object Endpoint "mqde01icinga2cl02" {
        host = "10.10.2.64"
}


object Zone "master" {
        endpoints = [ "mqde01icinga2cl01", "mqde01icinga2cl02" ]
}

//GLOBAL CONFIG
object Zone "global" {
        global = true
}

object Zone "director-global" {
  global = true
}

//MQDE01
object Endpoint "mqde01icinga2wk01" {
//        host = "mqde1icinga2wk01"
        host = "10.10.3.2"
}

object Endpoint "mqde01icinga2wk02" {
//        host = "mqde1icinga2wk02"
        host = "10.10.0.209"
}


object Zone "MQDE01" {
        endpoints = [ "mqde01icinga2wk01", "mqde01icinga2wk02" ]
        parent = "master"
}


//MQRO01
object Endpoint "mqro01icinga2wk01" {
        host = "mqro01icinga2wk01"
}

object Zone "MQRO01" {
        endpoints = [ "mqro01icinga2wk01" ]
        parent = "master"
}


//MQCN01
object Endpoint "mqcn01icinga2wk01" {
        host = "mqcn01icinga2wk01"
}

object Zone "MQCN01" {
        endpoints = [ "mqcn01icinga2wk01" ]
        parent = "master"
}

//MQMX01
object Endpoint "mqmx01icinga2wk01" {
        host = "mqmx01icinga2wk01"
}

object Zone "MQMX01" {
        endpoints = [ "mqmx01icinga2wk01" ]
        parent = "master"
}


//MQMK01
object Endpoint "mqmk01icinga2wk01" {
        host = "mqmk01icinga2wk01"
}

object Zone "MQMK01" {
        endpoints = [ "mqmk01icinga2wk01" ]
        parent = "master"
}

//MQUS01
object Endpoint "mqus01icinga2wk01" {
        host = "mqus01icinga2wk01"
}

object Zone "MQUS01" {
        endpoints = [ "mqus01icinga2wk01" ]
        parent = "master"
}


//MQUS02
object Endpoint "mqus02icinga2wk01" {
        host = "mqus02icinga2wk01"
}

object Zone "MQUS02" {
        endpoints = [ "mqus02icinga2wk01" ]
        parent = "master"
}

//MQFR01
object Endpoint "mqfr01icinga2wk01" {
        host = "mqfr01icinga2wk01"
}

object Zone "MQFR01" {
        endpoints = [ "mqfr01icinga2wk01" ]
        parent = "master"
}


//MQIN01
object Endpoint "mqin01icinga2wk01" {
        host = "mqin01icinga2wk01"
}

object Zone "MQIN01" {
        endpoints = [ "mqin01icinga2wk01" ]
        parent = "master"
}

//MQCN02
object Endpoint "mqcn02icinga2wk01" {
        host = "mqcn02icinga2wk01"
}

object Zone "MQCN02" {
        endpoints = [ "mqcn02icinga2wk01" ]
        parent = "master"
}

//MQTN01
object Endpoint "mqtn01icinga2wk01" {
        host = "mqtn01icinga2wk01"
}

object Zone "MQTN01" {
        endpoints = [ "mqtn01icinga2wk01" ]
        parent = "master"
}


//MQDE22
object Endpoint "mqde22icinga2wk01" {
        host = "mqde22icinga2wk01"
}

object Zone "MQDE22" {
        endpoints = [ "mqde22icinga2wk01" ]
        parent = "master"
}

And this the curl command output:

 {"results":[{"attrs":{"__name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance"
"acknowledgement":0.0
"acknowledgement_expiry":0.0
"action_url":""
"active":true
"check_attempt":1.0
"check_command":"nrpe"
"check_interval":86400.0
"check_period":""
"check_timeout":180.0
"command_endpoint":""
"display_name":"APP_SP_Nintex_Workflow_Maintenance"
"downtime_depth":0.0
"enable_active_checks":true
"enable_event_handler":true
"enable_flapping":true
"enable_notifications":true
"enable_passive_checks":true
"enable_perfdata":false
"event_command":""
"flapping":false
"flapping_current":16.199999999999999289
"flapping_last_change":0.0
"flapping_threshold":0.0
"flapping_threshold_high":30.0
"flapping_threshold_low":25.0
"force_next_check":false
"force_next_notification":false
"groups":["APP_Services"]
"ha_mode":0.0
"host_name":"mqde01pspweb01"
"icon_image":""
"icon_image_alt":""
"last_check":1562151453.678041935
"last_check_result":{"active":true
"check_source":"mqde01icinga2wk02"
"command":["/usr/lib/nagios/plugins/check_nrpe"
"-H"
"10.10.1.48"
"-c"
"check_nintex_workflow_maintenance"
"-p"
"5666"
"-t"
"180"
"-u"
"-a"
"C:\\FuM\\Script\\NintexMaintenance"
"Nintex_workflow_maintenance.log"
"Nintex_Workflow_Maintenance"]
"execution_end":1562151453.6779398918
"execution_start":1562151449.1948740482
"exit_status":0.0
"output":"Nintex Workflow maintenance: Succesfully processed URLs: https://ewp/rooms/pep_1422 https://ewp/rooms/pep_1423 https://ewp/rooms/pep_1424 https://ewp/rooms/pep_1438 https://ewp/rooms/pep_1458 https://ewp/rooms/pep_1459 https://ewp/rooms/pep_1476 https://ewp/rooms/pep_1488 https://ewp/rooms/pep_1509 https://ewp/rooms/pep_1510 https://ewp/rooms/pep_1514\n\nNintex Workflow maintenance: Succesfully processed URLs: -/rooms/pep_1422 -/rooms/pep_1423 -/rooms/pep_1424 -/rooms/pep_1438 -/rooms/pep_1458 -/rooms/pep_1459 -/rooms/pep_1476 -/rooms/pep_1488 -/rooms/pep_1509 -/rooms/pep_1510 -/rooms/pep_1514\n\nNintex Workflow maintenance: Succesfully processed URLs: https://ewp/rooms/pep_1422 https://ewp/rooms/pep_1423 https://ewp/rooms/pep_1424 https://ewp/rooms/pep_1438 https://ewp/rooms/pep_1458 https://ewp.marquardt.d"
"performance_data":[]
"schedule_end":1562151453.678041935
"schedule_start":1562150830.0
"state":0.0
"ttl":0.0
"type":"CheckResult"
"vars_after":{"attempt":1.0
"reachable":true
"state":0.0
"state_type":1.0}
"vars_before":{"attempt":1.0
"reachable":true
"state":0.0
"state_type":1.0}}
"last_hard_state":0.0
"last_hard_state_change":1562053063.3850541115
"last_reachable":true
"last_state":0.0
"last_state_change":1562053063.3850541115
"last_state_critical":1561968516.995112896
"last_state_ok":1562151453.6789219379
"last_state_type":1.0
"last_state_unknown":0.0
"last_state_unreachable":0.0
"last_state_warning":0.0
"max_check_attempts":3.0
"name":"APP_SP_Nintex_Workflow_Maintenance"
"next_check":1562237214.8089292049
"notes":""
"notes_url":""
"original_attributes":null
"package":"director"
"paused":true
"retry_interval":300.0
"severity":8.0
"source_location":{"first_column":1.0
"first_line":7568.0
"last_column":50.0
"last_line":7568.0
"path":"/var/lib/icinga2/api/packages/director/c8b51e14-8180-4c03-a6c6-aa9fb1942c69/zones.d/director-global/service_apply.conf"}
"state":0.0
"state_type":1.0
"templates":["APP_SP_Nintex_Workflow_Maintenance"
"APP_SP_Nintex_Workflow_Maintenance"
"NRPE"
"generic-service-24h-24x7"
"generic-service-24x7"
"OTRS"
"host var overrides (Director)"]
"type":"Service"
"vars":{"nrpe_address":"$host.address$"
"nrpe_arguments":["C:\\FuM\\Script\\NintexMaintenance"
"Nintex_workflow_maintenance.log"
"Nintex_Workflow_Maintenance"]
"nrpe_command":"check_nintex_workflow_maintenance"
"nrpe_no_ssl":"$host.vars.nrpe_no_ssl_host$"
"nrpe_port":"$host.vars.nrpe_port$"
"nrpe_timeout":"180"
"nrpe_timeout_unknown":true
"otrs_changetype":"CmCTStandard"
"otrs_priority":"3 normal - prio 3"
"otrs_queue":"2_PS::Workplace::SharePoint::Sharepoint Application"
"otrs_risk":"CmRLow"
"otrs_service":"Sharepoint Application"
"otrs_type":"Incident"}
"version":0.0
"volatile":false
"zone":"MQDE01"}
"joins":{}
"meta":{}
"name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance"
"type":"Service"}]}

So, the service belongs into this zone …

… whereas two endpoints are configured.

Now inspect both satellite endpoints on the logs and the REST API calls to find out which one is responsible for executing the checks. Hint: paused=false is the active endpoint for the service object.

I identified the satellite that is execuing the check: mqde01icinga2wk02
With debuglog feature enabled I clicked on Reschedule for the service but when I grep -i nintex debug.log it doesn’t find anything. What should I look for?

  • Does the satellite trust the master, e.g. is the zones.conf the same and includes all parent zones and endpoints
  • Are there cluster messages with the event::SetForceNextCheck relayed from the master to the satellites, and them receiving them.
  • If nothing is relayed, is your Icinga master operational, i.e. its monitoring health, self-checks, etc. Any insights on a possible deadlock.

Satellite trust the master (zones.conf of mqde01icinga2wk02):

/*
 * Generated by Icinga 2 node setup commands
 * on 2016-12-08 14:06:20 +0100
 */

object Endpoint "mqde01icinga2cl01" {
//  host = "10.10.2.63"
}

object Endpoint "mqde01icinga2cl02" {
//  host = "10.10.2.64"
}

object Zone "master" {
  endpoints = [ "mqde01icinga2cl01", "mqde01icinga2cl02" ]
}


// GLOBAL CONFIG
object Zone "global" {
  global = true
}

object Zone "director-global" {
  global = true
}

//MQDE01
object Endpoint "mqde01icinga2wk01" {
        host = "mqde01icinga2wk01"
}

object Endpoint "mqde01icinga2wk02" {
        host = "mqde01icinga2wk02"
}

object Zone "MQDE01" {
        endpoints = [ "mqde01icinga2wk01","mqde01icinga2wk02" ]
        parent = "master"
}

I can see them in both master and satellite debug logs:

root@mqde01icinga2cl01:/var/log/icinga2$ grep SetForceNextCheck debug.log
[2019-07-03 11:39:10 +0200] notice/JsonRpcConnection: Received 'event::SetForceNextCheck' message from 'mqde01icinga2wk01'
[2019-07-03 11:39:10 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 11:39:10 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2cl02'
[2019-07-03 11:39:17 +0200] notice/JsonRpcConnection: Received 'event::SetForceNextCheck' message from 'mqde01icinga2wk01'
[2019-07-03 11:39:17 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 11:39:17 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2cl02'
[2019-07-03 11:42:58 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 11:42:58 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2wk01'
[2019-07-03 11:42:58 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2cl02'
[2019-07-03 11:46:52 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 11:46:52 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2wk01'
[2019-07-03 11:46:52 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2cl02'

root@mqde01icinga2wk02:/var/log/icinga2$ grep SetForceNextCheck debug.log
[2019-07-03 11:39:10 +0200] notice/JsonRpcConnection: Received 'event::SetForceNextCheck' message from 'mqde01icinga2wk01'
[2019-07-03 11:39:10 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 11:39:17 +0200] notice/JsonRpcConnection: Received 'event::SetForceNextCheck' message from 'mqde01icinga2wk01'
[2019-07-03 11:39:17 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 11:42:58 +0200] notice/JsonRpcConnection: Received 'event::SetForceNextCheck' message from 'mqde01icinga2wk01'
[2019-07-03 11:42:58 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 11:46:52 +0200] notice/JsonRpcConnection: Received 'event::SetForceNextCheck' message from 'mqde01icinga2wk01'
[2019-07-03 11:46:52 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 14:00:56 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 14:00:56 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2wk01'
[2019-07-03 14:00:57 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 14:00:57 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2wk01'
[2019-07-03 14:01:13 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 14:01:13 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2wk01'
[2019-07-03 14:01:17 +0200] notice/JsonRpcConnection: Received 'event::SetForceNextCheck' message from 'mqde01icinga2cl01'
[2019-07-03 14:01:17 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2019-07-03 14:01:17 +0200] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'mqde01icinga2wk01'

The checker component logs about specific forced schedules, looks for lines starting with Scheduling info for checkable. If the checkables are missing from that, also verify that NTP is running and all instances and their time is in sync (ntp checks as listed in the “monitoring Icinga 2” chapter in the docs.

I checked the time and is synced on all four nodes:

Then I enabled debulog on all four, clicked on Reschedule button in IcingaWeb2 and looked for the service name, nothing on the sattelites:

If I look for Scheduling info for checkable there are several entries but none of this service.

Can you repeat the curl call on all endpoints and check whether the configuration objects really exist on the satellites?

root@mqde01icinga2cl01:~$ curl -k -s -u root:icinga 'https://localhost:5665/v1/objects/services/mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance'
{"results":[{"attrs":{"__name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","acknowledgement":0.0,"acknowledgement_expiry":0.0,"action_url":"","active":true,"check_attempt":1.0,"check_command":"nrpe","check_interval":86400.0,"check_period":"","check_timeout":180.0,"command_endpoint":"","display_name":"APP_SP_Nintex_Workflow_Maintenance","downtime_depth":0.0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":true,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":false,"event_command":"","flapping":false,"flapping_current":25.600000000000004974,"flapping_last_change":0.0,"flapping_threshold":0.0,"flapping_threshold_high":30.0,"flapping_threshold_low":25.0,"force_next_check":false,"force_next_notification":false,"groups":["APP_Services"],"ha_mode":0.0,"host_name":"mqde01pspweb01","icon_image":"","icon_image_alt":"","last_check":1562568088.369189024,"last_check_result":{"active":true,"check_source":"mqde01icinga2wk02","command":["/usr/lib/nagios/plugins/check_nrpe","-H","10.10.1.48","-c","check_nintex_workflow_maintenance","-p","5666","-t","180","-u","-a","C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"execution_end":1562568088.3691399097,"execution_start":1562568084.9632709026,"exit_status":0.0,"output":"Nintex Workflow maintenance: Successfully processed URLs (-): -/rooms/pep_1910 -/rooms/pep_1928 -/rooms/pep_1944 -/rooms/pep_1948 -/rooms/pep_1961 -/rooms/pep_1962 -/rooms/pep_1964 -/rooms/pep_1976 -/rooms/pep_1983 -/rooms/pep_1984 -/rooms/pep_1985","performance_data":[],"schedule_end":1562568088.369189024,"schedule_start":1562567453.3799231052,"state":0.0,"ttl":0.0,"type":"CheckResult","vars_after":{"attempt":1.0,"reachable":true,"state":0.0,"state_type":1.0},"vars_before":{"attempt":1.0,"reachable":true,"state":2.0,"state_type":1.0}},"last_hard_state":0.0,"last_hard_state_change":1562568088.370839119,"last_reachable":true,"last_state":2.0,"last_state_change":1562568088.370839119,"last_state_critical":1562484295.3806810379,"last_state_ok":1562568088.3708479404,"last_state_type":1.0,"last_state_unknown":0.0,"last_state_unreachable":0.0,"last_state_warning":0.0,"max_check_attempts":3.0,"name":"APP_SP_Nintex_Workflow_Maintenance","next_check":1562653167.3608601093,"notes":"","notes_url":"","original_attributes":null,"package":"director","paused":true,"retry_interval":300.0,"severity":8.0,"source_location":{"first_column":1.0,"first_line":7569.0,"last_column":50.0,"last_line":7569.0,"path":"/var/lib/icinga2/api/packages/director/d3d4e584-6842-4dc7-8236-86fea6fc7859/zones.d/director-global/service_apply.conf"},"state":0.0,"state_type":1.0,"templates":["APP_SP_Nintex_Workflow_Maintenance","APP_SP_Nintex_Workflow_Maintenance","NRPE","generic-service-24h-24x7","generic-service-24x7","OTRS","host var overrides (Director)"],"type":"Service","vars":{"nrpe_address":"$host.address$","nrpe_arguments":["C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"nrpe_command":"check_nintex_workflow_maintenance","nrpe_no_ssl":"$host.vars.nrpe_no_ssl_host$","nrpe_port":"$host.vars.nrpe_port$","nrpe_timeout":"180","nrpe_timeout_unknown":true,"otrs_changetype":"CmCTStandard","otrs_priority":"3 normal - prio 3","otrs_queue":"2_PS::Workplace::SharePoint::Sharepoint Application","otrs_risk":"CmRLow","otrs_service":"Sharepoint Application","otrs_type":"Incident"},"version":0.0,"volatile":false,"zone":"MQDE01"},"joins":{},"meta":{},"name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","type":"Service"}]}
root@mqde01icinga2cl02:~$ curl -k -s -u root:icinga 'https://localhost:5665/v1/objects/services/mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance'
{"results":[{"attrs":{"__name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","acknowledgement":0.0,"acknowledgement_expiry":0.0,"action_url":"","active":true,"check_attempt":1.0,"check_command":"nrpe","check_interval":86400.0,"check_period":"","check_timeout":180.0,"command_endpoint":"","display_name":"APP_SP_Nintex_Workflow_Maintenance","downtime_depth":0.0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":true,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":false,"event_command":"","flapping":false,"flapping_current":25.600000000000004974,"flapping_last_change":0.0,"flapping_threshold":0.0,"flapping_threshold_high":30.0,"flapping_threshold_low":25.0,"force_next_check":false,"force_next_notification":false,"groups":["APP_Services"],"ha_mode":0.0,"host_name":"mqde01pspweb01","icon_image":"","icon_image_alt":"","last_check":1562568088.369189024,"last_check_result":{"active":true,"check_source":"mqde01icinga2wk02","command":["/usr/lib/nagios/plugins/check_nrpe","-H","10.10.1.48","-c","check_nintex_workflow_maintenance","-p","5666","-t","180","-u","-a","C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"execution_end":1562568088.3691399097,"execution_start":1562568084.9632709026,"exit_status":0.0,"output":"Nintex Workflow maintenance: Successfully processed URLs (-): -/rooms/pep_1910 -/rooms/pep_1928 -/rooms/pep_1944 -/rooms/pep_1948 -/rooms/pep_1961 -/rooms/pep_1962 -/rooms/pep_1964 -/rooms/pep_1976 -/rooms/pep_1983 -/rooms/pep_1984 -/rooms/pep_1985","performance_data":[],"schedule_end":1562568088.369189024,"schedule_start":1562567453.3799231052,"state":0.0,"ttl":0.0,"type":"CheckResult","vars_after":{"attempt":1.0,"reachable":true,"state":0.0,"state_type":1.0},"vars_before":{"attempt":1.0,"reachable":true,"state":2.0,"state_type":1.0}},"last_hard_state":0.0,"last_hard_state_change":1562568088.3766019344,"last_reachable":true,"last_state":2.0,"last_state_change":1562568088.3766019344,"last_state_critical":1562484295.3819539547,"last_state_ok":1562568088.376609087,"last_state_type":1.0,"last_state_unknown":0.0,"last_state_unreachable":0.0,"last_state_warning":0.0,"max_check_attempts":3.0,"name":"APP_SP_Nintex_Workflow_Maintenance","next_check":1562650524.316617012,"notes":"","notes_url":"","original_attributes":null,"package":"_cluster","paused":false,"retry_interval":300.0,"severity":8.0,"source_location":{"first_column":1.0,"first_line":7569.0,"last_column":50.0,"last_line":7569.0,"path":"/var/lib/icinga2/api/zones/director-global/director/service_apply.conf"},"state":0.0,"state_type":1.0,"templates":["APP_SP_Nintex_Workflow_Maintenance","APP_SP_Nintex_Workflow_Maintenance","NRPE","generic-service-24h-24x7","generic-service-24x7","OTRS","host var overrides (Director)"],"type":"Service","vars":{"nrpe_address":"$host.address$","nrpe_arguments":["C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"nrpe_command":"check_nintex_workflow_maintenance","nrpe_no_ssl":"$host.vars.nrpe_no_ssl_host$","nrpe_port":"$host.vars.nrpe_port$","nrpe_timeout":"180","nrpe_timeout_unknown":true,"otrs_changetype":"CmCTStandard","otrs_priority":"3 normal - prio 3","otrs_queue":"2_PS::Workplace::SharePoint::Sharepoint Application","otrs_risk":"CmRLow","otrs_service":"Sharepoint Application","otrs_type":"Incident"},"version":0.0,"volatile":false,"zone":"MQDE01"},"joins":{},"meta":{},"name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","type":"Service"}]}
root@mqde01icinga2wk01:~$ curl -k -s -u root:icinga  'https://localhost:5665/v1/objects/services/mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance'
{"results":[{"attrs":{"__name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","acknowledgement":0.0,"acknowledgement_expiry":0.0,"action_url":"","active":true,"check_attempt":1.0,"check_command":"nrpe","check_interval":86400.0,"check_period":"","check_timeout":180.0,"command_endpoint":"","display_name":"APP_SP_Nintex_Workflow_Maintenance","downtime_depth":0.0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":true,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":false,"event_command":"","flapping":false,"flapping_current":25.600000000000004974,"flapping_last_change":0.0,"flapping_threshold":0.0,"flapping_threshold_high":30.0,"flapping_threshold_low":25.0,"force_next_check":false,"force_next_notification":false,"groups":["APP_Services"],"ha_mode":0.0,"host_name":"mqde01pspweb01","icon_image":"","icon_image_alt":"","last_check":1562568088.369189024,"last_check_result":{"active":true,"check_source":"mqde01icinga2wk02","command":["/usr/lib/nagios/plugins/check_nrpe","-H","10.10.1.48","-c","check_nintex_workflow_maintenance","-p","5666","-t","180","-u","-a","C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"execution_end":1562568088.3691399097,"execution_start":1562568084.9632709026,"exit_status":0.0,"output":"Nintex Workflow maintenance: Successfully processed URLs (-): -/rooms/pep_1910 -/rooms/pep_1928 -/rooms/pep_1944 -/rooms/pep_1948 -/rooms/pep_1961 -/rooms/pep_1962 -/rooms/pep_1964 -/rooms/pep_1976 -/rooms/pep_1983 -/rooms/pep_1984 -/rooms/pep_1985","performance_data":[],"schedule_end":1562568088.369189024,"schedule_start":1562567453.3799231052,"state":0.0,"ttl":0.0,"type":"CheckResult","vars_after":{"attempt":1.0,"reachable":true,"state":0.0,"state_type":1.0},"vars_before":{"attempt":1.0,"reachable":true,"state":2.0,"state_type":1.0}},"last_hard_state":0.0,"last_hard_state_change":1562568088.3709080219,"last_reachable":true,"last_state":2.0,"last_state_change":1562568088.3709080219,"last_state_critical":1562484295.3801739216,"last_state_ok":1562568088.3709170818,"last_state_type":1.0,"last_state_unknown":0.0,"last_state_unreachable":0.0,"last_state_warning":0.0,"max_check_attempts":3.0,"name":"APP_SP_Nintex_Workflow_Maintenance","next_check":1562653610.4009549618,"notes":"","notes_url":"","original_attributes":null,"package":"_cluster","paused":true,"retry_interval":300.0,"severity":8.0,"source_location":{"first_column":1.0,"first_line":7569.0,"last_column":50.0,"last_line":7569.0,"path":"/var/lib/icinga2/api/zones/director-global/director/service_apply.conf"},"state":0.0,"state_type":1.0,"templates":["APP_SP_Nintex_Workflow_Maintenance","APP_SP_Nintex_Workflow_Maintenance","NRPE","generic-service-24h-24x7","generic-service-24x7","OTRS","host var overrides (Director)"],"type":"Service","vars":{"nrpe_address":"$host.address$","nrpe_arguments":["C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"nrpe_command":"check_nintex_workflow_maintenance","nrpe_no_ssl":"$host.vars.nrpe_no_ssl_host$","nrpe_port":"$host.vars.nrpe_port$","nrpe_timeout":"180","nrpe_timeout_unknown":true,"otrs_changetype":"CmCTStandard","otrs_priority":"3 normal - prio 3","otrs_queue":"2_PS::Workplace::SharePoint::Sharepoint Application","otrs_risk":"CmRLow","otrs_service":"Sharepoint Application","otrs_type":"Incident"},"version":0.0,"volatile":false,"zone":"MQDE01"},"joins":{},"meta":{},"name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","type":"Service"}]}
root@mqde01icinga2wk02:~$ curl -k -s -u root:icinga  'https://localhost:5665/v1/objects/services/mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance'
{"results":[{"attrs":{"__name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","acknowledgement":0.0,"acknowledgement_expiry":0.0,"action_url":"","active":true,"check_attempt":1.0,"check_command":"nrpe","check_interval":86400.0,"check_period":"","check_timeout":180.0,"command_endpoint":"","display_name":"APP_SP_Nintex_Workflow_Maintenance","downtime_depth":0.0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":true,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":false,"event_command":"","flapping":false,"flapping_current":25.600000000000004974,"flapping_last_change":0.0,"flapping_threshold":0.0,"flapping_threshold_high":30.0,"flapping_threshold_low":25.0,"force_next_check":false,"force_next_notification":false,"groups":["APP_Services"],"ha_mode":0.0,"host_name":"mqde01pspweb01","icon_image":"","icon_image_alt":"","last_check":1562568088.369189024,"last_check_result":{"active":true,"check_source":"mqde01icinga2wk02","command":["/usr/lib/nagios/plugins/check_nrpe","-H","10.10.1.48","-c","check_nintex_workflow_maintenance","-p","5666","-t","180","-u","-a","C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"execution_end":1562568088.3691399097,"execution_start":1562568084.9632709026,"exit_status":0.0,"output":"Nintex Workflow maintenance: Successfully processed URLs (-): -/rooms/pep_1910 -/rooms/pep_1928 -/rooms/pep_1944 -/rooms/pep_1948 -/rooms/pep_1961 -/rooms/pep_1962 -/rooms/pep_1964 -/rooms/pep_1976 -/rooms/pep_1983 -/rooms/pep_1984 -/rooms/pep_1985","performance_data":[],"schedule_end":1562568088.369189024,"schedule_start":1562567453.3799231052,"state":0.0,"ttl":0.0,"type":"CheckResult","vars_after":{"attempt":1.0,"reachable":true,"state":0.0,"state_type":1.0},"vars_before":{"attempt":1.0,"reachable":true,"state":2.0,"state_type":1.0}},"last_hard_state":0.0,"last_hard_state_change":1562568088.369189024,"last_reachable":true,"last_state":2.0,"last_state_change":1562568088.369189024,"last_state_critical":1562484295.3799109459,"last_state_ok":1562568088.3691968918,"last_state_type":1.0,"last_state_unknown":0.0,"last_state_unreachable":0.0,"last_state_warning":0.0,"max_check_attempts":3.0,"name":"APP_SP_Nintex_Workflow_Maintenance","next_check":1562651246.3692040443,"notes":"","notes_url":"","original_attributes":null,"package":"_cluster","paused":false,"retry_interval":300.0,"severity":8.0,"source_location":{"first_column":1.0,"first_line":7569.0,"last_column":50.0,"last_line":7569.0,"path":"/var/lib/icinga2/api/zones/director-global/director/service_apply.conf"},"state":0.0,"state_type":1.0,"templates":["APP_SP_Nintex_Workflow_Maintenance","APP_SP_Nintex_Workflow_Maintenance","NRPE","generic-service-24h-24x7","generic-service-24x7","OTRS","host var overrides (Director)"],"type":"Service","vars":{"nrpe_address":"$host.address$","nrpe_arguments":["C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"nrpe_command":"check_nintex_workflow_maintenance","nrpe_no_ssl":"$host.vars.nrpe_no_ssl_host$","nrpe_port":"$host.vars.nrpe_port$","nrpe_timeout":"180","nrpe_timeout_unknown":true,"otrs_changetype":"CmCTStandard","otrs_priority":"3 normal - prio 3","otrs_queue":"2_PS::Workplace::SharePoint::Sharepoint Application","otrs_risk":"CmRLow","otrs_service":"Sharepoint Application","otrs_type":"Incident"},"version":0.0,"volatile":false,"zone":"MQDE01"},"joins":{},"meta":{},"name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","type":"Service"}]}

Looks good. The only thing which irritates me is that forced_next_check is set to false. When rescheduling a check, are you forcing it? If not, the scheduler will just re-insert it into the queue according to the best possible check interval, not immediately execute it.

This is the output after Rescheduling the check with Force option, now force_next_check is true, but still, nothing changed in IcingaWeb2:

image

root@mqde01icinga2cl01:~$ curl -k -s -u root:icinga 'https://localhost:5665/v1/objects/services/mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance'
{"results":[{"attrs":{"__name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","acknowledgement":0.0,"acknowledgement_expiry":0.0,"action_url":"","active":true,"check_attempt":1.0,"check_command":"nrpe","check_interval":86400.0,"check_period":"","check_timeout":180.0,"command_endpoint":"","display_name":"APP_SP_Nintex_Workflow_Maintenance","downtime_depth":0.0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":true,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":false,"event_command":"","flapping":false,"flapping_current":25.600000000000004974,"flapping_last_change":0.0,"flapping_threshold":0.0,"flapping_threshold_high":30.0,"flapping_threshold_low":25.0,"force_next_check":true,"force_next_notification":false,"groups":["APP_Services"],"ha_mode":0.0,"host_name":"mqde01pspweb01","icon_image":"","icon_image_alt":"","last_check":1562568088.369189024,"last_check_result":{"active":true,"check_source":"mqde01icinga2wk02","command":["/usr/lib/nagios/plugins/check_nrpe","-H","10.10.1.48","-c","check_nintex_workflow_maintenance","-p","5666","-t","180","-u","-a","C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"execution_end":1562568088.3691399097,"execution_start":1562568084.9632709026,"exit_status":0.0,"output":"Nintex Workflow maintenance: Successfully processed URLs (-): -/rooms/pep_1910 -/rooms/pep_1928 -/rooms/pep_1944 -/rooms/pep_1948 -/rooms/pep_1961 -/rooms/pep_1962 -/rooms/pep_1964 -/rooms/pep_1976 -/rooms/pep_1983 -/rooms/pep_1984 -/rooms/pep_1985","performance_data":[],"schedule_end":1562568088.369189024,"schedule_start":1562567453.3799231052,"state":0.0,"ttl":0.0,"type":"CheckResult","vars_after":{"attempt":1.0,"reachable":true,"state":0.0,"state_type":1.0},"vars_before":{"attempt":1.0,"reachable":true,"state":2.0,"state_type":1.0}},"last_hard_state":0.0,"last_hard_state_change":1562568088.370839119,"last_reachable":true,"last_state":2.0,"last_state_change":1562568088.370839119,"last_state_critical":1562484295.3806810379,"last_state_ok":1562568088.3708479404,"last_state_type":1.0,"last_state_unknown":0.0,"last_state_unreachable":0.0,"last_state_warning":0.0,"max_check_attempts":3.0,"name":"APP_SP_Nintex_Workflow_Maintenance","next_check":1562581617.0,"notes":"","notes_url":"","original_attributes":null,"package":"director","paused":true,"retry_interval":300.0,"severity":8.0,"source_location":{"first_column":1.0,"first_line":7569.0,"last_column":50.0,"last_line":7569.0,"path":"/var/lib/icinga2/api/packages/director/d3d4e584-6842-4dc7-8236-86fea6fc7859/zones.d/director-global/service_apply.conf"},"state":0.0,"state_type":1.0,"templates":["APP_SP_Nintex_Workflow_Maintenance","APP_SP_Nintex_Workflow_Maintenance","NRPE","generic-service-24h-24x7","generic-service-24x7","OTRS","host var overrides (Director)"],"type":"Service","vars":{"nrpe_address":"$host.address$","nrpe_arguments":["C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"nrpe_command":"check_nintex_workflow_maintenance","nrpe_no_ssl":"$host.vars.nrpe_no_ssl_host$","nrpe_port":"$host.vars.nrpe_port$","nrpe_timeout":"180","nrpe_timeout_unknown":true,"otrs_changetype":"CmCTStandard","otrs_priority":"3 normal - prio 3","otrs_queue":"2_PS::Workplace::SharePoint::Sharepoint Application","otrs_risk":"CmRLow","otrs_service":"Sharepoint Application","otrs_type":"Incident"},"version":0.0,"volatile":false,"zone":"MQDE01"},"joins":{},"meta":{},"name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","type":"Service"}]}
root@mqde01icinga2cl02:~$ curl -k -s -u root:icinga 'https://localhost:5665/v1/objects/services/mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance'
{"results":[{"attrs":{"__name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","acknowledgement":0.0,"acknowledgement_expiry":0.0,"action_url":"","active":true,"check_attempt":1.0,"check_command":"nrpe","check_interval":86400.0,"check_period":"","check_timeout":180.0,"command_endpoint":"","display_name":"APP_SP_Nintex_Workflow_Maintenance","downtime_depth":0.0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":true,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":false,"event_command":"","flapping":false,"flapping_current":25.600000000000004974,"flapping_last_change":0.0,"flapping_threshold":0.0,"flapping_threshold_high":30.0,"flapping_threshold_low":25.0,"force_next_check":true,"force_next_notification":false,"groups":["APP_Services"],"ha_mode":0.0,"host_name":"mqde01pspweb01","icon_image":"","icon_image_alt":"","last_check":1562568088.369189024,"last_check_result":{"active":true,"check_source":"mqde01icinga2wk02","command":["/usr/lib/nagios/plugins/check_nrpe","-H","10.10.1.48","-c","check_nintex_workflow_maintenance","-p","5666","-t","180","-u","-a","C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"execution_end":1562568088.3691399097,"execution_start":1562568084.9632709026,"exit_status":0.0,"output":"Nintex Workflow maintenance: Successfully processed URLs (-): -/rooms/pep_1910 -/rooms/pep_1928 -/rooms/pep_1944 -/rooms/pep_1948 -/rooms/pep_1961 -/rooms/pep_1962 -/rooms/pep_1964 -/rooms/pep_1976 -/rooms/pep_1983 -/rooms/pep_1984 -/rooms/pep_1985","performance_data":[],"schedule_end":1562568088.369189024,"schedule_start":1562567453.3799231052,"state":0.0,"ttl":0.0,"type":"CheckResult","vars_after":{"attempt":1.0,"reachable":true,"state":0.0,"state_type":1.0},"vars_before":{"attempt":1.0,"reachable":true,"state":2.0,"state_type":1.0}},"last_hard_state":0.0,"last_hard_state_change":1562568088.3766019344,"last_reachable":true,"last_state":2.0,"last_state_change":1562568088.3766019344,"last_state_critical":1562484295.3819539547,"last_state_ok":1562568088.376609087,"last_state_type":1.0,"last_state_unknown":0.0,"last_state_unreachable":0.0,"last_state_warning":0.0,"max_check_attempts":3.0,"name":"APP_SP_Nintex_Workflow_Maintenance","next_check":1562581617.0,"notes":"","notes_url":"","original_attributes":null,"package":"_cluster","paused":false,"retry_interval":300.0,"severity":8.0,"source_location":{"first_column":1.0,"first_line":7569.0,"last_column":50.0,"last_line":7569.0,"path":"/var/lib/icinga2/api/zones/director-global/director/service_apply.conf"},"state":0.0,"state_type":1.0,"templates":["APP_SP_Nintex_Workflow_Maintenance","APP_SP_Nintex_Workflow_Maintenance","NRPE","generic-service-24h-24x7","generic-service-24x7","OTRS","host var overrides (Director)"],"type":"Service","vars":{"nrpe_address":"$host.address$","nrpe_arguments":["C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"nrpe_command":"check_nintex_workflow_maintenance","nrpe_no_ssl":"$host.vars.nrpe_no_ssl_host$","nrpe_port":"$host.vars.nrpe_port$","nrpe_timeout":"180","nrpe_timeout_unknown":true,"otrs_changetype":"CmCTStandard","otrs_priority":"3 normal - prio 3","otrs_queue":"2_PS::Workplace::SharePoint::Sharepoint Application","otrs_risk":"CmRLow","otrs_service":"Sharepoint Application","otrs_type":"Incident"},"version":0.0,"volatile":false,"zone":"MQDE01"},"joins":{},"meta":{},"name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","type":"Service"}]}
root@mqde01icinga2wk01:~$ curl -k -s -u root:icinga  'https://localhost:5665/v1/objects/services/mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance'
{"results":[{"attrs":{"__name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","acknowledgement":0.0,"acknowledgement_expiry":0.0,"action_url":"","active":true,"check_attempt":1.0,"check_command":"nrpe","check_interval":86400.0,"check_period":"","check_timeout":180.0,"command_endpoint":"","display_name":"APP_SP_Nintex_Workflow_Maintenance","downtime_depth":0.0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":true,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":false,"event_command":"","flapping":false,"flapping_current":25.600000000000004974,"flapping_last_change":0.0,"flapping_threshold":0.0,"flapping_threshold_high":30.0,"flapping_threshold_low":25.0,"force_next_check":true,"force_next_notification":false,"groups":["APP_Services"],"ha_mode":0.0,"host_name":"mqde01pspweb01","icon_image":"","icon_image_alt":"","last_check":1562568088.369189024,"last_check_result":{"active":true,"check_source":"mqde01icinga2wk02","command":["/usr/lib/nagios/plugins/check_nrpe","-H","10.10.1.48","-c","check_nintex_workflow_maintenance","-p","5666","-t","180","-u","-a","C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"execution_end":1562568088.3691399097,"execution_start":1562568084.9632709026,"exit_status":0.0,"output":"Nintex Workflow maintenance: Successfully processed URLs (-): -/rooms/pep_1910 -/rooms/pep_1928 -/rooms/pep_1944 -/rooms/pep_1948 -/rooms/pep_1961 -/rooms/pep_1962 -/rooms/pep_1964 -/rooms/pep_1976 -/rooms/pep_1983 -/rooms/pep_1984 -/rooms/pep_1985","performance_data":[],"schedule_end":1562568088.369189024,"schedule_start":1562567453.3799231052,"state":0.0,"ttl":0.0,"type":"CheckResult","vars_after":{"attempt":1.0,"reachable":true,"state":0.0,"state_type":1.0},"vars_before":{"attempt":1.0,"reachable":true,"state":2.0,"state_type":1.0}},"last_hard_state":0.0,"last_hard_state_change":1562568088.3709080219,"last_reachable":true,"last_state":2.0,"last_state_change":1562568088.3709080219,"last_state_critical":1562484295.3801739216,"last_state_ok":1562568088.3709170818,"last_state_type":1.0,"last_state_unknown":0.0,"last_state_unreachable":0.0,"last_state_warning":0.0,"max_check_attempts":3.0,"name":"APP_SP_Nintex_Workflow_Maintenance","next_check":1562581617.0,"notes":"","notes_url":"","original_attributes":null,"package":"_cluster","paused":true,"retry_interval":300.0,"severity":8.0,"source_location":{"first_column":1.0,"first_line":7569.0,"last_column":50.0,"last_line":7569.0,"path":"/var/lib/icinga2/api/zones/director-global/director/service_apply.conf"},"state":0.0,"state_type":1.0,"templates":["APP_SP_Nintex_Workflow_Maintenance","APP_SP_Nintex_Workflow_Maintenance","NRPE","generic-service-24h-24x7","generic-service-24x7","OTRS","host var overrides (Director)"],"type":"Service","vars":{"nrpe_address":"$host.address$","nrpe_arguments":["C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"nrpe_command":"check_nintex_workflow_maintenance","nrpe_no_ssl":"$host.vars.nrpe_no_ssl_host$","nrpe_port":"$host.vars.nrpe_port$","nrpe_timeout":"180","nrpe_timeout_unknown":true,"otrs_changetype":"CmCTStandard","otrs_priority":"3 normal - prio 3","otrs_queue":"2_PS::Workplace::SharePoint::Sharepoint Application","otrs_risk":"CmRLow","otrs_service":"Sharepoint Application","otrs_type":"Incident"},"version":0.0,"volatile":false,"zone":"MQDE01"},"joins":{},"meta":{},"name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","type":"Service"}]}
root@mqde01icinga2wk02:~$ curl -k -s -u root:icinga  'https://localhost:5665/v1/objects/services/mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance'
{"results":[{"attrs":{"__name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","acknowledgement":0.0,"acknowledgement_expiry":0.0,"action_url":"","active":true,"check_attempt":1.0,"check_command":"nrpe","check_interval":86400.0,"check_period":"","check_timeout":180.0,"command_endpoint":"","display_name":"APP_SP_Nintex_Workflow_Maintenance","downtime_depth":0.0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":true,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":false,"event_command":"","flapping":false,"flapping_current":25.600000000000004974,"flapping_last_change":0.0,"flapping_threshold":0.0,"flapping_threshold_high":30.0,"flapping_threshold_low":25.0,"force_next_check":true,"force_next_notification":false,"groups":["APP_Services"],"ha_mode":0.0,"host_name":"mqde01pspweb01","icon_image":"","icon_image_alt":"","last_check":1562568088.369189024,"last_check_result":{"active":true,"check_source":"mqde01icinga2wk02","command":["/usr/lib/nagios/plugins/check_nrpe","-H","10.10.1.48","-c","check_nintex_workflow_maintenance","-p","5666","-t","180","-u","-a","C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"execution_end":1562568088.3691399097,"execution_start":1562568084.9632709026,"exit_status":0.0,"output":"Nintex Workflow maintenance: Successfully processed URLs (-): -/rooms/pep_1910 -/rooms/pep_1928 -/rooms/pep_1944 -/rooms/pep_1948 -/rooms/pep_1961 -/rooms/pep_1962 -/rooms/pep_1964 -/rooms/pep_1976 -/rooms/pep_1983 -/rooms/pep_1984 -/rooms/pep_1985","performance_data":[],"schedule_end":1562568088.369189024,"schedule_start":1562567453.3799231052,"state":0.0,"ttl":0.0,"type":"CheckResult","vars_after":{"attempt":1.0,"reachable":true,"state":0.0,"state_type":1.0},"vars_before":{"attempt":1.0,"reachable":true,"state":2.0,"state_type":1.0}},"last_hard_state":0.0,"last_hard_state_change":1562568088.369189024,"last_reachable":true,"last_state":2.0,"last_state_change":1562568088.369189024,"last_state_critical":1562484295.3799109459,"last_state_ok":1562568088.3691968918,"last_state_type":1.0,"last_state_unknown":0.0,"last_state_unreachable":0.0,"last_state_warning":0.0,"max_check_attempts":3.0,"name":"APP_SP_Nintex_Workflow_Maintenance","next_check":1562581617.0,"notes":"","notes_url":"","original_attributes":null,"package":"_cluster","paused":false,"retry_interval":300.0,"severity":8.0,"source_location":{"first_column":1.0,"first_line":7569.0,"last_column":50.0,"last_line":7569.0,"path":"/var/lib/icinga2/api/zones/director-global/director/service_apply.conf"},"state":0.0,"state_type":1.0,"templates":["APP_SP_Nintex_Workflow_Maintenance","APP_SP_Nintex_Workflow_Maintenance","NRPE","generic-service-24h-24x7","generic-service-24x7","OTRS","host var overrides (Director)"],"type":"Service","vars":{"nrpe_address":"$host.address$","nrpe_arguments":["C:\\FuM\\Script\\NintexMaintenance","Nintex_workflow_maintenance.log","Nintex_Workflow_Maintenance"],"nrpe_command":"check_nintex_workflow_maintenance","nrpe_no_ssl":"$host.vars.nrpe_no_ssl_host$","nrpe_port":"$host.vars.nrpe_port$","nrpe_timeout":"180","nrpe_timeout_unknown":true,"otrs_changetype":"CmCTStandard","otrs_priority":"3 normal - prio 3","otrs_queue":"2_PS::Workplace::SharePoint::Sharepoint Application","otrs_risk":"CmRLow","otrs_service":"Sharepoint Application","otrs_type":"Incident"},"version":0.0,"volatile":false,"zone":"MQDE01"},"joins":{},"meta":{},"name":"mqde01pspweb01!APP_SP_Nintex_Workflow_Maintenance","type":"Service"}]}

The endpoint with the last line has paused: false set being authoritative, inspect this one being operative. Are all checks late, is there any problem with the instance itself?

Sorry for the late reply. I checked wk02 endpoint and is operative and working. But I can tell you that almost all the time we have late checks (Overdue dashboard si full of late host and service checks), Next check value is negative, up to -10 min. Is there way to improve this?

Take your time. This topic isn’t one of these which are easily solved without having direct access to your environment. I am doing my best, but it is hard.

I would especially check the cluster communication. Since you’re saying that those reschedule events are never processed, it might be that everything is piled up.

This includes:

  • How many hosts/services are checked by this specific satellite zone
  • Which underlaying hardware is used (CPU, RAM, etc.)
  • /v1/status/ApiListener via REST API, especially the json_rpc clients.
  • Logs which include the JsonRpcConnection queues logging their items and rate of processing
  • Network traffic in general, how’s the graphs
  • Slow processing of data inside Icinga itself, e.g. when you reload the satellite, how long does it take to fully sync it again with the other endpoints.
  • Is there replay logs which are never processed, or holding off the sync mechanism? Check the content of /var/lib/icinga2/api/log and ensure that rotated files are deleted once sync completed.
  • Does the satellite have any sort of API clients connected to it, via HTTP? E.g. some scripts sending in check results
  • Last but not least, generate a full gdb backtrace from that satellite when it is running and you have sent the force next check. Not a bug hunt, and you won’t understand much of it, but maybe I can see a glance of a locked wait in there.

Cheers,
Michael