Icinga2 sevice does not start after unclean shutdown

kokes · July 1, 2021, 10:20am

After unclean shutdown of monitored nodes (windows server 2012, 2019), significant ammount of nodes remains in “not connected” status, while Icinga2 service does not start - respectively it starts and shuts down immediately.

we are experiencing same issue through multiple icinga2 client versions, from at least 2.10.5 up to 2.12.4
when node shuts down cleanly, it connect back to Icinga master without issues
tried to reinstall client without touching config, no result
tried to remove config from C:\ProgramData\icinga2\var\lib\icinga2\api\zones* (Windows Agent Service Crashing Randomly and wont restart - #17 by csmall), no result
tried to remove complete ProgramData\Icinga2 folder aand re-run config scripts, it did not bring service startup availability back

what only works is to remove agent completely, delete /programdata/icinga2 completely, reinstall agent and rerun assign script include generating new certificates.

Can you please help with further diagnostics? I’m quite lost here

Version used:
v2.12.4
Operating System and version:
2012R2, 2019 Windows server
Enabled features (icinga2 feature list):
Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

Icinga Web 2 version and modules (System - About):
|director|1.7.2|
|incubator|0.5.0|
|ipl|v0.5.0|
|monitoring|2.6.2|
|reactbundle|0.7.0|

Config validation (icinga2 daemon -C)
[2021-07-01 12:03:45 +0200] information/cli: Icinga application loader (version: v2.12.4)
[2021-07-01 12:03:45 +0200] information/cli: Loading configuration file(s).
[2021-07-01 12:03:45 +0200] information/ConfigItem: Committing config item(s).
[2021-07-01 12:03:45 +0200] information/ApiListener: My API identity: SQL01.domain.local
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Active Directory Domain Services’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 9:1-9:48) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Active Directory Web Services’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 18:1-18:45) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘DFS Replication’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 27:1-27:31) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Vyuziti disku’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 44:1-44:29) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Vytizeni CPU’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 52:1-52:28) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Vytizeni pameti’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 60:1-60:31) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Vytizeni site’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 68:1-68:29) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Uptime’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 76:1-76:22) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Vyzaduje restart’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 84:1-84:32) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Active Directory Topology’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 100:1-100:60) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Anti-spam Update’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 109:1-109:51) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Mailbox Transport Delivery’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 118:1-118:61) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange EdgeSync’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 141:1-141:43) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Search’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 150:1-150:41) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Frontend Transport’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 159:1-159:53) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Health Manager’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 168:1-168:49) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Information Store’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 177:1-177:52) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Mailbox Assistants’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 186:1-186:53) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Mailbox Replication’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 195:1-195:54) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange IMAP4’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 204:1-204:40) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange IMAP4 Backend’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 213:1-213:48) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Replication’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 244:1-244:46) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange RPC Client Access’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 253:1-253:52) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Service Host’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 262:1-262:47) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Mailbox Transport Submission’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 271:1-271:63) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Throttling’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 280:1-280:45) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Transport’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 289:1-289:44) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Microsoft Exchange Transport Log Search’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 298:1-298:55) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Tracing Service for Search in Exchange’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 329:1-329:54) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Cluster Service’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 357:1-357:31) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Hyper-V Host Compute Service’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 418:1-418:44) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Hyper-V Virtual Machine Management’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 460:1-460:50) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Local Session Manager’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 494:1-494:37) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Netlogon’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 504:1-504:24) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Network Location Awareness’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 513:1-513:42) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Remote Procedure Call (RPC)’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 522:1-522:43) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Security Accounts Manager’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 531:1-531:41) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Task Scheduler’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 540:1-540:30) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘User Profile Service’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 549:1-549:36) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Windows Event Log’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 558:1-558:33) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Windows Management Instrumentation’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 567:1-567:50) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘DNS Client’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 576:1-576:26) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Group Policy Client’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 585:1-585:35) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Server’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 594:1-594:22) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Windows Time’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 603:1-603:28) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Network List Service’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 612:1-612:36) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Remote Desktop Services’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 621:1-621:39) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Spooler’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 642:1-642:23) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘SQL Browser’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 657:1-657:27) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Hardware status’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 672:1-672:31) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘check_synology-SNMP’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 686:1-686:35) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Vytizeni disku’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 700:1-700:30) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘vytizeni CPU’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 708:1-708:28) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘Vytizeni RAM’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 716:1-716:28) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘check_fortigate_cpu’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 732:1-732:35) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘check_fortigate_mem’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 740:1-740:35) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘check_fortigate_network’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 748:1-748:39) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘check_fortigate_uptime’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 756:1-756:38) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] warning/ApplyRule: Apply rule ‘check_fortigate_hw’ (in C:/ProgramData/icinga2/var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 764:1-764:34) for type ‘Service’ does not match anywhere!
[2021-07-01 12:03:45 +0200] information/ConfigItem: Instantiated 14 HostGroups.
[2021-07-01 12:03:45 +0200] information/ConfigItem: Instantiated 1 TimePeriod.
[2021-07-01 12:03:45 +0200] information/ConfigItem: Instantiated 4 Zones.
[2021-07-01 12:03:45 +0200] information/ConfigItem: Instantiated 1 ServiceGroup.
[2021-07-01 12:03:45 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2021-07-01 12:03:45 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-07-01 12:03:45 +0200] information/ConfigItem: Instantiated 2 Endpoints.
[2021-07-01 12:03:45 +0200] information/ConfigItem: Instantiated 104 CheckCommands.
[2021-07-01 12:03:45 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2021-07-01 12:03:45 +0200] information/ScriptGlobal: Dumping variables to file ‘C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars’
[2021-07-01 12:03:45 +0200] information/cli: Finished validating the configuration file(s).

kokes · July 1, 2021, 1:31pm

I left one of affected servers in this state, so if it would help, I can provide actual logs, perform further testing etc.

Al2Klimov · July 12, 2021, 5:27pm

Hello @kokes!

This should help you:

github.com/Icinga/icinga2

Icinga2 Windows Agent v2.12.3 few seconds after start

opened 07:35PM - 22 May 21 UTC

scpcom

needs-feedback

## Describe the bug On one of our networks we monitor 33 Windows Servers (9 o…f them are RDS/Terminal servers) with agent Icinga2-v2.12.3-x86_64.msi installed. All 33 Servers use the same service apply rules etc defined in global-templates. 8 of the Terminal servers can not be monitored because the "Icinga 2" service crashes few seconds after starting it. Even if I complete remove the host object of an affected server from zones.d on master, uninstall the windows agent, remove C:\ProgramData\icinga2 and reinstall the windows agent it still fails. ".\icinga2.exe feature enable debuglog" does not help, no debug.log is created. icinga2.log does not contain any error icinga2 daemon -C -x notice does not show any error ## To Reproduce Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant. I can not provide all things here because the setup is more complex, it could be GPO or Antivirus related ore something totally different. You can request it here and I will provide more info/configs. ## Expected behavior A running Service sending data to the satellite. ## Screenshots None. ## Your Environment Include as many relevant details about the environment you experienced the problem in * Version used (`icinga2 --version`): v2.12.3 * Operating System and version: Windows Server 2019 Standard * Enabled features (`icinga2 feature list`): api checker debuglog mainlog * Icinga Web 2 version and modules (System - About): 2.8.2 * Config validation (`icinga2 daemon -C`): C:\Program Files\ICINGA2\sbin>icinga2 daemon -C > [2021-05-22 21:03:16 +0200] information/cli: Icinga application loader (version: v2.12.3) > [2021-05-22 21:03:16 +0200] information/cli: Loading configuration file(s). > [2021-05-22 21:03:17 +0200] information/ConfigItem: Committing config item(s). > [2021-05-22 21:03:17 +0200] information/ApiListener: My API identity: TS0001-RDP01.SUBENV0001.example.com > [2021-05-22 21:03:17 +0200] information/ConfigItem: Instantiated 1 CheckerComponent. > [2021-05-22 21:03:17 +0200] information/ConfigItem: Instantiated 4 Zones. > [2021-05-22 21:03:17 +0200] information/ConfigItem: Instantiated 2 Endpoints. > [2021-05-22 21:03:17 +0200] information/ConfigItem: Instantiated 1 ApiListener. > [2021-05-22 21:03:17 +0200] information/ConfigItem: Instantiated 235 CheckCommands. > [2021-05-22 21:03:17 +0200] information/ConfigItem: Instantiated 1 IcingaApplication. > [2021-05-22 21:03:17 +0200] information/ConfigItem: Instantiated 2 FileLoggers. > [2021-05-22 21:03:17 +0200] information/ScriptGlobal: Dumping variables to file 'C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars' > [2021-05-22 21:03:17 +0200] information/cli: Finished validating the configuration file(s). > * If you run multiple Icinga 2 instances, the `zones.conf` file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes. C:\Program Files\ICINGA2\sbin>icinga2 object list --type Endpoint > Object 'TS0001-RDP01.SUBENV0001.example.com' of type 'Endpoint': > % declared in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 15:1-15:53 > * __name = "TS0001-RDP01.SUBENV0001.example.com" > * host = "" > * log_duration = 86400 > * name = "TS0001-RDP01.SUBENV0001.example.com" > * package = "_etc" > * port = "5665" > * source_location > * first_column = 1 > * first_line = 15 > * last_column = 53 > * last_line = 15 > * path = "C:/ProgramData/icinga2/etc/icinga2/zones.conf" > * templates = [ "TS0001-RDP01.SUBENV0001.example.com" ] > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 15:1-15:53 > * type = "Endpoint" > * zone = "" > > Object 'mon0001-01.SUBENV0001.example.com' of type 'Endpoint': > % declared in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 6:1-6:51 > * __name = "mon0001-01.SUBENV0001.example.com" > * host = "10.20.130.243" > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 7:2-7:23 > * log_duration = 86400 > * name = "mon0001-01.SUBENV0001.example.com" > * package = "_etc" > * port = "5665" > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 8:2-8:14 > * source_location > * first_column = 1 > * first_line = 6 > * last_column = 51 > * last_line = 6 > * path = "C:/ProgramData/icinga2/etc/icinga2/zones.conf" > * templates = [ "mon0001-01.SUBENV0001.example.com" ] > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 6:1-6:51 > * type = "Endpoint" > * zone = "" > C:\Program Files\ICINGA2\sbin>icinga2 object list --type Zone > Object 'director-global' of type 'Zone': > % declared in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 27:1-27:29 > * __name = "director-global" > * endpoints = null > * global = true > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 28:2-28:14 > * name = "director-global" > * package = "_etc" > * parent = "" > * source_location > * first_column = 1 > * first_line = 27 > * last_column = 29 > * last_line = 27 > * path = "C:/ProgramData/icinga2/etc/icinga2/zones.conf" > * templates = [ "director-global" ] > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 27:1-27:29 > * type = "Zone" > * zone = "" > > Object 'master' of type 'Zone': > % declared in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 11:1-11:20 > * __name = "master" > * endpoints = [ "mon0001-01.SUBENV0001.example.com" ] > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 12:2-12:52 > * global = false > * name = "master" > * package = "_etc" > * parent = "" > * source_location > * first_column = 1 > * first_line = 11 > * last_column = 20 > * last_line = 11 > * path = "C:/ProgramData/icinga2/etc/icinga2/zones.conf" > * templates = [ "master" ] > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 11:1-11:20 > * type = "Zone" > * zone = "" > > Object 'TS0001-RDP01.SUBENV0001.example.com' of type 'Zone': > % declared in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 18:1-18:49 > * __name = "TS0001-RDP01.SUBENV0001.example.com" > * endpoints = [ "TS0001-RDP01.SUBENV0001.example.com" ] > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 19:2-19:54 > * global = false > * name = "TS0001-RDP01.SUBENV0001.example.com" > * package = "_etc" > * parent = "master" > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 20:2-20:18 > * source_location > * first_column = 1 > * first_line = 18 > * last_column = 49 > * last_line = 18 > * path = "C:/ProgramData/icinga2/etc/icinga2/zones.conf" > * templates = [ "TS0001-RDP01.SUBENV0001.example.com" ] > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 18:1-18:49 > * type = "Zone" > * zone = "" > > Object 'global-templates' of type 'Zone': > % declared in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 23:1-23:30 > * __name = "global-templates" > * endpoints = null > * global = true > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 24:2-24:14 > * name = "global-templates" > * package = "_etc" > * parent = "" > * source_location > * first_column = 1 > * first_line = 23 > * last_column = 30 > * last_line = 23 > * path = "C:/ProgramData/icinga2/etc/icinga2/zones.conf" > * templates = [ "global-templates" ] > % = modified in 'C:/ProgramData/icinga2/etc/icinga2/zones.conf', lines 23:1-23:30 > * type = "Zone" > * zone = "" > ## Additional context Event Log Application Error > Name der fehlerhaften Anwendung: icinga2.exe, Version: 1.0.0.0, Zeitstempel: 0x5fd8c0c1 > Name des fehlerhaften Moduls: KERNELBASE.dll, Version: 10.0.17763.1911, Zeitstempel: 0x206fb2f7 > Ausnahmecode: 0xe06d7363 > Fehleroffset: 0x0000000000039689 > ID des fehlerhaften Prozesses: 0x1d30 > Startzeit der fehlerhaften Anwendung: 0x01d74f3205bd9e72 > Pfad der fehlerhaften Anwendung: C:\Program Files\ICINGA2\sbin\icinga2.exe > Pfad des fehlerhaften Moduls: C:\Windows\System32\KERNELBASE.dll > Berichtskennung: b7efd99f-a57e-48d4-ae33-63b60d542297 > Vollständiger Name des fehlerhaften Pakets: > Anwendungs-ID, die relativ zum fehlerhaften Paket ist: ## Workaround Running as Administrator in foreground with "icinga2 daemon -x notice" all works as expected. Changing the service user from "Network Service" to "Local System" and all works as expected.

Best,
AK

kokes · July 16, 2021, 8:09am

Thanks for feedback,
I tried recommended solutions (changing account service runs under), but without siccess. Trying to run Icinga agent interactively ends with following (partial log):

[2021-07-16 10:08:00 +0200] information/ConfigItem: Instantiated 14 HostGroups.
[2021-07-16 10:08:00 +0200] information/ConfigItem: Instantiated 1 ServiceGroup.
[2021-07-16 10:08:00 +0200] information/ConfigItem: Instantiated 2 Endpoints.
[2021-07-16 10:08:00 +0200] information/ConfigItem: Instantiated 1 TimePeriod.
[2021-07-16 10:08:00 +0200] information/ConfigItem: Instantiated 4 Zones.
[2021-07-16 10:08:00 +0200] information/ConfigItem: Instantiated 104 CheckCommands.
[2021-07-16 10:08:00 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-07-16 10:08:00 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2021-07-16 10:08:00 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2021-07-16 10:08:00 +0200] information/ScriptGlobal: Dumping variables to file 'C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars'
[2021-07-16 10:08:00 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonUtility::LoadConfigFiles'
[2021-07-16 10:08:00 +0200] information/ConfigObject: Restoring program state from file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-16 10:08:00 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'ConfigObject::RestoreObjects'
[2021-07-16 10:08:00 +0200] critical/cli: Failed to restore state file: Error: Invalid NetString (missing :)

This is similar to this closed ticket: [dev.icinga.com #12631] Failed to restore state file: Error: Invalid NetString (missing :) · Issue #4628 · Icinga/icinga2 · GitHub

As I mentioned, we left one server in this state for further analysis, because this issue is appearing regullary on our systems. We can try do as many fix/diag attempts as we need.

jbrost · July 16, 2021, 8:32am

Can you take a look at the state file in a text editor and see if it looks suspicious in some way? It should be a netstring-encoded list of JSON objects, i.e. look something like 42:JSON object of 42 bytes length,23:JSON objects of 23 bytes length,....

So probably there’s an issue with how Icinga does file operations on Windows and what consistency guarantees it expects from this, see also

github.com/Icinga/icinga2

Verify consistency guarantees of file operations on Windows

opened 09:49AM - 22 Jun 21 UTC

julianbrost

core/evaluate

Icinga uses the pattern of writing a new version of a file to a temporary file l…ocation and once that's done, moving it to the final location replacing the old version. We should evaluate if this also gives the desired consistency on Windows, given that things might work differently and there's Boost in between potentially hiding which syscalls are actually used behind the scenes. Rationale: #8528 reported that an agent on Windows didn't start because the state file contained all spaces, which sounds unlikely to be caused a hardware fault or user error.

jbrost · July 16, 2021, 8:36am

Also, instead of completely reinstalling Icinga, you can also try to just remove that state file.

kokes · July 16, 2021, 9:31am

I looked into mentioned file - it has 13kb of size and is full of empty characters:

removing this file seems to fix the issue! Thanks for pointiing me right direction!