Yes, it is:
Well, the problem definitely is somewhere in the config or in the config distribution to the containers, I guess.
Can you restart one of the containers to make it resync. Maybe even delete the contents of /var/lib/icinga2/api/ to be sure the config will get resynced for sure.
I did the following steps now:
- Restarted the LXC-Container ā no change
- Deleted the whole content from /var/lib/icinga2/api/ and restarted the container again ā no change
- Deployed the host-config again from director to the agent ā no change
- I also restarted the master (also a LXC on the Proxmox-Host) ā no change
Is it possible to trace the messages from the agent on the master to check, if the packages are shown in the output? Maybe this could narrow if its a problem of the master or the agent?
You could try a tail -f /var/log/icinga2.log on both master and agent and see what is written there, when you hit recheck.
Maybe you will have to enable the debug log and do the same with the debug.log file.
I had no luck with the (debug)logsā¦
On Agent:
[2021-09-24 20:03:01 +0200] notice/JsonRpcConnection: Received 'event::ExecuteCommand' message from 'icinga2.home.domain.de'
[2021-09-24 20:03:01 +0200] notice/Process: Running command '/usr/lib/nagios/plugins/check_apt': PID 11158
[2021-09-24 20:03:03 +0200] notice/Process: PID 11158 ('/usr/lib/nagios/plugins/check_apt') terminated with exit code 1
[2021-09-24 20:03:03 +0200] notice/ApiListener: Sending message 'event::CheckResult' to 'icinga2.home.domain.de'
On master:
[2021-09-24 20:03:01 +0200] debug/CheckerComponent: Scheduling info for checkable 'grafana.home.domain.de!check_apt' (2021-09-24 20:03:00 +0200): Object 'grafana.home.domain.de!check_apt', Next Check: 2021-09-24 20:03:00 +0200(1.63251e+09).
[2021-09-24 20:03:01 +0200] debug/CheckerComponent: Executing check for 'grafana.home.domain.de!check_apt'
[2021-09-24 20:03:01 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2021-09-24 20:03:01 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2021-09-24 20:03:01 +0200] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2021-09-24 20:03:01 +0200] debug/Checkable: Update checkable 'grafana.home.domain.de!check_apt' with check interval '300' from last check time at 2021-09-24 19:58:37 +0200 (1.63251e+09) to next check time at 2021-09-24 20:07:50 +0200 (1.63251e+09).
[2021-09-24 20:03:01 +0200] notice/ApiListener: Sending message 'event::ExecuteCommand' to 'grafana.home.domain.de'
[2021-09-24 20:03:01 +0200] debug/CheckerComponent: Check finished for object 'grafana.home.domain.de!check_apt'
[2021-09-24 20:03:01 +0200] notice/ApiListener: Relaying 'event::SetLastCheckStarted' message
[2021-09-24 20:03:01 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2021-09-24 20:03:01 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2021-09-24 20:03:01 +0200] information/HttpServerConnection: HTTP client disconnected (from [::ffff:192.168.1.205]:37112)
Iām out of ideas 
No clue where this is coming from.
Okay, I did some research I got was the idea, that maybe the Icinga on which your grafana runs might be out of date (2.10 or less).
Could you check that?
3 Likes
I think you could be right!
I checked the versions on the different hosts:
On hosts where --list shows an output this version is installed:
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.1-1)
On host without the output of the packages this version is installed:
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.3-1)
But i am really amazed because all hosts are running on debian 10 and i installed icinga in the same way with apt.
Also there are no upgradeable packages shown for icinga.
1 Like
Odd, not a debian guy myself (for work anyways), but you might make sure that the repos match and the packages arenāt lockedā¦ and that they are enabledā¦ is that a thing on debian?
Well, that sounds like we found the underlying issue then, it will work with 2.11 and upwards 
yes, thank you very muchā¦ surely it was no critical thing, but its great to know the issue now.
i also checked the package sources in detail and there was the issue: on hosts with v2.13 the icinga repos were added:
deb https://packages.icinga.com/debian icinga-buster main
deb-src https://packages.icinga.com/debian icinga-buster main
on hosts with v2.10 the package came from debian source.
additionally i want give many many thanks to @log1c for his for his effort and patience!
2 Likes
great the issue is resolved
I donāt think I would have thought of the Icinga version or the packages, I was too caught up in the config 