Event Stream request returning 503.0 "Unhandled exception"

Icinga 2
1

I’m trying to access the API Event Stream (to minimize latency for my dashboard), but keep getting 503 “Unhandled exception” returned. I expected to see JSON fragments.

Configuration Information

  • Version used (icinga2 --version)
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.3-1)

Copyright (c) 2012-2021 Icinga GmbH (<link removed>)
License GPLv2+: GNU GPL version 2 or later <link removed>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Raspbian GNU/Linux
  Platform version: 10 (buster)
  Kernel: Linux
  Kernel version: 5.10.11-v7l+
  Architecture: armv7l

Build information:
  Compiler: GNU 8.3.0
  Build host: bm-wb-03

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

  • Operating System and version (uname -a)

    • uname -a returns Linux raspberrypi 5.10.11-v7l+ #1399 SMP Thu Jan 28 12:09:48 GMT 2021 armv7l GNU/Linux
    • running on a Raspberry Pi 4 (2GB), and starting with apt update and apt upgrade

  • Enabled features (icinga2 feature list)

Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-pgsql mainlog notification

  • Icinga Web 2 version and modules (System - About)

    • Not Installed

  • Config validation (icinga2 daemon -C)

[2021-02-12 13:32:27 +0000] information/cli: Icinga application loader (version: r2.10.3-1)
[2021-02-12 13:32:27 +0000] information/cli: Loading configuration file(s).
[2021-02-12 13:32:27 +0000] information/ConfigItem: Committing config item(s).
[2021-02-12 13:32:28 +0000] information/ApiListener: My API identity: raspberrypi
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 IdoPgsqlConnection.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 User.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 3 Zones.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 12 Services.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 Host.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 13 Notifications.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 Endpoint.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 Downtime.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 215 CheckCommands.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2021-02-12 13:32:28 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2021-02-12 13:32:28 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-02-12 13:32:28 +0000] information/cli: Finished validating the configuration file(s).
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes
    • Not Installed

Background

I first found this problem on my main Icinga2 machine that monitors 16 hosts and 84 services. I assumed that I’d messed up the configuration somehow, so I’ve built an new clean installation on a different Pi, following the instructions on latest/doc/02-installation/ and latest/doc/12-icinga2-api/ web pages.

Notes:

  • I choose the icinga2 package from the “normal” Raspberry Pi OS repository rather than “official repository”, which is why I’m running with Icinga2 version r2.10.3-1. However, I have looked through the issues and release notes and not seen anything that looks like my problem;
  • I choose to use a PostgreSQL database, relying mostly on Debians smart installation to set everything up.
  • I did edited the pg_hba.conf file as indicated on the Icinga2 Installation web page, adding
# icinga2
local   icinga          icinga                                  md5
host    icinga          icinga          127.0.0.1/32            md5
host    icinga          icinga          ::1/128                 md5
  • I got the “root” password from the api-users.conf file
  • I proved the basic API with curl -k -s -u root:cd528f862446cd02 'https://localhost:5665/v1'
  • I tried a few other requests asking for object\hosts and status
  • I cut-‘n’-paste the Event Stream request curl -k -s -u root:cd528f862446cd02 -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/events' -d '{ "queue": "myqueue", "types": "CheckResult", "filter": "event.check_result.exit_status==2" }' (editing the password).
  • it returns {"error":503.0,"status":"Unhandled exception"}
  • I tailed -f the /var/log/icinga2/icinga2.log file, repeating the request and observed:
[2021-02-12 13:52:17 +0000] information/ApiListener: New client connection from [127.0.0.1]:39678 (no client certificate)
[2021-02-12 13:52:17 +0000] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2021-02-12 13:52:17 +0000] information/HttpServerConnection: Request: POST /v1/events (from [127.0.0.1]:39678), user: root)
[2021-02-12 13:52:17 +0000] critical/HttpServerConnection: Unhandled exception while processing Http request: Error: Cannot convert value of type 'String' to an object.

	(0) icinga2: void boost::throw_exception<boost::exception_detail::error_info_injector<std::runtime_error> boost::exception_detail::error_info_injector<std::runtime_error> const&) (+0x80) [0x37740c]
	(1) icinga2: icinga::EventsHandler::HandleRequest(boost::intrusive_ptr<icinga::ApiUser> const&, icinga::HttpRequest&, icinga::HttpResponse&, boost::intrusive_ptr<icinga::Dictionary> const&) (+0xc10) [0x516e78]
	(2) icinga2: icinga::HttpHandler::ProcessRequest(boost::intrusive_ptr<icinga::ApiUser> const&, icinga::HttpRequest&, icinga::HttpResponse&) (+0x4a4) [0x51ef90]
	(3) icinga2: icinga::HttpServerConnection::ProcessMessageAsync(icinga::HttpRequest&, icinga::HttpResponse&, boost::intrusive_ptr<icinga::ApiUser> const&) (+0x3c) [0x523bbc]
	(4) icinga2: icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) (+0x30) [0x4793dc]
	(5) icinga2: icinga::WorkQueue::WorkerThreadProc() (+0x42c) [0x479990]
	(6) libboost_thread.so.1.67.0: <unknown function> (+0xfcdc) [0xb6ed0cdc]


[2021-02-12 13:52:17 +0000] information/HttpServerConnection: HTTP client disconnected (from [127.0.0.1]:39678)
  • Now I’m stuck.

I can go back to polling the API (which I have got working previously), but the Event Stream feels like the right interface for what I am trying to do. Does anyone have any suggestions on how to proceed?
Thanks in advance,
Steve

Using the Official Icinga Repository

I’ve tried again, starting from a completely clean micro-SD card on the same Pi, but this time I followed the instructions to use the official raspbian distribution.

  • icinga2 --version:
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.3-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Raspbian GNU/Linux
  Platform version: 10 (buster)
  Kernel: Linux
  Kernel version: 5.10.11-v7l+
  Architecture: armv7l

Build information:
  Compiler: GNU 8.3.0
  Build host: runner-hh8q3bz2-project-297-concurrent-0
  OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
  • icinga2 daemon -C
[2021-02-13 11:41:09 +0000] information/cli: Icinga application loader (version: r2.12.3-1)
[2021-02-13 11:41:09 +0000] information/cli: Loading configuration file(s).
[2021-02-13 11:41:09 +0000] information/ConfigItem: Committing config item(s).
[2021-02-13 11:41:09 +0000] information/ApiListener: My API identity: icinga2
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 Host.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 Downtime.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 13 Notifications.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 3 Zones.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 Endpoint.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 235 CheckCommands.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 User.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 12 Services.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2021-02-13 11:41:09 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2021-02-13 11:41:09 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-02-13 11:41:09 +0000] information/cli: Finished validating the configuration file(s).
  • curl -k -s -u root:73817938975e48b5 'https://localhost:5665/v1'
<html><head><title>Icinga 2</title></head><h1>Hello from Icinga 2 (Version: r2.12.3-1)!</h1><p>You are authenticated as <b>root</b>. Your user has the following permissions:</p> <ul><li>*</li></ul><p>More information about API requests is available in the <a href="https://icinga.com/docs/icinga2/latest/" target="_blank">documentation</a>.</p></html>
  • curl -k -s -u root:73817938975e48b5 -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/events' -d '{ "queue": "myqueue", "types": "CheckResult", "filter": "event.check_result.exit_status==2" }'
{"error":404.0,"status":"The requested path 'v1/events' could not be found or the request method is not valid for this path."}
  • tail /var/log/icinga2/icinga2.log
[2021-02-13 11:46:11 +0000] information/ApiListener: New client connection from [127.0.0.1]:49484 (no client certificate)
[2021-02-13 11:46:11 +0000] information/HttpServerConnection: Request: POST /v1/events (from [127.0.0.1]:49484), user: root, agent: curl/7.64.0, status: Not Found).
[2021-02-13 11:46:11 +0000] information/HttpServerConnection: HTTP client disconnected (from [127.0.0.1]:49484)

So now I’ve got a different error, but still not the output predicted by the https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#event-streams webpage. Has something changed and the webpage not been updated?

2

Hii @srpalmer, Welcome to Icinga Community.

Yes, there were syntax errors but they have all been fixed and is already merged. You just have to look at snapshot doc and not the latest.

And the same question was asked and answered here. You can take the API queries from there.

Hope this’ll Help you!

3

Thanks Yonas,
[...] around the “types” argument has cracked it.
Cheers,
Steve