REST API not responding

Icinga 2
,
1

Hello all,

I recently tried changing an existing Icinga service to an HA master-master set-up. I completed the initial config sync and set up the zones.conf on both masters. This worked fine at first but then the REST API has totally stopped responding to requests via curl e.g.

curl -k -u root:<password> "https://<primary_master>:5665/v1" -vvv from the primary master itself gives:

* About to connect() to <primary_fqdn> port 5665 (#0)
*   Trying <primary_ip>...
* Connected to <primary_fqdn> (<primary_ip>) port 5665 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=<primary_fqdn>
*       start date: Dec 11 08:50:36 2019 GMT
*       expire date: Dec 07 08:50:36 2034 GMT
*       common name: <primary_fqdn>
*       issuer: CN=Icinga CA
* Server auth using Basic with user 'root'
> GET /v1 HTTP/1.1
> Authorization: Basic cm9vdDozODRjYWQ2OGY0YTc5Njc3
> User-Agent: curl/7.29.0
> Host: <primary_fqdn>:5665
> Accept: */*
>
* Empty reply from server
* Connection #0 to host <primary_fqdn> left intact
curl: (52) NSS: client certificate not found (nickname not specified)

implying a succcesful connection and authentication but no response. I’ve since tried turned off the secondary master but the problem remains.

This is using Icinga2 2.11.5 on CentOS7. The ApiListener is in features-enabled/api.conf:

object ApiListener "api" {
  accept_config = true
  accept_commands = true

  ticket_salt = TicketSalt

  tls_handshake_timeout = 60
}

The ApiUser is in zones.d/icinga2-ha-zone/templates/api-users.conf:

object ApiUser "root" {
  password = "<password>"
  // client_cn = ""

  permissions = [ "*" ]
}

I can’t find any relevant log messages in icinga2.log.

Does anyone have any insights?

Many thanks,
Adam

2

Did you recently add the accept_config = true option after adding the second master or was it always like that? I historically have always had that off (causes a feedback loop on versions older than 2.11, I think they’ve changed that).

3

Unfortunately not, I added that after adding the secondary master as an attempt to get the API working again.

4

I’m tentatively saying I’ve fixed this by deleting /var/lib/icinga2/icinga2.state and everything under /var/lib/icinga2/api/{logs,zones,zones-stage} then restarting icinga.