Icinga DB is not running (cannot link to Redis?..)

Icinga Web
,
1

Hello,

Finally, IC2 is running, but mentionned the message :
“It seems that Icinga DB is not running. Make sure Icinga DB is running and writing into the database.”

Looking at the status, it cannot receive heartbeat from icinga, Following the doc, Redis was first installed, icinga2-selinux also but set as permissive following previous problem (perhaps linked). Password and acces seems ok.

When I set up selinux to enforcing the error message is :
“Can’t connect to Redis: Permission denied [tcp://localhost:6380]”

icingadb-redis is running
icingadb not working well (waiting for Icinga Heartbeat)

What am I suppose to parameter (not familiar with Selinux / Redis)
Could you help ? Thanks

sudo systemctl status icingadb
× icingadb.service - Icinga DB
     Loaded: loaded (/usr/lib/systemd/system/icingadb.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2024-12-04 15:01:08 CET; 2min 6s ago
   Duration: 5min 9ms
    Process: 31572 ExecStart=/usr/sbin/icingadb --config /etc/icingadb/config.yml (code=exited, status=1/FAILURE)
   Main PID: 31572 (code=exited, status=1/FAILURE)
        CPU: 1.297s

Dec 04 14:56:08 lmonicp24.lx.finbel.intra icingadb[31572]: Connecting to Redis at 'localhost:6380'
Dec 04 14:56:08 lmonicp24.lx.finbel.intra icingadb[31572]: Starting history sync
Dec 04 14:57:08 lmonicp24.lx.finbel.intra icingadb[31572]: heartbeat: Waiting for Icinga heartbeat
Dec 04 14:58:08 lmonicp24.lx.finbel.intra icingadb[31572]: heartbeat: Waiting for Icinga heartbeat
Dec 04 14:59:08 lmonicp24.lx.finbel.intra icingadb[31572]: heartbeat: Waiting for Icinga heartbeat
Dec 04 15:00:08 lmonicp24.lx.finbel.intra icingadb[31572]: heartbeat: Waiting for Icinga heartbeat
Dec 04 15:01:08 lmonicp24.lx.finbel.intra icingadb[31572]: retry deadline exceeded
                                                           github.com/icinga/icingadb/pkg/icingadb.
                                                          (....removed unecessary redondant text)
                                                                   runtime/asm_amd64.s:1695
Dec 04 15:01:08 lmonicp24.lx.finbel.intra systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 04 15:01:08 lmonicp24.lx.finbel.intra systemd[1]: icingadb.service: Failed with result 'exit-code'.
Dec 04 15:01:08 lmonicp24.lx.finbel.intra systemd[1]: icingadb.service: Consumed 1.297s CPU time.

My conf file : (defaut) !!! but I read “permission denied from the deamon” - how to fix ?
(warning/ConfigCompiler: Cannot compile file ‘/etc/icinga2/features-enabled/icingadb.conf’: Error: Function call ‘std::ifstream::open’ for file ‘/etc/icinga2/features-enabled/icingadb.conf’ failed with error code 13, ‘Permission denied’)
User icinga;icinga

sudo ls -ltr /etc/icinga2/features-enabled/icingadb.conf
lrwxrwxrwx. 1 icinga icinga 35 Nov 28 14:21 /etc/icinga2/features-enabled/icingadb.conf -> ../features-available/icingadb.conf

sudo cat /etc/icinga2/features-available/icingadb.conf
object IcingaDB "icingadb" {
  //host = "127.0.0.1"
  //port = 6380
  //password = "xxx"

Version used : Last release icinga2
Operating System and version : RHLE9
Enabled features (icinga2 feature list)
Last Icinga Web 2 version and Director modules
Config validation : icinga2 daemon -C

sudo icinga2 daemon -C
[2024-12-04 15:27:00 +0100] information/cli: Icinga application loader (version: r2.14.2-1)
[2024-12-04 15:27:00 +0100] information/cli: Loading configuration file(s).
[2024-12-04 15:27:00 +0100] warning/ConfigCompiler: Cannot compile file '/etc/icinga2/features-enabled/icingadb.conf': Error: Function call 'std::ifstream::open' for file '/etc/icinga2/features-enabled/icingadb.conf' failed with error code 13, 'Permission denied'

Context:

        (0) Compiling configuration file '/etc/icinga2/features-enabled/icingadb.conf'
[2024-12-04 15:27:00 +0100] information/ConfigItem: Committing config item(s).
[2024-12-04 15:27:00 +0100] information/ApiListener: My API identity: lmonicp24.lx.finbel.intra
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 12 Notifications.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 2 HostGroups.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 Host.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 Downtime.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 3 Zones.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 User.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 Endpoint.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 246 CheckCommands.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 3 ServiceGroups.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 3 TimePeriods.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2024-12-04 15:27:00 +0100] information/ConfigItem: Instantiated 11 Services.
[2024-12-04 15:27:00 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2024-12-04 15:27:00 +0100] information/cli: Finished validating the configuration file(s).
2

image
image1183×411 12.6 KB

3

? No one, nobody can help or give me a clue ?
:slight_smile:

icingadb-redis status :

systemctl status icingadb-redis
● icingadb-redis.service - Redis database for Icinga DB
     Loaded: loaded (/usr/lib/systemd/system/icingadb-redis.service; enabled; preset: disabled)
     Active: active (running) since Wed 2024-12-04 16:20:00 CET; 18h ago
   Main PID: 2678 (icingadb-redis-)
     Status: "Ready to accept connections"
      Tasks: 5 (limit: 35733)
     Memory: 10.8M
        CPU: 1min 28.627s
     CGroup: /system.slice/icingadb-redis.service
             └─2678 "/usr/bin/icingadb-redis-server 127.0.0.1:6380"
4

Please try to have some patience, even if things are not working as expected.

The essential error seems to be the following:

warning/ConfigCompiler: Cannot compile file '/etc/icinga2/features-enabled/icingadb.conf': Error: Function call 'std::ifstream::open' for file '/etc/icinga2/features-enabled/icingadb.conf' failed with error code 13, 'Permission denied'

You have posted information about the symlink in your initial message, but could you please show the stats of the actual file, /etc/icinga2/features-available/icingadb.conf, and could you please try to cat this file as your Icinga user?

As you have written about SELinux, could you please show the audit.log?

5

Thanks for reply !

here some info :

ls -l  /etc/icinga2/features-available/icingadb.conf
-rw-r-----. 1 root root 91 Dec  2 11:57 /etc/icinga2/features-available/icingadb.conf

more  /etc/icinga2/features-available/icingadb.conf
object IcingaDB "icingadb" {
  //host = "127.0.0.1"
  //port = 6380
  //password = "xxx"
}

I extracted the last 70 last lines of (hopefully enough) :

more /var/log/audit/audit.log

3002F6574632F6D6574726963626561742D6C78732F6D6574726963626561742E796D6C002D2D706174682E686F6D65002F7573722F73686172652F6D65747269636265
type=SYSCALL msg=audit(1733395466.348:177230): arch=c000003e syscall=49 success=yes exit=0 a0=10 a1=c002bd40b4 a2=c a3=0 items=0 ppid=1 pid=1025 auid=4294967295 uid=0 gid=0 euid=0
suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="metricbeat" exe="/usr/share/metricbeat/bin/metricbeat" subj=system_u:system_r:unconfined_service_t:s0 key="exte
rnal-access"ARCH=x86_64 SYSCALL=bind AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=SOCKADDR msg=audit(1733395466.348:177230): saddr=100000000000000000000000SADDR={ saddr_fam=netlink nlnk-fam=16 nlnk-pid=0 }
type=PROCTITLE msg=audit(1733395466.348:177230): proctitle=2F7573722F73686172652F6D6574726963626561742F62696E2F6D657472696362656174002D2D656E7669726F6E6D656E740073797374656D64002D6
3002F6574632F6D6574726963626561742D6C78732F6D6574726963626561742E796D6C002D2D706174682E686F6D65002F7573722F73686172652F6D65747269636265
type=SYSCALL msg=audit(1733395466.348:177231): arch=c000003e syscall=49 success=yes exit=0 a0=10 a1=c002bd4144 a2=c a3=0 items=0 ppid=1 pid=1025 auid=4294967295 uid=0 gid=0 euid=0
suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="metricbeat" exe="/usr/share/metricbeat/bin/metricbeat" subj=system_u:system_r:unconfined_service_t:s0 key="exte
rnal-access"ARCH=x86_64 SYSCALL=bind AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=SOCKADDR msg=audit(1733395466.348:177231): saddr=100000000000000000000000SADDR={ saddr_fam=netlink nlnk-fam=16 nlnk-pid=0 }
type=PROCTITLE msg=audit(1733395466.348:177231): proctitle=2F7573722F73686172652F6D6574726963626561742F62696E2F6D657472696362656174002D2D656E7669726F6E6D656E740073797374656D64002D6
3002F6574632F6D6574726963626561742D6C78732F6D6574726963626561742E796D6C002D2D706174682E686F6D65002F7573722F73686172652F6D65747269636265
type=SYSCALL msg=audit(1733395466.962:177232): arch=c000003e syscall=42 success=yes exit=0 a0=b a1=c0020aa44c a2=10 a3=0 items=0 ppid=1 pid=1004 auid=4294967295 uid=0 gid=0 euid=0
suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="filebeat" exe="/usr/share/filebeat/bin/filebeat" subj=system_u:system_r:unconfined_service_t:s0 key="external-a
ccess"ARCH=x86_64 SYSCALL=connect AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=SOCKADDR msg=audit(1733395466.962:177232): saddr=020000350A1576A30000000000000000SADDR={ saddr_fam=inet laddr=10.21.118.163 lport=53 }
type=PROCTITLE msg=audit(1733395466.962:177232): proctitle=2F7573722F73686172652F66696C65626561742F62696E2F66696C6562656174002D2D656E7669726F6E6D656E740073797374656D64002D63002F657
4632F66696C65626561742D6C78732F66696C65626561742E796D6C002D2D706174682E686F6D65002F7573722F73686172652F66696C6562656174002D2D706174682E
type=SYSCALL msg=audit(1733395466.962:177233): arch=c000003e syscall=42 success=yes exit=0 a0=11 a1=c00167b6cc a2=10 a3=0 items=0 ppid=1 pid=1004 auid=4294967295 uid=0 gid=0 euid=0
 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="filebeat" exe="/usr/share/filebeat/bin/filebeat" subj=system_u:system_r:unconfined_service_t:s0 key="external-
access"ARCH=x86_64 SYSCALL=connect AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=SOCKADDR msg=audit(1733395466.962:177233): saddr=020000350A1576A30000000000000000SADDR={ saddr_fam=inet laddr=10.21.118.163 lport=53 }
type=PROCTITLE msg=audit(1733395466.962:177233): proctitle=2F7573722F73686172652F66696C65626561742F62696E2F66696C6562656174002D2D656E7669726F6E6D656E740073797374656D64002D63002F657
4632F66696C65626561742D6C78732F66696C65626561742E796D6C002D2D706174682E686F6D65002F7573722F73686172652F66696C6562656174002D2D706174682E
type=SYSCALL msg=audit(1733395466.963:177234): arch=c000003e syscall=42 success=no exit=-115 a0=b a1=c00167b76c a2=10 a3=0 items=0 ppid=1 pid=1004 auid=4294967295 uid=0 gid=0 euid=
0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="filebeat" exe="/usr/share/filebeat/bin/filebeat" subj=system_u:system_r:unconfined_service_t:s0 key="external
-access"ARCH=x86_64 SYSCALL=connect AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=SOCKADDR msg=audit(1733395466.963:177234): saddr=020001BB0A1B141C0000000000000000SADDR={ saddr_fam=inet laddr=10.27.20.28 lport=443 }
type=PROCTITLE msg=audit(1733395466.963:177234): proctitle=2F7573722F73686172652F66696C65626561742F62696E2F66696C6562656174002D2D656E7669726F6E6D656E740073797374656D64002D63002F657
4632F66696C65626561742D6C78732F66696C65626561742E796D6C002D2D706174682E686F6D65002F7573722F73686172652F66696C6562656174002D2D706174682E
type=SYSCALL msg=audit(1733395469.558:177235): arch=c000003e syscall=59 success=yes exit=0 a0=1db7200 a1=1db7290 a2=1dba590 a3=7f9adf610398 items=2 ppid=29459 pid=29911 auid=429496
7295 uid=984 gid=984 euid=984 suid=984 fsuid=984 egid=984 sgid=984 fsgid=984 tty=(none) ses=4294967295 comm="check_load" exe="/usr/lib64/nagios/plugins/check_load" subj=system_u:sy
stem_r:nagios_unconfined_plugin_t:s0 key="exec"ARCH=x86_64 SYSCALL=execve AUID="unset" UID="icinga" GID="icinga" EUID="icinga" SUID="icinga" FSUID="icinga" EGID="icinga" SGID="icin
ga" FSGID="icinga"
type=EXECVE msg=audit(1733395469.558:177235): argc=5 a0="/usr/lib64/nagios/plugins/check_load" a1="-c" a2="10,6,4" a3="-w" a4="5,4,3"
type=CWD msg=audit(1733395469.558:177235): cwd="/"
type=PATH msg=audit(1733395469.558:177235): item=0 name="/usr/lib64/nagios/plugins/check_load" inode=17932817 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:
nagios_unconfined_plugin_exec_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PATH msg=audit(1733395469.558:177235): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=16800525 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s
0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PROCTITLE msg=audit(1733395469.558:177235): proctitle=2F7573722F6C696236342F6E6167696F732F706C7567696E732F636865636B5F6C6F6164002D630031302C362C34002D7700352C342C33
type=SYSCALL msg=audit(1733395469.561:177236): arch=c000003e syscall=59 success=yes exit=0 a0=555cba9d0320 a1=555cba9d0340 a2=7fff0e33ecb0 a3=7fb1dbb97008 items=2 ppid=29911 pid=29
912 auid=4294967295 uid=984 gid=984 euid=984 suid=984 fsuid=984 egid=984 sgid=984 fsgid=984 tty=(none) ses=4294967295 comm="uptime" exe="/usr/bin/uptime" subj=system_u:system_r:nag
ios_unconfined_plugin_t:s0 key="exec"ARCH=x86_64 SYSCALL=execve AUID="unset" UID="icinga" GID="icinga" EUID="icinga" SUID="icinga" FSUID="icinga" EGID="icinga" SGID="icinga" FSGID=
"icinga"
type=EXECVE msg=audit(1733395469.561:177236): argc=1 a0="/usr/bin/uptime"
type=CWD msg=audit(1733395469.561:177236): cwd="/"
type=PATH msg=audit(1733395469.561:177236): item=0 name="/usr/bin/uptime" inode=8566398 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORM
AL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PATH msg=audit(1733395469.561:177236): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=16800525 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s
0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"
type=PROCTITLE msg=audit(1733395469.561:177236): proctitle="/usr/bin/uptime"

thank you very much for your help !

6

:slight_smile:
Replying to your reply I saw a first problem I already corrected !!

For sure : I chown icinga instead of root and this resolved half the problem :

-rw-r-----. 1 root root 91 Dec  2 11:57 icingadb.conf

is now

-rw-r-----. 1 icinga icinga  91 Dec  2 11:57 icingadb.conf

The remaining issue is now about Redis ::

Redis by icingadb is CRITICAL
Can't connect to Redis: Permission denied [tcp://localhost:6380]

However I want to keep selinux by enforcing mode ! THX

7

Assuming the webinterface and the redis running on the same machine:
Your icingadb-redis instance is published on 127.0.0.1:6380 and icingadb-web tries to connect via localhost:6380, which is not necessarily the same.

Check the bind parameter in /etc/icingadb-redis/icingadb-redis.conf.
If it is set to 127.0.0.1 try changing the connection settings in the icingadb modules redis part to 127.0.0.1.

If your webinterface runs on a different machine than the redis, you need to change the bind paramter to e.g. 0.0.0.0 and then connect to the ip address of the host.

8

Thanks for reply, Yes it is running on the same host (everything, db, webserver…)

I tried but, nothing better (here screen shot attached)

Again, something somewhere with permissio / rignt (but where / what !?)

Failed to connect to primary Redis: Permission denied [tcp://127.0.0.1:6380]

image
image1542×855 43.9 KB

Redis conf file is set as :

sudo cat /etc/icingadb-redis/icingadb-redis.conf | grep 127.0.0.1
# bind 127.0.0.1 ::1              # listens on loopback IPv4 and IPv6
bind 127.0.0.1 -::1
# only accepts local connections from the IPv4 address (127.0.0.1), IPv6 address
#         IPv4 address (127.0.0.1), IPv6 address (::1) or Unix domain sockets.
9

There must be something wrong from either : selinux or redis
Did I setup a password somewhere which unallow access ?
But, there’s no Redis server installed on RHLE, Redis seems to be part of Icinga (plugin, modules,…)
I can’t get out of this maze,;…

10

If I remember correctly, Icinga has it’s own preconfigured Redis package.
We use Ansible to not have to deal with such problems.

11

Good to know about Icinga-redis. I keep this in mind… thanks

But how can I configure / modify / adapt it (if this is the problem) to get out of the maze, out of my current problem ?

12

I am still facing the issue…
I do not know were / what to modify.
Will perhaps redo all steps…

13

So, if I understood correctly, you are using “your own” redis?
Why not use the icingadb-redis package that should have come as an dependency when installing the icingadb package?

I don’t know, what difference there is between the icingadb-redis and a “not icinga” redis.

14

I use icingadb-redis :
/etc/icingadb-redis/icingadb-redis.conf
but don’t know why / how I’m blocked with it…

error :
Redis by icingadb is CRITICAL
Can’t connect to Redis: Permission denied [tcp://localhost:6380]

15

Okay what happens if you configure icingadb to connect to redis on 127.0.0.1:6380 instead of localhost:6380?
127.0.0.1 are not the same in terms of bind.

16

I did without success - thanks for reply

17

SO FAR I discovered progress :

YEs it is a Selinux issue - when set up to permissive it works now, but I don’t want to leave it as permissive, I want to keep it “enforcing” status.

What / how / where parameter should I set up ?

(I understand it is more a selinux problem - the doc says to also configure a icinga2-selinux - but so far didn’t find the doc on how to install / parameter it)

18

icinga2-selinux is the package name. Install it and see if that changes anything

19

I meant icinga2-selinux is installed but how to configure it ?

ie (with chcon,… ?)

example (for icingaweb2,… don’t know about my db problem) - I can see policy as following : (parhaps not the best example)

sudo ls -lZ /etc/icingaweb2/
total 20
-rw-rw----. 1 apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0      57 Dec  4 14:43 authentication.ini
-rw-rw----. 1 apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0     223 Dec  4 14:43 config.ini
drwxrws---. 2 root   icingaweb2 system_u:object_r:icingaweb2_config_t:s0     118 Nov 29 15:47 enabledModules
-rw-rw----. 1 apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0      57 Dec  4 14:43 groups.ini
drwxrws---. 5 root   icingaweb2 system_u:object_r:icingaweb2_config_t:s0      56 Dec  4 23:03 modules
-rw-r--r--. 1 apache icingaweb2 unconfined_u:object_r:icingaweb2_config_t:s0 584 Dec  4 23:20 resources.ini
-rw-rw----. 1 apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0      82 Dec  4 14:43 roles.ini
20

It was a Redis issue !

1 Like