Script works locally but fails to work in icinga2

pgiorda · July 3, 2024, 4:36pm

I have a perl script to test for file existence on an AIX system. The script works correctly when I run it in the shell, but when I use it in Icinga it always returns “OK” when the file actually is in existance, and OK when not.

Any help - I’ve been over this too long, and am obviously missing something.

Example:
works:

root@icinga:/usr/lib/nagios/plugins# ./check_file_exists -H grsprod1 -p /usr/local/scripts/logs/showstbygap.log
CRITICAL: Found file /usr/local/scripts/logs/showstbygap.log |

works:

root@icinga:/usr/lib/nagios/plugins# ./check_file_exists -H grsprod1 -p /usr/local/scripts/logs/showstbygapa.log
OK: File /usr/local/scripts/logs/showstbygapa.log does not exist |
root@icinga:/usr/lib/nagios/plugins#

fails:

Icinga Details

Version used
2.14.0-1
Operating System and version
Ubuntu 22.04.3
Enabled features (icinga2 feature list)

Disabled features: compatlog debuglog elasticsearch gelf influxdb influxdb2 journald livestatus opentsdb perfdata statusdata
Enabled features: api checker command graphite icingadb ido-mysql mainlog notification syslog

Icinga Web 2 version and modules (System - About)
2.12.0
Config validation (icinga2 daemon -C)

root@icinga:/usr/lib/nagios/plugins# icinga2 daemon -C
[2024-07-03 11:15:50 -0500] information/cli: Icinga application loader (version: r2.14.0-1)
[2024-07-03 11:15:50 -0500] information/cli: Loading configuration file(s).
[2024-07-03 11:15:50 -0500] information/ConfigItem: Committing config item(s).
[2024-07-03 11:15:50 -0500] information/ApiListener: My API identity: icinga
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/conf.d/notifications.conf: 23:1-23:48) for type 'Notification' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'notify-service-by-email_level1-helpdesk' (in /var/lib/icinga2/api/packages/director/71c37371-db05-45e9-a652-7c185f475818/zones.d/icinga/notification_apply.conf: 51:1-51:71) for type 'Notification' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'notify-service-by-email_level1_database' (in /var/lib/icinga2/api/packages/director/71c37371-db05-45e9-a652-7c185f475818/zones.d/icinga/notification_apply.conf: 73:1-73:71) for type 'Notification' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/conf.d/notifications.conf: 11:1-11:45) for type 'Notification' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'backup-downtime' (in /etc/icinga2/conf.d/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'apt' (in /etc/icinga2/conf.d/apt.conf: 1:0-1:18) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'ping6' (in /etc/icinga2/conf.d/services.conf: 34:1-34:21) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'ssh' (in /etc/icinga2/conf.d/services.conf: 47:1-47:19) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 57:1-57:65) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 65:1-65:53) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'icinga' (in /etc/icinga2/conf.d/services.conf: 73:1-73:22) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'load' (in /etc/icinga2/conf.d/services.conf: 81:1-81:20) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'procs' (in /etc/icinga2/conf.d/services.conf: 92:1-92:21) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'users' (in /etc/icinga2/conf.d/services.conf: 100:1-100:21) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'windows check memory' (in /var/lib/icinga2/api/packages/director/71c37371-db05-45e9-a652-7c185f475818/zones.d/director-global/service_apply.conf: 11:1-11:36) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'windows check disk D' (in /var/lib/icinga2/api/packages/director/71c37371-db05-45e9-a652-7c185f475818/zones.d/director-global/service_apply.conf: 31:1-31:36) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] warning/ApplyRule: Apply rule 'windows check disk E' (in /var/lib/icinga2/api/packages/director/71c37371-db05-45e9-a652-7c185f475818/zones.d/director-global/service_apply.conf: 42:1-42:36) for type 'Service' does not match anywhere!
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 6 NotificationCommands.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 3002 Notifications.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 IcingaApplication.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 269 Hosts.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 48 HostGroups.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 Downtime.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 SyslogLogger.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 IcingaDB.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 2 Comments.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 FileLogger.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 3 Zones.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 CheckerComponent.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 Endpoint.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 3 ApiUsers.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 NotificationComponent.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 ApiListener.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 272 CheckCommands.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 9 TimePeriods.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 6 UserGroups.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 17 Users.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 1287 Services.
[2024-07-03 11:15:51 -0500] information/ConfigItem: Instantiated 4 ServiceGroups.
[2024-07-03 11:15:51 -0500] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2024-07-03 11:15:51 -0500] information/cli: Finished validating the configuration file(s).

Script:

#! /usr/bin/perl -w
#
#
# License Information:
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
#
############################################################################

use POSIX;
use strict;
use Getopt::Long;
use vars qw($host $opt_V $opt_h $opt_v $verbose $PROGNAME $pattern $opt_p $opt_t $opt_H $status $state $msg $msg_q $CMD $RMTOS);
use lib  "/usr/local/nagios/libexec" ;
use utils qw(%ERRORS &print_revision &support &usage );

sub print_help ();
sub print_usage ();
sub process_arguments ();

$ENV{'PATH'}='';
$ENV{'BASH_ENV'}='';
$ENV{'ENV'}='';
$PROGNAME = "check_file_exists";

Getopt::Long::Configure('bundling');
$status = process_arguments();

if ($status){
        print "ERROR: processing arguments\n";
        exit $ERRORS{"UNKNOWN"};
}

$SIG{'ALRM'} = sub {
        print ("ERROR: timed out waiting for $CMD on $host\n");
        exit $ERRORS{"WARNING"};
};

$host = $opt_H;
$pattern = $opt_p;

alarm($opt_t);

$CMD = "/opt/freeware/bin/stat " . $pattern . " 2>/dev/null | /usr/bin/grep -v stat | /usr/bin/wc -l ";
## get cmd output from remote system

if (! open (OUTPUT, "/usr/bin/ssh -q -i '/etc/icinga2/id_gersdsa' -o HostkeyAlgorithms='+ssh-dss,ssh-rsa' -o PubkeyAcceptedAlgorithms='+ssh-dss,ssh-rsa' -o KexAlgorithms='+diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1' $host $CMD| " ) ) {
        print "ERROR: could not open $CMD on $host\n";
        exit $ERRORS{'UNKNOWN'};
}

my $perfdata = "";
my $state = "-1";

# only first line is relevant in this iteration.
while (<OUTPUT>) {
        print $_ if $verbose;
                chomp;
                my $val = $_;
                if ($val < 1) {
                        $state = $ERRORS{'OK'};
                        $msg = "OK: File $pattern does not exist";
                } else {
                        $state = $ERRORS{'CRITICAL'};
                        $msg = "CRITICAL: Found file $pattern";
                }
}

close (OUTPUT);

# Perfdata support
print "$msg | $perfdata\n";
exit $state;


#####################################
#### subs


sub process_arguments(){
        GetOptions
                ("V"   => \$opt_V, "version"    => \$opt_V,
                 "v"   => \$opt_v, "verbose"    => \$opt_v,
                 "h"   => \$opt_h, "help"       => \$opt_h,
                 "p=s" => \$opt_p, "pattern=s"  => \$opt_p,
                 "t=i" => \$opt_t, "timeout=i"  => \$opt_t,
                 "H=s" => \$opt_H, "hostname=s" => \$opt_H
                 );

        if ($opt_V) {
                print_revision($PROGNAME,'$Revision: 1.0 $ ');
                exit $ERRORS{'OK'};
        }

        if ($opt_h) {
                print_help();
                exit $ERRORS{'OK'};
        }

        if (defined $opt_v ){
                $verbose = $opt_v;
        }

        unless (defined $opt_t) {
                $opt_t = $utils::TIMEOUT ;      # default timeout
        }

        unless (defined $opt_H) {
                print_usage();
                exit $ERRORS{'UNKNOWN'};
        }

        return $ERRORS{'OK'};
}

sub print_usage () {
        print "Usage: $PROGNAME -H <hostname> -p <directory> -t <timeout>] [-v verbose]\n";
}

sub print_help () {
        print_revision($PROGNAME,'$Revision: 0.1 $');
        print_usage();
        print "\n";
        print "   Checks the amount of files older than specified on remote system over SSH or RSH\n";
        print "-H (--hostname)  = remote server name (required)";
        print "-p (--pattern)   = Directory pattern for find command\n";
        print "-t (--timeout)   = Plugin timeout in seconds (default = $utils::TIMEOUT)\n";
        print "-h (--help)\n";
        print "-V (--version)\n";
        print "-v (--verbose)   = debugging output\n";
        print "\n\n";
        support();
}

rivad · July 3, 2024, 5:20pm

What is the exit code if called in the shell?

echo $?

moreamazingnick · July 3, 2024, 6:28pm

icinga runs as the user nagios on ubuntu, check the file permissions…

you can also test using:

sudo -u nagion /path/check_file_exists -H grsprod1 -p /usr/local/scripts/logs/showstbygapa.log

pgiorda · July 5, 2024, 11:04am

The exit code from the script is “2”.

pgiorda · July 5, 2024, 12:06pm

Well, color me embarrassed. So I set up the nagios user to have ssh key access and dropped the key that I had in the command - and voila. It works. Thanks Nick for pointing me in the right direction.

lorenz · July 8, 2024, 4:17pm

Wonderful mouse cursor btw.