Check_nwc_health performance

I’m struggling to scale out check_nwc_health for pulling back switch interface usage, errors, and duplex, on 48 port switches.
I’ve deployed a satellite node to offload checks from my Master, however I’m running into issues with cpu load.

My checks appear to work without issue and are as follows;

'/usr/lib/nagios/plugins/check_nwc_health' '--community' 'xxxx' '--hostname' '192.x.x.x' '--mode' 'interface-usage' '--name' '1' '--statefilesdir' '/var/tmp'
'/usr/lib/nagios/plugins/check_nwc_health' '--community' 'xxxx' '--hostname' '192.x.x.x' '--mode' 'interface-duplex' '--name' '1' '--statefilesdir' '/var/tmp'
'/usr/lib/nagios/plugins/check_nwc_health' '--community' 'xxxx' '--hostname' '192.x.x.x' '--mode' 'interface-errors' '--name' '1' '--statefilesdir' '/var/tmp'

The frequency for each is, note I have made these values longer trying to improve performance;

Errors: 24000s

The Timeout on all 3 is 20s.

My host is configured with an array for each interface, approximately 48 per switch.

When I start Icinga or redeploy I see situations where hundreds of processes are created and the whole thing grinds to a halt. I suffer timeouts and things seem to backup I’ve tested the command line against each switch and they respond instantly.

My satellite is running 2 cores with 4GB, on a VM,

Is there anything I can do further optimise this?

I’ve now increased the resources on the satellite VM to 4 cores and 12GB, it’s certainly helped, but seems a little excessive. I’d be interested to know if there is a better way of achieving the same result.

I’ve also got a switch with over a hundred interfaces, when using check_nwc_health to query int usages and duplex, my box grinds to a halt. Throwing excessive cpu and ram at the problem is great and we can’t really spare it either. Is there a better option?


what you describe is like “work as design” and how Icinga, the check_nwc_health script, interpreter languages etc. works.
First of all if you look into the script you will see it’s a perl script with 77391 lines. A lot of!
Depending how many switches you want to check you fire this script with this many lines against the switches - 3 times! (for every mode one time).So not very efficient on large environments.

if icinga triggers the start of the check parallel - maybe after a (re)start of the daemon - there will be a lot of processes in the task manager.

Simplified calculation to make this clear:
In our setup the script needs 0,2 % from memory and 3,6% cpu, the execution time is about 0,3 to 1 sec (depending on switch and what to check) - not much if you read this. But let’s say you have 1000 switches to check and you want the script with 3 diffrent modes like in your example. 0,2 %mem x 1000 switches x 3 modes = 600 % memory. Ok I know this calculation is not 100% correct, because there are some more influences than this simple calculation, like a delay for running the processes, but it should show you how many ressources the checks need. In this thread of I found this command:

pmap $(ps -ef | grep | grep -v grep | awk ‘{print $2}’)

With this command I see that if perl excute the script and has to load a lot of modules it need about 212092K to check the cpu-load on our switches!

You can observe this if you stop the icinga daemon, start a second shell with a task manager like (h)top, wait and start the daemon again. You will see in the task manager how many proccesses icinga has to start and the cpu load and memory of your server is rising. Similar behavior after after a deployment or if you have to restart your icinga server. Here did the devs a good job so icinga has some mechanism inside, that icinga know what is to check now. (see Technical Concepts - Icinga 2 or Technical Concepts - Icinga 2) .

What can you do?

  • Create a wrapper script to add another artificial delay, like

sleep echo $RANDOM % 3 + 1 | bc
/usr/lib64/nagios/plugins/ $*
exit $?

  • cluster your satellites so that the load is shared a little. In addition, you get a reliability.

  • Use another check which saves resources. Instead of check_nwc for interfaces we for example are using this one: Icinga Template Library - Icinga 2, You can get the check from here It’s a C program. And it’s much faster. e.g. in our setup this program needs 0.7 % cpu and 0.0%mem and the execution time is about 0:00:03 sec

I hope this explanation makes things clearer


You could also try limiting the maximum concurrent checks of Icinga 2.
You can set a specific attribute in constants.conf do change the default number:

1 Like

Thanks for the detail. You’re input is certainly aligned with my experience. I’d prefer to go with the simpler solution, as it should be easier for others to maintain long term. I’m therefore going to look at the check interfaces option.

1 Like

wow check_interfaces is so much faster than check_nwc_health. That certainly solves the performance issues.

Only thing is, I can’t seem to get bytes/s in and out from the metrics.
The docs mention -p

-p|--perfdata      last check perfdata
                        Performance data from previous check (used to calculate traffic)

Adding that to a command line test doesn’t generate what I need, does it produce the sort of metric I’m after?

We don’t have to use this parameter. We get the performance data by default. E.g. this is for one interface:

Grafana does the rest for us.

thanks could I ask what your command looks like?

‘/usr/lib64/nagios/plugins/check_interfaces’ ‘–aliases’ ‘–auth-phrase’ ‘???’ ‘–auth-proto’ ‘???’ ‘–hostname’ ‘< IP/DNS Name >’ ‘–if-names’ ‘–priv-phrase’ ‘??’ ‘–priv-proto’ ‘??’ ‘–regex’ ‘< searchregex as regex >’ ‘–timeout’ ‘60000’ ‘–user’ ‘< username >’

Thanks that’s basically what i’m using, but still having a few issues, think I’ll start another thread specifically on the check

1 Like

hi @stevie-sy

I understand correctly that you count the utilization of interfaces in Grafana?
and event notifications are also sent through it ?

Icinga not used it process?

Hi @orsa!

Yes and no. We use “check_nwc_health” and/or “check_interfaces” to check the interfaces from our switches.

it is how Icinga works:
Icinga triggers the checks (scripts). The return value of the used script is in the case of the mentioned scripts besides up/down also performance data. They will be shipped into Grafana.
Like every other used check (script)

Since the check_nwc_health plugin is written in perl and perl is an interpreter language, the perl interpreter is executed every time the plugin is run which causes a lot of CPU and IO.

We had the same problem in our setup with many network devices.

There is a newer project called “Thola” which is written in Go, is very fast and requires few system resources.

Please have a look here and try it:


I’m very excited to try this :slight_smile: we monitor ~3,000 network devices at ~80 sites and we’ve had to significantly increase the resources allocated to our satellites since using check_nwc_health for interface monitoring.

Here the same, we will try the new check the next time. We have also a lot of network devices to check and the mix from check_interfaces and check_nwc_health makes some troubles at some locations and hardware types

I am interested in your feedback. If something is still missing or not working, it’s best to open an issue on GitHub.

It may well be that vendors / models are missing. But these can be added independently via YAML config file: Writing a device class | Thola Documentation
Please upload the YAML config files via PullRequest on GitHub.
Alternatively, you can also provide an SNMP record file and we add the device.

1 Like

I’ve had a chance to test this evening and so far it looks great, and we will definitely make use of the features.
One of the reasons we make use of the check_nwc_health plugin is it has the cability to calculate interface saturation as a percentage.

Although we are writing our perfdata to InfluxDB and OpenTSDB (and we can graph utilisation from there), it is still useful for us to have the check exit with Warning or Critical when an interface becomes fully saturated.

I believe check_nwc_health does this by storing the previous values in a temporary file, and calculating the delta, and dividing by the time between the checks.

Is this something you would likely look at implementing? I understand that it may be compute intensive to do all of those calculations, hence the reason why check_nwc_health is so heavy on resources.

We are also writing our perfdata to an InfluxDB cluster.

That’s why the capability to calculate interface percentage is not so important for us.
But we plan to add the capability later this years (Q3 2021).

Thola already has a database backend (using SQLlite or Redis) to store cached / historical values.
So it’s easy to add such a feature.

Currently support of telnet / SSH protocols is more important for our own purposes.

If you like you can write us an email to to discuss in more detail.

1 Like