Better Dynamic Monitoring for "Difficult" Fortigate Devices?

I work for an ISP in a mixed vendor environment. A couple of years back we went heavy into Fortinet (fortigates) for our new backbone devices.

Some of these devices (things like the Fortigate 7030e) can have multiple nodes, each with their own slot. Each slot (let’s say that there are 16 slots total in an given device like this) has it’s own SNMP port.

To solve the problem for CPU/Memory monitoring, I have setup 32 apply rules in Icinga Director (16 for CPU and 16 for Memory) that applies a Centreon command to check for Fortigate CPU or Memory respectively.

That’s a lot of apply rules to manage, and if a model gets added, or we rely on a different variable, I have to touch each and every apply rule.

My question is:
Is there a better way that I am not thinking of to handle these apply rules, and maybe be more dynamic? Can be via director, API, or just about anything else.

Is anyone out there handling these types of devices differently that might be easier/better?

Not sure if there’s going to be a “right” answer to this, or any at all but I thought it would be worth to give it a shot and see if others are handling this problem in a different way :slight_smile:

Hi,

first of all I’m a fan of ServiceSets in the director. If one or a couple of servers have to get more than one Service a bundle them into a ServiceSet. I think it makes things a little tidier. That is the first thing you could do.

The next idea would be to save the oid to check in a custom var in the host object. In your service definition you acces it with a macro.
This works fine for us. Because we have two storage boxes from the same manufacturer, but diffrent generation. And of course is the oid diffrent … I hate that also :man_facepalming:

ServiceSets may be worth looking into – admittingly I don’t know much about them.

The problem with this isn’t so much the OID as it is the differing SNMP port numbers. We get around this by relying on the Fortigate model as a custom var – if it matches one of a few different model numbers that we support it gets checks applied like:

I found this guide on service sets, surely enough, and merits more looking into.

Could you show us the host object as well some of your service apply to?

BTW: The director renders the service set as service apply rule. So it’s for better grouping the rules and this gives a better overview in the UI

I could, but before I expose a bunch of stuff on the host that may not make sense to you, is there something you’re looking for?

All of the vars we use for a given device comes from an internal database where Network Engineers put in a device (hostname, customer, etc…) and then the rest of the information is polled via SNMP (make/model, bgp peers, interfaces, etc…)

Icinga consumes a good number of these vars (like a customer ID, make/model, bgp peers, SNMP info).

I want to know how you created your services to get an idea how to optimize that like your question in your inital post :wink:

We are doing a lot of automation importing data from our CMDB and other databases - with the director.

Same as above – any information we need is ported from a db via director. Nearly all services (such as the ones in this question) are based on the make and/or model field (in the case of the services I am inquiring about screen shot, it’s model host.vars.ops2_model with ops2 being the name of the database).

To be clear – nothing is broken, but with the possibility of me adding a “model” to 32 services in the near future (~6 months), I thought I would ask.

Would it be an option to define multiple host templates, one for each model, and configure the options (snmp port, community whatever) inside the template?
If the host objects are created via Import&Sync as well you then could assign the template base on the value in the model column.
Then you would only need one service set containing both the services and configure them to look into the hosts variables.

If this is not an option I would go for service sets nevertheless, because this would halve the objects you have to configure (if I calculated correctly :sweat_smile: ). In the end it will be the same config, as the services inside a service set will be broken up into single apply rules

Service set example
zones.d/director-global/servicesets.conf
/**
 * Service Set: dell-idrac-checks
 * 
 * assign where "dell-server" in host.templates
 */

apply Service "bmc-status" {
    import "idrac_health_bmc-status"

    assign where "dell-server" in host.templates
    vars.notification_period = "8x5"
    vars.service_owner = "VMware"

    import DirectorOverrideTemplate
}

apply Service "fan-status" {
    import "idrac_health_fan-status"

    assign where "dell-server" in host.templates
    vars.notification_period = "8x5"
    vars.service_owner = "VMware"

    import DirectorOverrideTemplate
}
...

Host templates may very well be a viable solution. It would take a rehaul of how we do imports (we do a mass import from the db regardless of device type, so we need to have the sync properties determine which host template to use).

Probably a combination of the 2 (service sets and host templates) would be best for long term management.

At any rate, this probably isn’t going to be something I get throughput for to complete any time soon – Jan - Oct are busy months for my side of our industry.