Importing from multiple import sources

hi,

I am having some trouble configuring our import sources correctly.
we are using the vspheredb module and the x509 (Certificate plugin) and the fileshipper module.

i created a CSV from the data in our old monitoring software (opsview) and have imported that without problems. currently i have 2 issue’s:

  • some of the objects that have been imported are also picked up by the vspheredb and x509 module. but have different capitalisation so they are reported as new objects. but when i try import them i get an error: “Trying to recreate icinga_host” so even if i choose ignore as update policy i cannot import them.
    i tried to use the lowercase and uppercase modifier for the import job but they throw an error:
    Uncaught Error: Call to undefined function mb_strtolower() in /usr/share/icingaweb2/modules/director/library/Director/PropertyModifier/PropertyModifierLowercase.php:11

  • another thing i cannot figure out is how to get the different import jobs to work together. some object are picked up by both jobs. and they overwrite each other if i try to merge them. for example if the vsphere import runs i want it to add the host to a ‘vm’ host group. when the X509 job picks up that host i want it to also append it to a ‘certificate’ host group but i the last job overwrites the host group and is not appending to it. i tried to add custom variables to the import and using them as a condition to a one or more groups but couldn’t get it working.

can somebody give me some clues how to configure this?

we are running director 1.7.2
vspheredb 1.1.0
X509 1.0.0
fileshipper 1.1.0
the import from fileshipper was just a one time import the other 2 we would like to run on a schedule

any advice would be appreciated
thanks in advance

Hi,

about the modifiers.
Can you Show us how you create the modifier? In our Environment it works fine. We are using the same director Version.

about the data from many sources:
First of all you have to create the diffrent Import sources. The rest you have to do at the sync rules. For every property you have to choose the Import source, the coumn and the Destination at the host object. That’s all. Works very well for us

1 Like

Check if you have the mbstring extension for php installed (e.g. php-mbstring for Ubuntu/Debian).
If not, install it and restart the webserver and/or php.

1 Like

hi,

i just go to the import job > modifiers, enter a property, target property and select the modifier.
then i store the config and go to preview. there i get the error.

the reply of log1c also makes sense, i checked the module and it does not seem to be installed so i trying that now. i added a screenshot of the error.

regarding the import jobs, i configured the properties i want to import for every source as you said but if import job ‘A’ adds the object to group ‘A’ and import job ‘B’ adds the object to group ‘B’ i would expect the object to be in both group ‘A’ and ‘B’ but it seems that the last job allways sets its own group and removes the other

thanks @log1c installing the PHP module did indeed fix the issue with the modifier

Could you Show us a harcopy from the sync rule which makes the problem and to see what you mean/you want to have. Also the merge policy would be interesting. Do you choose one or more templates with the sync rule?

1 Like

i created a new simple test to illustrate my problem.
i created 2 csv files, one for VM’s and one for Certs. and added 1 vm object in the VM csv and 1 cert object in the cert csv. then i added the vm object to the cert csv and edited the group (hoping to append a second group to the object)

importing the vm csv and syncing it adds the host as expected, importing the cert csv adds the new cert object and modifies the vm object’s group from the vm group to the certificate group:

the import rules are practicly identical except the file they load:

same for the modifiers:

the sync rules are also practicly the same:

the sync property’s are also the same:

and this for the cert property’s:

i hope this helps to illustrate my issue. i am not sure what you mean by te templates with the sync rule
i also added the csv files

cert_test.csv (162 Bytes) VM_test.csv (106 Bytes)

i just noticed that i made a mistake with the names in the csv.
the vm object has the certificate group and the certificate object has the vm group.
and the cert csv has the same certificate object with the certificate group.

sorry for the confusion but it still shows the problem i am having

Maybe try this for a workaround:
With each import add a custom host variable to the host object and then assign the host group based on this variable.
But I guess you would have to have a column in your import source that has a value (e.g. “yes”) for this variable.

The problem is that the update policy “merge” favors the import source over the existing object, so the host group will always be replaced.

to add the custom variable do you mean in the import source or with the sync rule?
with the import source i do not see an option for that at first sight

i think have been trying with that at the sync rule but couldn’t get it to work immediatly. in the example i am using a csv file so the column there wouldn’t be much of an issue but in the ind i want to get this to work with the vspheredb and x509 db as import source so the extra column there would not be so easy i think

i will try again with the custom variable

The custom variable has to be added in the sync rule.
Maybe there is a column that has a static value that doesn’t change and you can use that as a value for the variable.

1 Like

@log1c could you maybe give an example on how to set and check that variable?

i am trying something like this:

which creates an object like this:
image

but now how should i check for it?

Switch the destination field to “custom variable” and the name the new variable:


Then set the source column.

After this you can go to the specific host group and assign it based on the variable and/or it’s content.
image
or
image

1 Like

@log1c awesome, that is working.
only instead of using a column for the source column i used a custom expression for the source column.
which i just set to yes and i check for that in the assign where property of the host group.

thanks a lot for the solution i was looking at it the wrong way.

Good to know that this works, thanks for the feedback :slight_smile:

I think the suggestion from @log1c is the best for you. Because I never saw a function to add individual entries to an array from diffrent import sources. You can only add a ready prepared array like
host;[“entryA”, “entryB”, “entryC”] in a CSV file.
but it would be a nice feature.

1 Like

yeah, but i just used the csv as an example to show the problem.
for my actual imports i am using the vspheredb and x509 module as import sources.
there i don’t have the option to add the extra columns. so adding the custom expression seems to do the trick for my problem.because i use different variable names it’s appending the different variables and allows me to check for them in the group ‘assign where’ option

If you want to use an array it would be difficult with two import sources. That mean merge two arrays from a CSV into one custom var in the director.

So what we do is to merge the CSV with the base data from diffrent sources in one CSV file.
And some addtional information like addtional IP adresses for check, we put in an extra csv file to add them into the specific array. After we can do things like this: https://github.com/Icinga/icingaweb2-module-director/blob/master/doc/14-Fields-example-interfaces-array.md and and this: https://github.com/Icinga/icingaweb2-module-director/blob/master/doc/15-Service-apply-for-example.md.
But for host groups maybe our use case is to complicated/costly for your use case.

1 Like

ok i see,

thanks for the advice. but indeed i think for our current needs just adding the custom variable to the import jobs is sufficient for now. i will keep it in mind if our needs should change