Hello @powelleb,
thanks for the feedback, the only problem with Icingaweb in this configuration is that: icingaweb2/monitoring/health/info will show you only the information about first satellite zone connected to the database, icingaweb unfortunately reads only first id from table in the database and show statistic based on that. But the good news here that all hosts are visible and doesn’t have any conflicts inside database
Also i’ve tested icinga-agent move from one satellite zone to another, all works like a charm and it doesn’t make conflicts in database.
When everything is connected Icingaweb will read a single database and it makes queries based on host/service id + environmentid
Database: with that amount of clients take care to create daily partitions for tables like history and notifications they will grow with time, default database cleaner use delete methods and its fairly slow for this scale, with partitions you will be able to drop unnecessary data fast and without service interruptions.
Also there is 2.12RC available and soon(fingers-crossed) we will get a 2.12 released, 2.12 use redis with redis-streams which is really fast, so i hope it would be enough to have 2 baremetal masters that would handle all load, on the other side, for infinitive scaling current concept will not work in 2.12 with icingadb, at least i hope they will add support for this kind of setup later(modern micro-service architectures doesn’t really respect monsters like that and for simple management/update/upgrade all used to have a lot of small VMs, also it much more failure tolerant)
P.S. in your setup you have Level 3 with additional satellites, which makes extreme overhead in terms of data processing and management:
agent send data => Satellite L3 handle connection + transfer data => Satellite L2 transfer data + push data to the database => Master accepts data and drop it
I don’t see a real reason to use L3, i would just increase amount of satellites on L2
With best regards,
Dmitriy