I was trying to find some capacity planning GYAN on the icinga docs, but not able to get to exactly what I am looking at. I know there is no thumb rule on capacity planning and all is dependent and little vague.
I had long discussion on this topic with Thilo during our training - but honestly I am still at dark.
I am running 3 very large environment at 3 geographies (continents). I have been asked time and again by management to save cost why don’t I have 1 pair of master / DB / Web and use satellite at the respective continent DC. I have skipped the question by always giving excuse of Data Privacy Law / Practice (China US etc etc) But some day I will get caught of giving run around . I need to have a solid figure that with a pair of Master I can run X amount of Nodes (Agents). Also I can run Y amount of checks (local or remote) etc The good part is I have always made the agents talk to satellite only and not directly to master. But still I am little skeptical to control the whole world (~30k node) with a pair of Master. Leaving aside the network latency. Please correct if my thoughts are in wrong direction
Q2: At one time I know I have to build another pair of satellite for monitoring more servers. What is the thumb rule or indicators that will tell me that I need more satellite. Again lets say I have ~2500 servers hosted on a pair of satellite. Only host alive checks are running from satellite. Rest all are agent based check (run on agent). Considering this how far I can push my luck ? Do I say after 3000 nodes I should stop. Should I say 4000 Agents is the limit? How do I know that my satellites are not able to take any more? I have already run into the following issue and have followed the documentation to resolve it. Question here is how far I can push my luck?