How to build your own icingaweb2 packages for CentOS 8 as the security fix was not released for it

If you did carefully read https://icinga.com/blog/2022/02/16/announcing-official-icinga-packages-for-rhel-amazon-linux-2-and-sles/ you may recognized there are currently no packages build for RHEL 8 derivates like CentOS Stream 8, Alma or Rocky Linux.

When you now read the security announcement https://icinga.com/blog/2022/03/08/releasing-icinga-web-v2-8-6-and-v2-9-6/ and do not see any package for in the centos/8 directory, you may recognize you have a problem.

But it is quiet easy to build the packages your own until Icinga decides to do this again for you.

On my Fedora 35 laptop I only need to commands:

wget -P /tmp/ https://packages.icinga.com/fedora/34/release/src/icingaweb2/icingaweb2-2.9.6-1.fc34.icinga.src.rpm
mock -r centos-stream+epel-8-x86_64 /tmp/icingaweb2-2.9.6-1.fc34.icinga.src.rpm 

This will build the packages for you based on the same source (downloaded with wget from packages.icinga.com) for the system you need (in my case CentOS Stream 8 with EPEL enabled, but I could run this also for others if I wanted to ensure they are build exactly with the system I am using and do not relay on API/ABI compatibilty)!

So please do not leave your systems vulnerable!

6 Likes

There are also none for RHEL8.

Since we changed the structure around building those repositories recently, we accidentally didnā€™t include them yesterday.
Sincerest Apologies!
They are up and running now, so please update your systems!

But this is only true for RHEL8 were a subscription is needed, so only addresses the post of @rsx.

1 Like

On this subject, does anyone know if there are any plans by any entities to maintain an el8 compatible repository for Stream, or the other clones (Alma, Rocky, etc) that are out there? Or is the subscription repository going to be the best (albeit expensive) way to go on this?

I see Index of /centos/8/release/noarch/icingaweb2 does not contain 2.9.6. (side note, a little odd to me that the repo rpm provides a url that requires a redirect instead of the real url.)

Should any RPMs built against RHEL also just work with Alma, Rocky, etc? Are those (source) RPMs available publicly from Icinga? Or are those the ones under fedora?

To answer the question of both of you:

From the blogpost linked above ā€œWeā€™re monitoring the situation to decide if we can justify the efforts to build packages for those operating systems in the future. Our decision process is mainly driven by the adoption rate and requests from users and customers.ā€ So I do not know anything else (just working for a Icinga partner), but the blogpost sounds like a ā€œcould happenā€ and more like using an alternative than using CentOS Stream.

If the subscription repository is your way to go, I can not decide. If you want to have the promise of Icinga that it will definitely work on RHEL because the packages are build on it, then yes. If you relied on the API/ABI compatibility of RHEL and CentOS replicating it, then packages build on any of the different derivates should be fine.

This brings me to if it should work, which I would answer as an experienced user and packager with yes. The API/ABI compatibility promise of RHEL is very good and there are really a very low number of exceptions. I can remember one time the SELinux subpackage had a problem being build with not the latest updates of the default policy missing one permission and by this causing an error. Another time Icinga 2 was hit by a security fix, but this one broke many more software. With CentOS Linux there was a rebuild with about two to three weeks delay and most people took compatibility for guaranteed which was perhaps a bit high of an assumption, but mostly it worked fine. Now we will have alternatives which do the same, but we also have CentOS Stream which will be the upstream for RHEL so it is before the schedule and perhaps a bit less stable as there is not Red Hatā€™s full QA process and certification system before the release. So building with CentOS Stream gives you the advantage of being upstream and so recognizing problems earlier and the disadvantage that a package could be incompatible until its dependencies hit the downstream. Building with RHEL gives you the middle ground and with the alternatives you can be sure all dependencies are available everywhere but could be already updated in upstream so being incompatible again. But the risk is quite low as stated earlier.

Source packages for Icinga are all the same as it uses the same tarball and the same spec just with different conditions for specific versions. This is why I used the one from Fedora 34 as the first one I found with none being published for CentOS 8 or Fedora 35 at the time of testing.

Just to make sure the compatibility or better the incompatibility between Fedora and RHEL derivates is clear. Fedora is a bit of a rolling release and much more current versions of software than stabilized enterprise RHEL derivates (why I happily use it as my client for about 16 years now, but not on any servers). RHEL or now CentOS Stream will use experience gained and software versions from one Fedora version to stabilize by taking some older software versions or introducing newer if required and then after release it more or less only backporting patches to the version released. By this rebuilding packages from Fedora source works most of the time, but chances that something will not work any more increase over time. But in this case it does not matter as Icinga is using the same source for all RPM builds.

And on your side note, this is a result of having changed it from el to centos with the release of the subscription repositories, but not updating the release rpm to reflect this or if they did this in general not before stopping to build for derivates.

There are still no icingaweb2-2.9.6 packages for RHEL8.
Looking into the github repo for the icinga-packaging shows failed/canceled pipelines:
image

Where are you looking? subscription packages for RHEL8 are available, as far as I can see.

I have looked here: https://packages.icinga.com ā†’ epel ā†’ 8 ā†’ release ā†’ noarch
But I just noticed that this is just a link to Index of /centos/8ā€¦
Also I missed the blog post about the paid subscription :exploding_head:

We have just recently migrated our servers from CentOS8 to CentOS Stream.
Now I read that there may be no more packages for CentOS Stream in the future.
What does this mean for me? How long can I expect support for Stream? Will I have to pay for Stream packages or will they be discontinued?
We would absolutely need support for CentOS Stream!

3 Likes

Is there a clear/simple table of which distros have packages provided by Netways/Icinga (free vs subscription)? The directory listing at packages.icinga.com is not necessarily clear or consistent and the future seems ambiguous at best.

They are already discontinued.

Itā€™s visible rather clearly on the website Iā€™d say. A general rule however is: Free distribution, Free Icinga packages

1 Like

CentOS 8 Stream falls under

in my mind, but you also stated in the same post that

Hopefully Iā€™m missing something, but it does not seem very clear to me.

2 Likes

I think some people have misunderstood the situation.
According to official FAQ:

Updates for the CentOS Stream 8 distribution continue through the full RHEL support phase.

That full support ends May 31, 2024.

So Stream-8 and Stream-9 will co-exist for some time and will be updated.

Cheers!

1 Like

We provide a full list of all operating systems we create packages for on Icinga Ā» Download and additional information about supported OS versions are located listed at Icinga Ā» Subscription Ā» Support Matrix

Currently we do not provide packages for CentOS Stream but for CentOS Linux (version 7 only, version 8 is EOL). Packages for CentOS Stream did not disappear, we never had them in the first place.

1 Like

I totally understand that you are not willing to support Centos Stream editions since it is a fast moving target. What I would like to know, though, is:

  1. Are there no plans for future inclusion of Centos Stream in your build pipelines?
  2. If the above is true, can we at least have the src rpm for RHEL 8 (instead of trying to build on fedora 34 which will also soon disappear)? I seem to be unable to find such an rpm but I maybe need more coffee!

Cheers

From the blog post:

There are very promising alternatives to CentOS Linux arising. Rocky Linux and AlmaLinux are getting more and more attention. Weā€™re monitoring the situation to decide if we can justify the efforts to build packages for those operating systems in the future. Our decision process is mainly driven by the adoption rate and requests from users and customers.

RHEL 8 packages are only available to our customers.

Monitoring, no pun intended, the situation isnā€™t a very strong answer to the original question regarding CentOS Stream. I would also not equate Rocky or Alma with CentOS Stream, as they are on the other side of Red Hatā€™s pipeline.

In my view, CentOS Stream is basically just another step between what Fedora to Red Hat used to be. I would assume it to be more stable / less turbulent than Fedoraā€¦but I could be wrong. I welcome any info from people with more build/packaging experience, or specific to the recent changes in Red Hat.

Fedora is ā€˜supportedā€™, so Iā€™m not sure how much faster a moving target CentOS Stream would be.

1 Like

After reading this thread and the blog post many times and some internal discussion in our company here some thoughts:

I think a lot of companies, like us, choose CentOS Linux for the server farm because of the binary compatibility to RHEL and also to have a cost-effective and stable infrastructure. And if the installed software needs RHEL, ok youā€™ll pay the licence for it. But than came IBM und everybody knows the story like to change the strategy behind CentOS etc.

We understand that it is/could be a little bit complicated with an upstream OS to get a full compatibility with your own software with every OS update (incl. library etc). And yes, since CentOS Stream is realeased CentOS Linux 8 is EOL. But as @gkoutsog already mentioned CentOS Stream 8 is EOL May 31, 2024. And as @leeclemens wrote is CentOS Stream between RHEL and Fedora.

We could install Icinga on CentOS Stream because of using the CentOS Linux 8 repo and it works (for us). But if you discontinue it, itā€™s like you cut off every CentOS user from the world arround Icinga.

After some answers here, for us there are still open questions:

  • In your blog post you only mentioned the replacement of CentOS Linux to stream. And some alternatives. But as @leeclemens wrote, itā€™s another piplineā€¦ In the blog post is no notice about the future plans about CentOS Stream. In yesterdays post from @bsheqa here, we read that there are no future plans about not providing packages for Stream

  • @dgoetz wrote a way to install Icinga updates on CentOS Stream. But is this a safe way to go in future with CentOS Stream? Because on the Support-Matrix @bsheqa postetd yesterday, this would only works/supported for agents setups, not for a the master/satellite-server. Fedora is only on the list for agents! And in an world of automation itā€™s the question if this is the way to go.

  • If we choose the subcription for RHEL, would it work for CentOS Stream? If I read this

it would, but it sounds not like a definitly. Ok, adjustments for SELinux we can do on our own. We had to do this many times for our self-written checks etc.

  • What do you recommend how to use Icinga on CentOS Stream on hundreds (or maybe like some other community members up to thousands of servers)?
    Should everybody change the OS because using Icinga while the actual installed main software ā€“ which should be monitored ā€“ works properly on Stream?

  • Please think about that, that maybe some of our bosses will decide to look for an other monitoring solutions which are working fine with CentOS Stream, instead of changing the OS for the complete server farm, only because of Icinga.

  • What we also canā€™t find (in the docs as well in the blog post) is an information how the maintenance of the mentioned community repos will work and if we find packages there for stream.

  • If we would choose a support contract from an Icinga partner, would it help? Because in generally you donā€™t support CentOS Stream.

We hope, we can get some more (official) information to bring more light in the dark. Thank you!

2 Likes