Rocky Linux Review

October 1, 2022 546 0

Rocky Linux steps into the breach left by CentOS with a community-based alternative to RHEL. By Bruce Byfield. Read our Rocky Linux Review.

Check Amazon

Red Hat Linux bought CentOS in 2014, but largely ignored it for years. Essentially, CentOS was the community version of Red Hat Enterprise Linux (RHEL), both deriving from Fedora. Then, in December 2020, Red Hat announced that CentOS would be discontinued, and it would be replaced by CentOS Stream. Within days, Rocky Linux (Figure 1), named for CentOS cofounder Rocky McGaugh, was announced [1]. Four months later, Rocky Linux released its first version. Like CentOS, it offers a community-based alternative to Red Hat.

Recently, I sent questions about Rocky’s current state to Brian Clemens, the project manager at the Rocky Enterprise Software Foundation (RESF), who got the development team to collaborate on answers. Here’s what the RESF development team has to say about Rocky Linux.

Figure 1: Rocky Linux is one of the main CentOS replacements.

Linux Magazine (LM): What were the relations between Red Hat and CentOS leading up to the discontinuation of CentOS?

Rocky Linux (RL): There were no formal relations or communication between Red Hat, CentOS, and future members of the Rocky Linux project/RESF prior to the discontinuation of CentOS. We were all surprised. Gregory Kurtzer, one of the original CentOS founders, announced an effort to build a replacement project within hours of the “CentOS Project shifts focus” post.

LM: What were the goals behind starting Rocky Linux?

RL: The primary goal of the Rocky Linux project is to produce a free, community-driven, bug-for-bug rebuild of RHEL. Imperatively, we are dedicated to operating in an open, transparent, and reproducible manner that embraces community participation and contribution. We are committed to enriching the entire Enterprise Linux ecosystem with open build processes and tools, beyond just adding another distribution to the mix.

LM: How does Rocky Linux compare to CentOS and to other spin-off Enterprise Linux distributions in terms of goals, features, and popularity?

RL: Aside from our previously mentioned goals, a lot of effort was put into ensuring that what happened to CentOS would not happen to Rocky Linux. To do this, we had to build a resilient organizational structure with proper checks and balances, as well as a provider-agnostic infrastructure that doesn’t depend on the whims of a partner or sponsor.

The infrastructure for Rocky Linux was built from scratch. Special care was taken to ensure no opportunity for control by an individual or company. All the development requirements we leverage are open source and freely available. We refuse to release any software unless it can be reproduced with free and open tools and processes. We leverage technologies such as distributed key generation to achieve community ownership and control of our security

Table 1: Rocky Linux 9 ISO Downloads by Date

Date	ISOs	Torrents
14/Jul/2022	4,899	1,034
15/Jul/2022	21,896	2,496
16/Jul/2022	9,036	1,407
17/Jul/2022	7,176	1,172
18/Jul/2022	11,719	1,448
19/Jul/2022	22,892	1,307
20/Jul/2022	11,382	1,241
21/Jul/2022	9,092	984

94,227 downloads of Rocky Linux 9 as of 2022-07-21

1,000,729 downloads of Rocky Linux 8 as of 2022-07-21

Table 2: Community Statistics

7,763 members on chat.rockylinux.org ~3,600 members on forums.rockyl i nux.org 251 members on IRC

1,319 members in the main development channel (chat.rockylinux.org and IRC)

1,182 members in the testing channel ~140 unique committers in GitHub

artifacts (signing keys, secure boot shims, etc.).

However, popularity can be a difficult metric to measure, and we don’t see this as a popularity contest. Competition and choice is healthy for the overall advancement of open source. It’s a good thing that multiple projects exist to solve similar problems for various communities and use cases. We prefer to let our community speak for Rocky Linux.

LM: Describe the target audience for Rocky Linux.

RL: Anyone and everyone who needs a stable, trusted, and transparent operating system for servers, appliances, cloud instances, containers, virtual machines, edge devices, compute nodes, and/or simply wants a reliable distribution for their home lab.

LM: Can you provide any stats about downloads, commits, and developers?

RL: See Table 1 for downloads and usage and Table 2 for community statistics.

I’ve also included some charts (Figures 2 and 3) that indicate the relative usage of Rocky Linux. It’s important to note in these charts that they do not represent the total usage of the distributions listed, as they are derived from usage statistics of the Extra Packages for Enterprise Linux (EPEL) repository, which not all users will enable. Still, an approximately equal proportion of users of each distribution will enable it, making it useful for determining the relative usages of each distribution. The ephemeral (< 1 week old instance) usage graph is highly variable (Figure 3), as it can be majorly swayed by large numbers of short-lived instances like supercomputing jobs, CI/CD systems, etc.

LM: How is Rocky Linux organized and governed?

RL: The roots of our governance model are based on how we developed, which is a pure meritocracy. After the announcement of the project, thousands of developers, contributors, and community members all joined a Slack instance around HPCng (the next generation of HPC).

Figure 2: Long-term statistics from the EPEL repository show Rocky Linux as a contender in the Fedora/Red Hat family.

With so many people joining the Slack community, it was unmanageable initially. Thousands of people were messaging Gregory Kurtzer [the creator of CentOS and of Rocky Linux] directly! So the community was broken into channels for different areas of what we needed to create. People joined the channels that they were most interested in and qualified for. From there, cooperation began, and the community began to develop. Within each channel, people started to emerge as natural leaders for each team. This occurred with people not even knowing anything about anyone else. Nobody checked where people worked, qualifications, gender, race, religion, or beliefs, and most people were using aliases with cartoon avatars. People got where they are within the Rocky organization based on their merit and contribution to the project alone.

These team leads became the controlling structure for Rocky Linux.

We then created the Rocky Enterprise Software Foundation (RESF) as the legal entity behind Rocky. We decided to go with a public benefit corporation (PBC) as it provides more flexibility than a 501 (c) organization, which will allow us to better protect the project.

Long-term protection of the RESF and its projects is imperative to many in our community who were involved in projects with seemingly safe structures that shifted away from their community’s wishes. For example, CentOS was founded as part of a 501 (c) (3) nonprofit organization. Other forms of the 501 (c) do not offer any more protection. For example, a 501 (c) (6) ’s purpose is to promote the common business interest of its sponsor (see 1.501(c)(6) in [2]).

RESF projects are composed of diverse sets of project members and backed by large communities of individuals and organizations all working together to serve their communities. Thus we needed a structure that we felt best protects and serves these communities.

This is why the integrity, accountability, and transparency of the people involved in the project are the most critical aspects for determining the long-term sustainability and viability of any project. We believe that the RESF mitigates the risk of diversions from community goals and takeover by external entities by implementing a series of checks and balances.

The structure that we created for the RESF is forged by the lessons learned from the many projects and organizations that have preceded us and will continue to evolve to meet the needs of the future. Information on our current structure can be found at [3].

LM: What features make Rocky Linux stand out, especially for security and networking?

RL: Rocky Linux is a bug-for-bug compatible and freely available implementation of RHEL. Beyond compatibility and feature parity, we are adding value with various security accreditations like FIPS, STIGs, Common Criteria, and others forthcoming.

We can’t stress enough how significant it is to have a freely available open source operating system pursuing a Federal Information Processing Standard Publication 140-3 (FIPS-140-3) certification for our cryptographic modules. Rocky Linux will be the first community-driven operating system to have this, and this means that all the cryptographic modules included in Rocky will have been verified and certified by independent accredited organizations like the National Institute of Standards and Technology (NIST). We are also pursuing an Evaluation Assurance Level (EAL4 +) certification, which is again the first time a community Linux distribution will have achieved this level of validation. If compliance and verification matters to you, then these are all huge advantages to using Rocky.

Figure 3: While highly variable, short-term statistics from the EPEL repository show Rocky rivalling and at times surpassing RHEL and CentOS Stream.

LM: How does Rocky Linux differ from RHEL and other CentOS replacements?

RL: We’ve differentiated ourselves further with our open build infrastructure, which is managed via many members of our community. This infrastructure was built by the community for the Rocky Linux project. There is no reliance on specific individuals or companies, and all the infrastructure we use is owned by the project.

This includes not only the infrastructure, but the cryptographic material for signing packages, access to infrastructure, and our secure boot shim. Our secure boot shim was built for and is owned by the Rocky project itself.

Going forward, we want to welcome other security professionals and organizations to participate in Rocky Linux Special Interest Groups [SIGs] to develop tools, guides, and standards around Rocky Linux. To date, we have already worked with Center for Internet Security (CIS) [4] to add security benchmarks for Rocky Linux 8, and SCAP definitions with Open-SCAP [5]. We have also coordinated with commercial vendors to add vulnerability scanning and compliance testing support for Rocky Linux and written several of our own guides for supporting DISA STIG [6] and PCI DSS compliance guidelines. We’d like to continue that effort and add support for more open and commercial tools to serve the Rocky Linux community.

From end to end, all engineering, development, testing, and collaboration are done openly and transparently. We share all of our tools, and we seek independent organizations like NIST and CIS to validate our methodology wherever we can. This is one of the reasons why our Mat-termost chat and IRC community are so large and inclusive; everyone can take part and be part of the team.

LM: The Peridot build system is new in Rocky Linux 9. Why is it important?

RL: Rocky Linux has never been built in a black box. Every version of Rocky Linux, as an open source and freely available operating system, has always included everything someone would need to replicate the entire project. This is what it means to be community first and it is our unwavering commitment to a collaborative open source project.

For Rocky Linux 8, we used Koji, Fedora’s build system, but for version 9, we decided we needed something easier for others to adopt, cloud-native, and something that would fit better into our goals of including Special Interest Groups and community contributions to the operating system. For that, we built Peridot.

Peridot is a cloud-native build system built upon a strong foundation of separation, security, ease of use, scaling, and modern architectures like microservices. Peridot can be used to not only build an RPM-based distribution, but it’s also fully capable of forking a distribution and optimizing long term maintenance. Additionally, it enables third-party contributors to extend the base distribution in the form of SIGs and extensions without compromising compatibility or our trusted artifacts (e.g., signing keys). Providing access to volunteers and contributors in a secure and trusted manner to build packages for Rocky Linux has been a challenge with our previous build system, and Peridot solves this.

As an Enterprise Linux operating system provider, we have to be fast and correct on releasing security updates and timely on all updates. Peridot allows us to distribute build tasks to multiple build sites and can automatically scale up workers in Kuber-netes to handle pressure as well as scale down when needed. After the package builds have completed, we still need to get these out to our users, and to make that process quicker, as well as build a proper microservice platform, it was required to operate statelessly and move beyond NFSbased on-disk repository generation and updates, so we also built Yumrepofs.

Yumrepofs doesn’t require a persistent host-based state for managing a YUM repository by swapping in and out necessary RPMs from a specific build or a configuration change on demand. This allows us to always have our updated repositories ready to release.

This also optimizes security by tamper-proofing the resulting build, artifacts, logs, and metadata by signing all of them, which guarantees provenance and trust of the entire operating system after Peridot has completed the builds.

We are so excited about Peridot because it:

allows us to easily maintain multiple major versions of distribution forks/ rebuilds
allows community members to securely engage with SIGs
enables multiple security measures to prevent private signing key breaching [and] package, artifact, and repository tampering
[and offers] unprecedented levels of operating system build transparency, all while enabling engineers to push out updates in record speed.

LM: Where is Rocky Linux heading in the future?

RL: More community involvement via packaging, enhancements, SIGs, testing, and bug reports all ensure the long-term success of our community and users.

We continue to grow our community of sponsors and partners, with many amazing organizations eager to be a part of what we are doing.

We are currently codifying the formal board structure; more changes will be recognized in terms of decision making processes and leadership.

LM: Where can people volunteer to get involved?

RL: Everyone is welcome! We are a community of super-friendly people, and we have lots of opportunities for more people to contribute and take leadership roles. We actively seek leaders from within our community, and we are looking for more people interested in both the technical and nontechnical aspects of running a community. We invite all to help regardless of skill level. Join us on IRC or Mattermost at https://chat. rockylinux.org!

Info