Modularity: Requirements and use cases

In order to make deployment of users’ applications simpler, we need to make sure that software can be installed into the common, expected locations on the system. This includes (but is not limited to):

Requirement: Installation must occur in the same locations as traditional RPM software delivery.

As Linux containers grow more popular, the number of disreputable images on container registries has also been rising. Delivering trustworthy binaries to users is something that traditional distributions have always done well. They provide assurance (in the form of signatures, Secure Boot and other trusted delivery paths) that the software you’re running is the software you think it is.

Requirement: It must be possible for users to construct container images based on Fedora content matching the software version they need for their applications.

It is very common for Fedora to update to the latest major version of packages at each new semiannual release. This ensures that Fedora remains at the leading edge of software development, but it can wreak havoc on anyone trying to maintain a deployment on Fedora. If they are running an app that is built for PostgreSQL 9.6 and Fedora switches to carrying PostgreSQL 10 in the next major release, upgrading to that release may break their app (possibly in ways undetectable by the upgrade process).

However, staying on an old version of Fedora forever has its own problems. Not least of these is the problem of security updates: Once a release has been out for about 13 months, it stops receiving errata. Moreover, new releases of the Fedora platform may have other useful enhancements (better security defaults, increased performance thanks to compiler improvements, etc.).

Requirement: We need to allow users to “lock” themselves onto certain dependencies as long as the packager is maintaining them. These dependencies must continue to receive updates.

Requirement: There must be appropriate and helpful UX for dealing with when those dependencies go EOL.

Developers often want to build their applications using the latest-and-greatest version of their dependencies. However, that may not have been released until after the most recent Fedora release. In non-Modular Fedora, that means waiting up to six months to be able to work on it there.

Requirement: It must be possible to gain access to newer software than was available at the Fedora release GA.

Additionally, Dev/Ops people are rapidly switching to a new paradigm of development and deployment (containers) to solve the above issue. However, most containers today are retrieved from public repositories. The public repositories are generally user-managed and have not been verified and validated for security.

Requirement: Provide a mechanism for building trusted container base and application images with content alternatives.

It’s not enough that other versions of software are available to install. They also need to be kept up to date with bug fixes and security updates. In non-Modular Fedora, users had the ability to force DNF to lock to a specific RPM NEVRA, but they wouldn’t get updates from it.

Requirement: Alternative software must receive be able to recieve and apply updates.

Having alternative versions available is important but not sufficient. It is also necessary for users to be able to locate these alternatives. Some of our early explorations into this area failed this ease-of-use test because they require the user to have knowledge of external sites and then to search those sites for what they think they want.

Requirement: Users must be able to discover what alternative software versions are available with tools that are shipped with the OS by default. Ideally, these should be the same tools that they are already comfortable with.

Users are slow to adapt to changes in the way they need to behave. Requiring them to learn a new set of commands to interact with their system will likely result in frustration and possibly exodus to other distributions.

Requirement: It must remain possible to continue to operate with only the package management commands used in traditional Fedora. We may provide additional commands to support new functionality, but we must not break the previous workflow.

Requirement: Existing automation tools such as anaconda’s kickstart and Ansible must continue to work.

Because very little software today is wholly self-contained, it must be possible for Modules to depend on each other.

Requirement: There must be a mechanism for packagers to explicitly list dependencies on other software, including alternative versions. This mechanism must support both build-time and run-time dependencies.

Some software is very restrictive about which dependencies it can work with. Other software may work with several different major releases of a dependency. For example, a user may ship two Ruby-based web applications, one which is capable of running on Ruby 2.5 and the other that can run on either Ruby 2.5 or Ruby 2.6. In non-modular Fedora, only one version of Ruby would be available. If the system version was 2.5, then both applications could run fine. But if in the next release of Fedora the Ruby 2.6 release becomes the system copy, one of those applications will have to be dropped (or patched) to work with it.

Requirement: It must be possible to build software that can be run against multiple versions of its dependencies.

Requirement: The packaging process for creating software that supports multiple versions of their dependencies must not be significantly more difficult than packaging for a single dependency.

As more and more things become modules, there is concern that such things will grow into an unbounded matrix. For this, we need to establish policies on when the use of alternative dependencies is preferable or when it is better to constrain it to a single version or small set.

Requirement: Packaging guidelines need to provide advice on when to use multiple alternative dependencies or to select a single one.

When a person decides that they want Fedora to carry a particular package and decides to do the work to accomplish this, it is not uncommon to discover that the package they care about has additional dependencies that are not yet packaged in Fedora. Traditionally, this has meant that the packager has needed to package up those dependencies and then continue to maintain them for anyone who may be using them for other purposes. This can sometimes be a significant investment in time and energy, all to support a package they don’t necessarily care about except for how it supports the primary package.

Since the earliest days of Linux, the “package” has been the fundamental unit of installable software. If you want to have some functionality on the system, you need to learn the name of the individual packages that provide that functionality (not all of which are named obviously). As we build modules, one of the goals is to try to focus installation around use-cases rather than around upstream projects. A big piece of this is that we want to have a way to install a subset of a module that supports specific use-cases. A common example being “server” and “client” cases.

Requirement: It must be possible to install a subset of artifacts from an alternative version. These installation groups should be easily discoverable.

Recommendation: Installation groups should be named based on the use-case they are intended to solve. This will provide a better user experience.

Another of the major issues faced by Fedora is maintaining a release schedule when all of the components within it follow vastly differing schedules. There are two main aspects to this problem:

Requirement: It must be possible for new alternative versions of software to become available to the Fedora Package Collection between release dates.

Requirement: It must be possible for alternative versions of software to go end-of-life during a Fedora release. This does not mean that the software must disappear from the repositories, merely that an assertion exists somewhere that after a certain date, the package will not receive updates.

Requirement: For alternative versions whose lifecycle will continue through at least part of the next Fedora release, it must be possible to upgrade from one release to the next and remain with the fully-compatible version.

Some third-party add-on repositories (particularly EPEL) have been limited in the past by relying on the system copies of packages in the base distribution of the release. In the particular case of EPEL, little can be done to upgrade these system copies. In order to be able to package much of the available FOSS software out there, it may be necessary to override some of the content shipped in the base system with packages known to work properly.

Requirement: It must be possible for third party repositories to create alternative versions that override base distribution content at the user’s explicit choice.

Requirement: It must be possible for third party repositories to create alternative versions of software that exist in the base distribution.

There is plenty of software out in the wild that maintains compatibility over time and is therefore useful to carry in multiple releases of Fedora. With traditional packaging, this means carrying and building separate branches of the packages for each release of Fedora. In the case of software “stacks” which are tightly bound, this means also manually building each of its dependencies in each release of Fedora.

Requirement: It must be possible to build multiple component software packages in the same build process.

Requirement: It must be possible for the packager to specify the order in which packages must be built (and to indicate which ones can be built in parallel).

Requirement: It must be possible to be build for all supported platforms using the same specification and with a single build command.

As mentioned in the Background section, the goals of Modularity are specifically to not implement parallel-installability. We recommend instead that users should rely on other mechanisms such as virtualization or containerization to accomplish this. If parallel-installation is unavoidable, then Modularity is not the correct tool for this job.

Module streams are intended to be compatible update streams. That means they must follow the same rules regarding RPM package-level updates within the stream. By definition, two streams of the same module exist because upgrades (or downgrades or cross-grades…) are not capable of being done in a safe, automated fashion.

That does not mean that stream switching should be impossible, but it does mean that we will not build any tools intended to handle such switching in a generic manner. Stream switches should be handled on a module-by-module basis and detailed instructions and/or tools written for each such case.