Package Update Guide

This document shows how to submit an update for a package you maintain in Fedora. It assumes you already have a package in the Fedora repositories. It is not a guide to using the Fedora package source control system: see Package Maintenance Guide for that.

Overview

This page is intended for new and existing package maintainers. Testers and regular users may be interested in the updates-testing repository and the Update feedback guidelines. This page specifically covers the update submission process.

There are two significantly different package update submission workflows in Fedora:

  • Before updates-testing activation, which is used for Rawhide and Branched up to the Updates-testing Activation.

  • After updates-testing activation, which is used for Branched releases after Updates-testing Activation, and stable releases.

The repository layouts differ somewhat for Rawhide, Branched and stable releases, but the update workflows split up as described above.

There is also a difference between single-package updates and multi-package updates.

Single-package updates before updates-testing activation

Most packages can be updated alone, without needing to be grouped with other packages for consistent dependencies. These are "single package builds". If the change you are making affects other packages, see Multi-package updates.

The update workflow for single package builds before updates-testing activation is simple:

  1. Build the package with fedpkg build. See Package Maintenance Guide for more details.

This is all you need to do, a Bodhi update will be created automatically, from which potential tests will be run, some of which may gate the update. In the Bodhi web UI, gating tests are indicated by an asterisk.

  • If the built package doesn’t have gating tests, or if they succeed, the update will be marked as stable and your package will appear in subsequently created build roots, as well as in the next daily compose of Rawhide or Branched and will be used in any image composes built from that tree.

  • If the built package has gating tests which fail, this will be recorded in the update. You can now fix whatever is wrong with the package and build it again, which in turn will create a new update, running the tests (on the now hopefully fixed package), and so forth. If you’re absolutely sure that the test shouldn’t fail, you can waive the failing tests, but please check with the Fedora CI and/or Quality team before waiving a failure.

Single-package updates after updates-testing activation

At the updates-testing activation point, the Bodhi update feedback system is enabled by Release Engineering and builds submitted with fedpkg build are no longer automatically sent to any official repository. The update workflow for releases of this type is:

  1. Build the package with fedpkg build

  2. Submit an update for the package with fedpkg update, the Bodhi web interface, or the bodhi-cli tool. This causes the package to be sent to the updates-testing repository.

  3. Monitor the update’s status and the feedback you receive via the web interface or email, and modify it with updated or additional builds if necessary.

  4. After the update meets the criteria in the Updates Policy and you are satisfied it should be released as a stable update, submit the update to stable with bodhi updates request <update_id> stable or the web interface.

Just like early Branched and Rawhide updates, gating tests may be run on the update. If any fail, you should fix the problem, send a new build, and edit it into the update.

At the time you submit the update, you will be asked for several attributes. See section Update attributes for guidance.

Multi-package updates

Some updates require changes in multiple related packages. For example, if you maintain bar which depends on libfoo, and you need to update libfoo, you should check that bar still works with the updated libfoo. If it does not, you must ensure the appropriate changes are made to bar, and include the updated bar in your update along with the updated libfoo.

Side tags

Fedora has the concept of using side tags for these situations, which means the builds are done "on the side" and do not affect packages out of the side tag, nor are they available for installing until the side tag is merged. Updates are never automatically created for side tag builds. Using a side tag is the best and recommended way to do multi-package updates.

Packagers can create side tags on their own, allowing them to build disruptive components in isolation and submit the builds in a side tag as one update in Bodhi, to be tested and subsequently merged into the main distribution.

Side tags are cleaned up 30 days after creation, or 14 days if they have not been used at all. Make sure and use your side tag before then.

Creating a side tag

The easiest way to create a side tag is to be in the checked out branch (e.g. rawhide) of one of the packages and issue the following command:

fedpkg request-side-tag

Alternatively, the following variants allow specifying the Fedora release or the base tag for the side tag:

fedpkg --release f41 request-side-tag
fedpkg request-side-tag --base-tag f41-build

fedpkg will tell you the commands to build a package in the specific side tag and to wait for the respective build root to be recreated:

$ fedpkg request-side-tag
Side tag 'f41-build-side-7863' (id 7863) created.
Use 'fedpkg build --target=f41-build-side-7863' to use it.
Use 'koji wait-repo f41-build-side-7863' to wait for the build repo to be generated.

The latter is important if any builds depend on previous ones in the side tag. Use koji wait-repo --build <package-nvr> <side-tag> to ensure that the respective build is available in the build root for subsequent builds.

Bodhi update for builds in a side tag

When you’re done building all packages you want in a side tag, you have to submit them as an update to Bodhi before they can be made available generally to be installed and built upon. Even for Rawhide and early Branched, when using a side tag, you must create the update manually.

In the Create New Update form in Bodhi, choose the Use Side-Tag drop-down to create an update from the latest package builds in the respective tag:

Bodhi-builds-from-side-tag.png

As with single packages, tests will be run whose result affects if the update can be moved to stable. The difference is that gating tests have to succeed (or be waived) for all builds in the update. If you have to update the list of builds, e.g. to fix problems found during testing, send the new build to your side tag, then edit the update and refresh the list of builds using the 🔃 (refresh) button:

Bodhi-builds-refresh-from-side-tag.png

This will change the update to use the latest builds from the side tag. If there are multiple builds for the same package in the side tag, it will always use the most recently tagged one.

The web interface only works if you are the creator of the side tag. If you are a proven packager submitting an update for a side tag, you currently need to use the bodhi cli:

bodhi updates new --from-tag --notes "whatever" f41-build-side-7863

For more general information on creating updates, see Update attributes. Once the update is created, it will behave similarly to a single-package update. For Rawhide and early Branched, it will go to stable as soon as any gating tests pass. For later Branched and stable releases, it will require positive feedback or some time in updates-testing, as explained in the earlier section.

Once the update moves to stable, the builds will be tagged to the main tag of the release, i.e. are available for the general public. The side tag will be automatically removed.

Adding/Removing builds to/from a side tag

As a packager, you can add or remove builds from your side tag using these commands:

koji tag <side-tag> <nvr>
koji untag <side-tag> <nvr>

This can be used to remove a build that made a test fail or to add a build that was originally missed. If you add or remove a build from a side tag, you will have to refresh the corresponding update in Bodhi.

Removing the last build in a side-tag will automatically remove/delete the side tag.

Using macros in a side tag

Koji allows macros to be set in a side tag. This is useful in certain scenarios, such as changing configuration of builds in this side tag without changing their sources. The following commands may be used to set and unset macro respectively:

$ koji edit-sidetag --rpm-macro <macro>=<value> <side-tag>
$ koji edit-sidetag --remove-rpm-macro <macro> <side-tag>
Please make sure python3-koji-cli-plugins is installed on your system to have the koji edit-sidetag command available.

The following commands can be used to review macros configured for the side tag:

$ koji taginfo <side-tag>
$ koji list-history --tag <side-tag>
Please note that the macro changes won’t take effect immediately. It is necessary to wait using $ koji wait-repo <side-tag> command until the repository data is refreshed.

While modifying the macro configuration on side-tags can be useful, it can also be problematic. As such, macros must be approved by FESCo and the list of currently supported macros is available in the Policy for Buildroot Macros.

Removing a side tag

To conserve build system resources, side tags that are not needed anymore should be removed. This can be achieved with fedpkg as follows:

fedpkg remove-side-tag <side-tag>

Bodhi automatically removes side tags used for updates pushed to stable, so removing them manually is usually not necessary.

Bodhi multi-package updates

You can also create an update with multiple packages directly in Bodhi, either with the Bodhi web application, or with the bodhi command line tool. This is usually a worse method than using a side tag, because managing the build root is more difficult and will affect packages outside of the group. It may be appropriate if the packages logically belong together as an update, but do not have any build-time interdependencies. You cannot use this method for Rawhide or early Branched updates, because of the automatic update creation system.

You can pass multiple package names to bodhi updates new command to create a new multi-package update, or use bodhi updates edit to edit an existing update.

It is possible you will run into problems with permissions when you are not the maintainer of the package you are trying to add to an update, or when you are trying to add packages to an update created by somebody else. If you encounter a situation like this, you should contact the release engineering team or a proven packager for help.

Buildroot overrides

When trying to do a multi-package update without using side tags, you may need to use buildroot overrides. A buildroot override puts a build that is not yet stable into the main build root for the release. Before on-demand side tags, this was the usual way to create a multi-package update when the packages have build-time interdependencies. Unlike with side tags, buildroot overrides are not isolated and thus should be only used in special cases where side tags do not suffice.

Update attributes

At the time you submit an update, you will be asked for several attributes. The type of the update should be fairly self-explanatory: Either it fixes bugs, adds new features, or is a new package.

If you are asked whether you want to send the update to updates-testing or stable, this is a no-op: all updates now go through updates-testing. It does not matter what you choose.

There are several schools of thought on filling out the update description. Some would suggest you consider the target audience: for a stable release, in particular, many Fedora users will see this text, and many of them may not be particularly familiar with your package. Consider not simply describing literally the changes in the update, but explaining as if to an outsider why your are updating the package, what benefits it will bring to them (if any), and anything they may want to note in order to have a smooth update experience.

If you associate one or more bug reports with your update, Bodhi will post comments into Bugzilla to alert those following the bug reports that an update is available. If you mark your update as fixing the bug(s), Bodhi will move the report(s) through the MODIFIED, ON_QA and CLOSED ERRATA states of the Bug Status Workflow as your update reaches various points in the process. Using this mechanism can be very useful both for you and for users of your package.

You may set thresholds for automatic push to stable based on positive feedback (karma) and/or time spent in updates-testing. If your update’s total karma (items of positive feedback minus items of negative feedback) reaches the karma threshold, it will be pushed stable automatically. If your update reaches the threshold for time in updates-testing without the builds changing or the total karma going so low it is unpushed, it will be pushed stable automatically.

There are defaults for these thresholds which vary at different points in the release process.

If you choose to use the karma auto-push threshold, please carefully consider an appropriate feedback level. For a relatively obscure package which is quite stable, 1 or 2 may be an appropriate value. For a popular, sensitive and complex package such as firefox or kernel, the default may be insufficient and a choice of 5 or even 10 may be appropriate.

Handling feedback from automated tests

Fedora’s automated testing systems, including Fedora CI, openQA, and Fedora CoreOS CI, may run automated tests on your update.

In the Bodhi web interface, updates have an Automated Tests tab which displays the results of all automated tests. Tests with an asterisk (star) at the left-hand side of the row are "gating" tests. These are the tests which prevent your update going stable if they fail. If a test does not have an asterisk, it is not gating, and does not affect whether your update can be pushed stable. For updates in the critical path, some gating tests are always present. For updates not in the critical path, there may be gating tests if any package in the update has configured them in its package-level gating configuration. If a failed gating test is waived, the asterisk is replaced with a thumbs-up.

  • Tests shown with a red background failed.

  • Tests shown with a green background passed.

  • Tests shown with a blue background and an "i" icon had some minor issue, but effectively passed.

  • Tests shown with a blue background and an hourglass icon are pending.

The tests are not all 100% accurate, but they are fairly often correct. Especially failures of openQA tests - those whose name starts with update. - should always be investigated and resolved before an update is pushed stable. If you see a failure, it is a very good idea to click on the result (which will take you to a detailed log) and investigate the issue. If you are unsure what the test indicates, you can contact the relevant team for help. When a test has failed gating, a box is shown with contact details.

The Fedora Quality team proactively investigates failed openQA tests and will usually resolve them or post a comment on the update explaining the problem and suggesting a resolution.

Waive a result

Any failure of a gating test will prevent your update from being released. On the update’s Details page in the Bodhi web interface, the Test Gating status will be shown as N of N required tests failed, and one or more red rows with asterisks should be present on the Automated Tests tab. If you are absolutely sure such a failure is a false one, you can 'waive' all failed gating tests from the Bodhi web interface by clicking the Waive Test Results button. This button is only present when an update has failed gating tests, and you have privileges to control the update. At present it is not possible to waive the absence of a result for a gating test in this way. If needed, this can be done with waiverdb-cli. Please do not waive failures of openQA tests - those whose names start with update. - without checking with the Fedora Quality team.

A tool called waiverdb-cli exists for filing waivers manually, though this should rarely be necessary. The tool is also missing some features needed to correctly specify waivers in some cases, like the ability to specify a scenario. It is not currently packaged in Fedora. You can get it from the the waiverdb repository, which also has some instructions on usage.

Who will receive your update, when?

Before updates-testing activation, when an update goes stable it is tagged for inclusion in the next compose. Composes are usually run daily at 05:15 UTC and take several hours. After a compose completes, it takes some more time for it to reach the public mirrors. All users will see your update when they update their systems after a successful compose containing your update has reached the mirrors.

For Branched releases after updates-testing activation, the updates-testing repository is enabled by default so most users will get your update soon after it is pushed there, but only packages that have been pushed to stable are used in building composes (both nightlies and the Beta and Final candidates and releases). The updates-testing repository is regenerated daily.

When a release is in stable state, the updates-testing repository is disabled by default, but Quality team members and others run with it enabled in order to provide testing and Bodhi feedback. The main user population will see your update only when it passes Bodhi, is marked as stable and reaches the updates repository. Both updates-testing and updates repositories are regenerated daily.

Where a package goes when it is marked as stable differs between Branched and stable releases. In Branched releases, stable packages are pushed to the base fedora repository. In stable releases, stable packages are pushed to the updates repository. However, from the point of view of the packager, this is an insignificant implementation detail. For more details, see Repositories.

Branched milestone freezes

For a short period before each milestone release, the stable fedora repository is frozen. These periods are shown as Milestone freezes (Post-branch Freeze, Beta Freeze, Final Freeze) on schedules. During these periods, builds will not be marked stable and pushed from updates-testing to fedora even after being submitted manually or automatically. In the normal course of events, they will be pushed after the milestone release is approved at a Go No Go Meeting. If you believe your update deserves to break a milestone freeze, a freeze exception may be granted through the freeze exception process. Accepted release blocking bugs are granted the same status through the blocker bug process.

For more on the Fedora development process, see Fedora Release Life Cycle.

Security updates

For bugs identified as security issues, there is an additional process that layers over the regular update process. If a bug is assigned to you that blocks a Security Tracking Bug, you must follow that process in addition to this one.

New package submissions

If you want to build a new package, but you aren’t sure which releases to send it to:

  • New packages should always be built for Rawhide

  • New packages can be built for Branched and stable releases if adding them would provide value to users of those releases without significant risk of causing harm

The submission process for new packages, after they have passed the Package Review Process and been given an SCM repository, is exactly the same as that for package updates.

Consider creating a package test plan

If you create test cases for your package, and categorize them appropriately, they will be automatically linked in Bodhi, so that testers will have some guidance for planned update testing.

Want to help? Learn how to contribute to Fedora Docs ›