Fedora Release Infrastructure SOP

This SOP contains all of the steps required by the Fedora Infrastructure team in order to get a release out. Much of this work overlaps with the Release Engineering team (and at present share many of the same members). Some work may get done by releng, some may get done by Infrastructure, as long as it gets done, it doesn’t matter.

Contact Information

Owner

Fedora Infrastructure Team, Fedora Release Engineering Team

Contact

#admin (matrix), #releng (matrix), sysadmin-main, sysadmin-releng

Location

N/A

Servers

All

Purpose

Releasing a new version of Fedora

Preparations

Before a release ships, the following items need to be completed.

  1. Verify mirror space (for all test releases as well)

  2. Verify with rel-eng permissions on content are right on the mirrors. Don’t leak.

  3. Announce the start of the infrastructure change freeze, update ansible variable

  4. Modify Template:FedoraVersion to reference new version. (Final release only)

  5. Announce the end of the infrastructure change freeze, update ansible variable

Change Freeze

Starting the infrastructure freeze

The schedule will list the date the infrastructure freezes should be imposed (three weeks before the early target release date, for Beta; three weeks before, for Final). On this date, some hours after the release freeze goes into effect, set the InfraFrozen variable in Frozen.yaml to True. Then send out an email to the infrastructure list. The subject should be "Fedora <NN> <Beta/Final> Freeze now in effect!" (insert release number and delete milestone as appropriate). The body should read:

Greetings.

we are now in the infrastructure freeze leading up to the Fedora <NN>
Final release. This is a <final/pre> release freeze.

We do this to ensure that our infrastructure is stable and ready to
release Fedora <NN> <Beta> when it's available.

You can always check if an infrastructure freeze is in place by
checking the value of the InfraFrozen variable at:
https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/Frozen.yaml

You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:

git clone
https://infrastructure.fedoraproject.org/infra/ansible.git
ansible/scripts/freezelist -i inventory

Any host listed as "freezes" is frozen until <YYYY-MM-DD> (or later if
release slips). Frozen hosts should have no changes made to them without
a sign-off on the change from at least 2 sysadmin-main or rel-eng
members, along with (in most cases) a patch of the exact change to be
made to this list and/or a pull-request to the infra/ansible repo.

Thanks,
Fedora infrastructure team

You must make manual edits everywhere you see angle brackets <>, depending on whether this is the Beta or Final infra freeze, and to insert the appropriate date for the freeze to be lifted. This date should be one day after the earliest target release date on the schedule. Replace <NN> with the appropriate release number.

The exact timing of the freeze is a little flexible to allow for last- minute changes. Co-ordinate with the rest of the team on this. If nothing in particular is pending, aim to send the mail four to six hours after the release freeze is announced.

Ending the infrastructure freeze

A day after the release happens, the infrastructure freeze should be lifted. Do this only after making sure there are no important outstanding tasks related to the release that should be addressed before the freeze ends.

Set the InfraFrozen variable in Frozen.yaml to False. Then send out an email to the infrastructure list. The subject should be "Fedora <NN> <Beta/Final> Freeze now over!" (insert release number and delete milestone as appropriate). The body should read:

With the release of Fedora <NN> <Beta> yesterday, infrastructure freeze is now
over.

You can always check if an infrastructure freeze is in place by
checking the value of the InfraFrozen variable at:
https://pagure.io/fedora-infra/ansible/blob/main/f/vars/all/Frozen.yaml

Our next freeze is for Fedora <release> <Beta>, currently scheduled for
<YYYY-MM-DD>.

Thanks,
Fedora infrastructure team

You must make manual edits everywhere you see angle brackets <>, depending on whether this is the Beta or Final infra freeze, and to insert the appropriate date for the next freeze. Replace <NN> with the appropriate release number.

Infrastructure freeze rules

The rules are simple:

  • Hosts with the ansible variable "freezes" "True" are frozen. These are in general hosts that possibly could affect the release somehow. ie, they are needed to compose / distribute / manage the release.

  • You may make changes as normal on hosts that are not frozen. (For example, staging is never frozen)

  • Changes to frozen hosts requires a freeze break request sent to the fedora infrastructure list, containing a description of the problem or issue, actions to be taken and (if possible) patches to ansible that will be applied. These freeze breaks must then get two approvals from sysadmin-main or sysadmin-releng group members before being applied.

  • Changes to recover from outages are acceptable to frozen hosts if needed.

You can get a list of frozen/non-frozen hosts by:

git clone https://pagure.io/fedora-infra/ansible.git
scripts/freezelist -i inventory

Notes about release day

Release day is always an interesting and unique event. After the final sprint from test to the final release a lot of the developers will be looking forward to a bit of time away, as well as some sleep. Once Release Engineering has built the final tree, and synced it to the mirrors it is our job to make sure everything else (except the bit flip) gets done as painlessly and easily as possible.

All communication is typically done in #admin. Typically these channels are laid back and staying on topic isn’t strictly enforced. On release day this is not true. We encourage people to come, stay in the room and be quiet unless they have a specific task or question releated to release day. Its nothing personal, but release day can get out of hand quick.

During normal load, our websites function as normal. On release day our load spikes a great deal.

Recent releases have been quite smooth due to a number of changes: we have a good deal more bandwith on master mirrors, more cpus and memory, as well as prerelease versions are much easier to come by for those interested before release day.

Day Prior to Release Day

Step 1 (Website)

Verify the website design / content has been finalized with the websites team.

Step 2 (Mirrors)

Verify enough mirrors are setup and have Fedora ready for release. If for some reason something is broken it needs to be fixed. Many of the mirrors are running a check-in script. This lets us know who has Fedora without having to scan everyone. Hide the Beta, and Preview releases from the publiclist page.

You can check this by looking at:

wget "http://mirrors.fedoraproject.org/mirrorlist?path=pub/fedora/linux/releases/test/28-Beta&country=global"

(replace 28 and Beta with the version and release.)

Release day

Step 1 (Prep and wait)

Verify the mirrors are ready and that the torrent has valid copies of its files (use sha1sum)

Step 2 (Bit flip)

The mirrors sit and wait for a single permissions bit to be altered so that they show up to their services. The bit flip (done by the releng team) will replicate out to the mirrors. Verify that the mirrors have received the change by seeing if it is actually available, just use a spot check. Once that is complete move on.

Step 3 (Website)

Once all of the distribution pieces are verified (mirrors and torrent), all that is left is to publish the website.

Step 4 (wiki version)

Update the Fedora version number wiki template if this is a final release. It will need to be changed in https://fedoraproject.org/wiki/Template:CurrentFedoraVersion

Step 5 (Monitor)

Once the website is live, keep an eye on various news sites for the release announcement. Closely watch the load on all of the boxes, proxy, application and otherwise. If something is getting overloaded, see suggestions on this page in the "Juggling Resources" section.

Step 6 (Done)

Just chill, keep an eye on everything and make changes as needed. If you can’t keep a service up, try to redirect randomly to some of the mirrors.

Priorities

Priorities of during release day (In order):

  1. Website

    Anything related to a user landing at fedoraproject.org, and clicking through to a mirror or torrent to download something must be kept up. This is distribution, and without it we can potentially lose many users.

  2. Linked addresses

    We do not have direct control over what Hacker News, Phoronix or anyone else links to. If they link to something on the wiki and it is going down or link to any other site we control a rewrite should be put in place to direct them to http://fedoraproject.org/

  3. Torrent

    The torrent server has never had problems during a release. Make sure it is up.

  4. docs.fedoraproject.org

    People will want to see whats new in Fedora and get further documentation about it. Much of this is in the release notes.

  5. wiki

    Because it is so resource heavy, and because it is so developer oriented we have no choice but to give the wiki a lower priority.

  6. Everything else.

CHECKLISTS:

Beta:

  • Announce infrastructure freeze 3 weeks before Beta

  • mail infrastucture list a reminder.

  • File all tickets

  • new website

  • check mirror permissions, mirrormanager, check mirror sizes, release day ticket.

After release is a "go":

  • Make sure torrents are setup and ready to go.

After release:

  • post to infrastructure list that freeze is over.

Final:

  • Announce infrastructure freeze 2 weeks before Final

  • mail infrastucture list a reminder.

  • File all tickets

  • new website, check mirror permissions, mirrormanager, check

  • mirror sizes, release day ticket.

After release is a "go":

  • Make sure torrents are setup and ready to go.

  • update wiki version numbers and names.

After release:

  • post to infrastructure list that freeze is over.

Want to help? Learn how to contribute to Fedora Docs ›