Troubleshooting

Fedora Kinoite is a new way of deploying and managing your desktop operating system, so you may occasionally run into problems while going through your day to day. Below are some of the more common problems reported and any workarounds for those problems.

"Forbidden base package replacements"

This can happen when a package that is being layered has a dependency on a package that is in the base OS. In the problematic case, the layered package requires a newer version of the dependent package which is not available in the base OS.

In most cases, waiting for a newer OSTree compose will resolve this problem. The dependent package will be updated in the compose and the package that was going to be layered will be successful.

However, if you continue to encounter this problem with a newer compose, you can try to cleanup the metadata with rpm-ostree cleanup -m and then retrying the rpm-ostree install.

Alternatively, you can try rebasing to any updates ref, like fedora/30/updates/x86_64 after the cleanup operation.

For more information, see rpm-ostree#415.

Installing packages to /opt or /usr/local

Installing into /opt was commonly raised as a problem when users where trying to install Google Chrome. A partial solution has been implemented that allows users to layer Google Chrome, however it is not a complete solution for applications that write mutable data to /opt.

This issue is tracked in rpm-ostree#233.

Using NVIDIA drivers

You can install the official NVIDIA binary drivers from the RPM Fusion repositories.

The NVIDIA binary drivers are not maintained by the Fedora Project and may sometimes not be available for the kernel version included in Fedora Kinoite.
The Universal Blue project creates operating system images for Fedora Kinoite with the NVIDIA drivers included. The Universal Blue images are based on the official Fedora images with additional changes at their discretion. The Universal Blue images are not officially endorsed by the Fedora Project. Use them at your own discretion.

  1. First, ensure that your system is fully updated by running sudo rpm-ostree upgrade and rebooting.

  2. Then setup the RPM Fusion repositories following the documentation, including the two reboots.

  3. Finally, install the drivers:

    # rpm-ostree install kmod-nvidia xorg-x11-drv-nvidia
# rpm-ostree kargs --append=rd.driver.blacklist=nouveau --append=modprobe.blacklist=nouveau --append=nvidia-drm.modeset=1 --append=initcall_blacklist=simpledrm_platform_driver_init
# systemctl reboot
When using Secure Boot, the locally installed NVIDIA drivers have to be signed with a local key that is enrolled using mokutil. See the fedora-silverblue#499 issue for more details.

You may also encounter the following issues during installation: #286, #331

Thanks to Alex Larsson who made the required changes to the akmods and kmodtools packages. You can read more about the work that Alex did on his blog.

Out of tree kernel modules and drivers using DKMS

Fedora Kinoite currently does not have support for DKMS. See the upstream issue rpm-ostree#1091.

Instead, we recommend that you make kmods packages for out of tree kernel modules and submit them to the RPM Fusion repos. The kmods packages will then be used by akmods which is supported on Fedora Kinoite.

Adding external package repositories

This section discusses third-party software sources not officially affiliated with or endorsed by the Fedora Project. Use them at your own discretion.
If you want to use RPM Fusion repositories, please follow the Enabling RPM Fusion repos section.

Some sofware may only be available from a third-party repository. You can add an external repository manually on Fedora Kinoite by placing the .repo file into /etc/yum.repos.d/ and the GPG key into /etc/pki/rpm-gpg/. The following is a full example for setting up the Tailscale repo:

  1. Fetch and install the repo config:

    $ curl -O https://pkgs.tailscale.com/stable/fedora/tailscale.repo
[tailscale-stable]
name=Tailscale stable
baseurl=https://pkgs.tailscale.com/stable/fedora/$basearch
enabled=1
type=rpm
repo_gpgcheck=1
gpgcheck=0
gpgkey=https://pkgs.tailscale.com/stable/fedora/repo.gpg
$ sudo install -o 0 -g 0 -m644 tailscale.repo /etc/yum.repos.d/tailscale.repo

  2. Fetch and install the GPG keys:

    $ curl -O https://pkgs.tailscale.com/stable/fedora/repo.gpg
$ sudo install -o 0 -g 0 -m644 repo.gpg /etc/pki/rpm-gpg/tailscale.gpg

  3. Replace the gpgkey= URL in the repo config by the path the GPG keys:

    $ sudo $EDITOR /etc/yum.repos.d/tailscale.repo
$ cat /etc/yum.repos.d/tailscale.repo
[tailscale-stable]
name=Tailscale stable
baseurl=https://pkgs.tailscale.com/stable/fedora/$basearch
enabled=1
type=rpm
repo_gpgcheck=1
gpgcheck=0
# Update this line
gpgkey=file:///etc/pki/rpm-gpg/tailscale.gpg
#      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  4. Install the new packages with:

    $ rpm-ostree install tailscale

Better support in rpm-ostree for this use case is tracked in rpm-ostree#4014.

SELinux problems

As users work with Fedora Kinoite day-to-day, it is possible that they have modified the default SELinux policy in an effort to workaround one or more problems related to SELinux. This is usually done when a user sees a SELinux denial in the journal. If this is the case and one wishes to revert back to the default SELinux policy, you can try these set of actions.

  1. Check the state of the SELinux policy

    $ sudo ostree admin config-diff | grep policy
M    selinux/targeted/active/policy.linked
M    selinux/targeted/active/policy.kern
M    selinux/targeted/policy/policy.31
A    selinux/targeted/policy/policy.30

    If anything is returned by this command, then your SELinux policy has been modified from the default.

  2. Copy the default SELinux policy shipped in the OSTree compose

    $ sudo cp -al /etc/selinux{,.bak}
$ sudo rsync -rlv /usr/etc/selinux/ /etc/selinux/

    After doing this, the output from ostree admin config-diff | grep policy should no longer indicate the policy is modified.

    If your policy still appears to be modified, you can try the following approach.

  3. Remove the SELinux policy; copy in the default policy

    $ sudo rm -rf /etc/selinux
$ sudo cp -aT /usr/etc/selinux /etc/selinux

    After this, the ostree admin config-diff | grep policy command should return no modifications.

Unable to add user to group

Due to how rpm-ostree handles user + group entries, it may not be possible to use usermod -a -G to add a user to a group successfully. Until rpm-ostree moves to using systemd sysusers, users will have to populate the /etc/group file from the /usr/lib/group file before they can add themselves to the group.

For example, if you wanted to add a user to the libvirt group:

$ grep -E '^libvirt:' /usr/lib/group | sudo tee -a /etc/group
$ sudo usermod -aG libvirt $USER
You will need to log off and log back in to apply these changes.

This issue is tracked in rpm-ostree#29 and rpm-ostree#49.

ostree fsck reports file corruption

It is possible to end up in a situation where one or more files on the disk have become corrupted or missing. In this case, ostree fsck will report errors in certain commits. The workaround in this case is to mark the entire OSTree commit as partially retrieved and then re-pull the commit.

Read-only /boot/efi prevents any upgrades

This issue is most commonly seen when users have installed Fedora Kinoite on Apple hardware. The /boot/efi partition on Apple hardware is formatted as HFS+ and is not always resilient to power failures or other kinds of hard power events.

Since Fedora Kinoite now includes the hfsplus-tools package in the base compose, it has become relatively easy for users to workaround this kind of error.

# umount /boot/efi
# fsck.hfsplus /dev/sda1
# mount -o rw /boot/efi

See the rpm-ostree#1380 GitHub issue for additional details.

Unable to install Fedora Kinoite on EFI systems

Users have reported that they cannot install Fedora Kinoite on an EFI based system where they previously had another OS installed. The error that is often observed looks like:

ostree ['admin', '--sysroot=/mnt/sysimage', 'deploy', '--os=fedora-workstation', 'fedora-workstation:fedora/28/x86_64/workstation'] exited with code -6`

A couple of possible workarounds exist:

  • During the install process, select "Custom Partitioning" and create an additional EFI partition. Assign the newly created EFI partition to /boot/efi. You will then be able to boot the previous OS(s) alongside Fedora Kinoite. If this workaround is not successful follow the below step.

  • Reformat the EFI partition on the host during the install process. This can be done by selecting "Custom Partitioning" and checking the Reformat box when creating the /boot/efi partition.

Choosing to reformat /boot/efi will likely result in the inability to boot any other operating systems that were previously installed. Be sure that you have backed up any important data before using this workaround.

This issue is tracked in Bugzilla #1575957.

toolbox: failed to list images with com.redhat.component=fedora-toolbox

As of podman version 1.4.0 this workaround is not necessary. Ensure podman is up to date by issuing rpm-ostree upgrade before attempting this workaround.

When issuing the toolbox list command, systems using podman versions newer than 1.2.0, will generate the following error:

toolbox: failed to list images with com.redhat.component=fedora-toolbox
The following workaround might be useful for other toolbox errors caused by podman versions greater than 1.2.0. See Toolbox Github Repo

As a workaround, it is possible to override podman packages newer than version 1.2.0 by issuing:

$ rpm-ostree override --remove=podman-manpages replace https://kojipkgs.fedoraproject.org//packages/podman/1.2.0/2.git3bd528e.fc30/x86_64/podman-1.2.0-2.git3bd528e.fc30.x86_64.rpm

Reboot the system to apply the changes.

For reference, it is also possible to override the package by following these steps:

  1. Download podman-1.2.0-2.git3bd528e.fc30.x86_64.rpm from Koji

  2. Remove podman-manpages issuing: rpm-ostree override remove podman-manpages

  3. Override the currently installed podman package (using the package you have downloaded on the first step) by: rpm-ostree override replace podman-1.2.0-2.git3bd528e.fc30.x86_64.rpm

You can now reboot the system for the change to take effect.

To revert this workaround issue the following command:

$ rpm-ostree override reset podman; rpm-ostree override reset podman-manpages

Unable to enter a toolbox due to permissions errors

With certain versions of podman, trying to enter a toolbox will result in errors. You can fix this by resetting the permissions on the overlay-containers with the following command.

$ sudo chown -R $USER ~/.local/share/containers/storage/overlay-containers

This will reset the permissions on your containers and allow you to enter them again.

See the upstream podman issue: podman#3187.

Running restorecon

You should never run restorecon on a Fedora Kinoite host. See the following bug for details - https://bugzilla.redhat.com/show_bug.cgi?id=1259018

However, if you happened to do this, it is possible to recover.

  1. Boot with enforcing=0 on the kernel command line

  2. Create a new, "fixed" commit locally

  3. Deploy the new "fixed" commit

  4. Run restorecon

  5. Reboot

  6. Cleanup

$ rpm-ostree status -b | grep BaseCommit
                BaseCommit: 696991d589980aeaef5eda352dd7ad3d33c444c789c209f793a84bc6e7269aee
$ sudo ostree checkout -H 696991d589980aeaef5eda352dd7ad3d33c444c789c209f793a84bc6e7269aee /ostree/repo/tmp/selinux-fix
$ sudo ostree fsck --delete
$ sudo ostree commit --consume --link-checkout-speedup --orphan --selinux-policy=/ /ostree/repo/tmp/selinux-fix
$ sudo restorecon -Rv /var
$ sudo restorecon -Rv /etc
$ sudo ostree admin deploy fedora:fedora/41/x86_64/kinoite
$ sudo reboot

The caveat to this recovery is that your layered packages will be removed; you’ll need to relayer them after the recovery.

See this upstream comment for additional details: ostree#1265.

Resetting passwords in Rescue Mode

In the case where you are unable to remember your user password or root password, you can reset the password using the following steps.

  1. While the system is booting, interrupt the boot sequence at the GRUB2 menu by using the Esc key.

  2. Select the boot entry that you wish to edit using the arrow keys.

  3. Edit the selected entry with the e key.

  4. Use the arrow keys to select the line beginning with linux, linux16, or linuxefi.

  5. Go to the end of that line and append init=/bin/bash to the end of the line.

  6. Press Ctrl-x or F10 to boot the entry.

  7. At the resulting bash prompt, run the following commands:

# mount -t selinuxfs selinuxfs /sys/fs/selinux
# /sbin/load_policy
# passwd
# sync
# /sbin/reboot -ff

If you want to change the password for a user account, replace the passwd command with passwd <username>.

After the system finishes rebooting, you should be able to login with the username and new password.

Want to help? Learn how to contribute to Fedora Docs ›