Fedora CoreOS Cincinnati SOP
Cincinnati is the update service/backend for Fedora CoreOS (FCOS) machines. This SOP describes how to access and how to troubleshoot it.
Details
- Source
- Playbook
-
https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/coreos-cincinnati.yml
- Location
-
OpenShift cluster (production): https://console-openshift-console.apps.ocp.fedoraproject.org/
- Project
-
coreos-cincinnati: https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/cluster/projects/coreos-cincinnati
- Deployment
- Containers
-
-
fcos-graph-builder(GB - raw updates graph) -
fcos-policy-engine(PE - frontend handling client requests)
-
- Routes
-
-
coreos-updates-raw(GB web service) -
coreos-updates-raw-status(GB status and metrics) -
coreos-updates(PE web service) -
coreos-updates-status(PE status and metrics)
-
Troubleshooting
Each FCOS Cincinnati service exposes live metrics in Prometheus format:
- Graph-builder
- Policy-engine
Upgrades
Build and deploy a new version
FCOS Cincinnati is built as container image directly from source
from the main branch on the https://github.com/coreos/fedora-coreos-cincinnati
repo.
Here are the steps to trigger a new build:
-
SSH to
batcave01.rdu3.fedoraproject.org -
schedule a new build by running
sudo rbac-playbook -t build openshift-apps/coreos-cincinnati.ymlusing your FAS password and your second-factor OTP -
Wait for the coreos-cincinnati-img:latest tag to be updated
Things that could go wrong
Application build is stuck
Issues in the underlying OpenShift cluster may result in builds being permanently stuck.
If a build does not complete within a reasonable amount of time (i.e. 15 minutes):
-
go to the build overview at https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/builds
-
click on the build
-
cancel it through the "Cancel Build" button
-
go back to the build overview page
-
schedule a new build through the "Start Build" button
Want to help? Learn how to contribute to Fedora Docs ›