Fedora Debuginfod Service - SOP

Debuginfod is the software that lies behind the service at https://debuginfod.fedoraproject.org/ and https://debuginfod.stg.fedoraproject.org/ . These services run on 1 VM each in the stg and prod infrastructure at IAD2.

Contact Information

Owner

RH perftools team + Fedora Infrastructure Team

Contact

@fche in #fedora-noc

Servers

VMs

Purpose

Serve elf/dwarf/source-code debuginfo for supported releases to debugger-like tools in Fedora.

Repository

https://sourceware.org/elfutils/Debuginfod.html https://fedoraproject.org/wiki/Debuginfod

How it works

One virtual machine in prod NFS-mount the koji build system’s RPM repository, read-only. The production VM has a virtual twin in the staging environment. They each run elfutils debuginfod to index designated RPMs into a large local sqlite database. They answers HTTP queries received from users on the Internet via reverse-proxies at the https://debuginfod.fedoraproject.org/ URL. The reverse proxies apply gzip compression on the data and provide redirection of the root / location only into the fedora wiki.

Normally, it is autonomous and needs no maintenance. It should come back nicely after many kinds of outage. The software is based on elfutils in Fedora, but may occasionally track a custom COPR build with backported patches from future elfutils versions.

Configuration

The daemon uses systemd and /etc/sysconfig/debuginfod to set basic parameters. These have been tuned from the distro defaults via experimental hand-editing or ansible. Key parameters are:

  • The -I/-X include/exclude regexes. These tell debuginfod what fedora versions to include RPMs for. If index disk space starts to run low, one can eliminate some older fedoras from the index to free up space (after the next groom cycle).

  • The --fdcache related parameters. These tell debuginfod how much data to cache from RPMs. (Some debuginfo files - kernel, llvm, gtkweb, …​) are huge and worth retaining instead of repeated extracting.) This is straight disk space vs. time tradeoff.

  • The -t (scan interval) parameter. Scanning lets an index get bigger, as new RPMs in koji are examined and their contents indexed. Each pass takes a bunch of hours to traverse the entire koji NFS directory structure to fstat() everything for newness or change. A smaller scan interval lets debuginfod react quicker to koji builds coming into existence, but increases load on the NFS server. More -n (scan threads) may help the indexing process go faster, if the networking fabric & NFS server are underloaded.

  • The -g (groom interval) parameter. Grooming lets an index get smaller, as files removed from koji will be forgotten about. It can be run very intermittently - weekly or less - since it takes many hours and cannot run concurrently with scanning.

A quick:

systemd restart debuginfod

activates the new settings.

In case of some drastic failure like database corruption or signs of penetration/abuse, one can shut down the server with systemd, and/or stop traffic at the incoming proxy configuration level. The index sqlite database under /var/cache/debuginfod may be deleted, if necessary, but keep in mind that it takes days to reindex the relevant parts of koji. Alternately, with the services stopped, the 150GB+ sqlite database files may be freely copied between the staging and production servers, if that helps during disaster recovery.

Monitoring

Prometheus

The debuginfod daemons answer the standard /metrics URL endpoint to serve a variety of operational metrics in prometheus. Important metrics include:

  • filesys_free_ratio - free space on the filesystems. (These are also monitored via fedora-infra nagios.) If the free space on the database or tmp partition falls low, further indexing or even service may be impacted. Add more disk space if possible, or start eliding older fedora versions from the database via the -I/-X daemon options.

  • thread_busy - number of busy threads. During indexing, 1-6 threads may be busy for minutes or even days, intermittently. User requests show up as "buildid" (real request) or "buildid-after-you" (deferred duplicate request) labels. If there are more than a handful of "buildid" ones, there may be an overload/abuse underway, in which case it’s time to identify the excessive traffic via the logs and get a temporary iptables block going. Or perhaps there is an outage or slowdown of the koji NFS storage system, in which case there’s not much to do.

  • error_count. These should be zero or near zero all the time.

Logs

The debuginfod daemons produce voluminous logs into the local systemd journal, whence the traffic moves to the usual fedora-infra log01 server, /var/log/hosts/debuginfod*/YYYY/MM/DD/messages.log. The lines related to HTTP GET identify the main webapi traffic, with originating IP addresses in the XFF: field, and response size and elapsed service time in the last columns. These can be useful in tracking down possible abuse. :

Jun 28 22:36:43 debuginfod01 debuginfod[381551]: [Mon 28 Jun 2021 10:36:43 PM GMT] (381551/2413727): 10.3.163.75:43776 UA:elfutils/0.185,Linux/x86_64,fedora/35 XFF:*elided* GET /buildid/90910c1963bbcf700c0c0c06ee3bf4c5cc831d3a/debuginfo 200 335440 0+0ms

The lines related to prometheus /metrics are usually no big deal.

The log also includes info about errors and indexing progress. Interesting may be the lines like:

Jun 28 22:36:43 debuginfod01 debuginfod[381551]: [Mon 28 Jun 2021 10:36:43 PM GMT] (381551/2413727): serving fdcache archive /mnt/fedora_koji_prod/koji/packages/valgrind/3.17.0/3.fc35/x86_64/valgrind-3.17.0-3.fc35.x86_64.rpm file /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so

which identify the file names derived from requests (which RPMs the buildids to). These can provide some indirect distro telemetry: what packages and binaries are being debugged and for which architectures?

Want to help? Learn how to contribute to Fedora Docs ›