DNS repository for fedoraproject

We’ve set this up so we can easily (and quickly) edit and deploy dns changes with a record of who changed what and why. This system also lets us edit out proxies from rotation for our many and varied websites quickly and with a minimum of opportunity for error. Finally, it checks to make sure that all of the zone changes will actually work before they are allowed.

DNS Infrastructure SOP

We have 5 DNS servers:

ns02.fedoraproject.org

hosted at ibiblio (ipv6 enabled)

ns05.fedoraproject.org

hosted at internetx (ipv6 enabled)

ns13.rdu2.fedoraproject.org

in rdu2, internal to rdu2.

ns01.iad2.fedoraproject.org

in iad2, internal to iad2.

ns02.iad2.fedoraproject.org

in iad2, internal to iad2.

Contents

Contact Information

Owner

Fedora Infrastructure Team

Contact

#fedora-admin, sysadmin-main, sysadmin-dns

Location

ServerBeach and ibiblio and internetx and phx2.

Servers

ns02, ns05, ns13.rdu2, ns01.iad2, ns02.iad2

Purpose

Provides DNS to our users

Troubleshooting, Resolution and Maintenance

Check out the DNS repository

You can get the dns repository from /srv/git/dns on batcave01:

$ git clone /srv/git/dns
Only members of the group sysadmin-dns may push changes to this repo.

Adding a new Host

Adding a new host requires to add it to DNS and to ansible, see Virtual Host Addition SOP for the details.

Editing the domain(s)

We have three domains which needs to be able to change on demand for proxy rotation/removal:

  • fedoraproject.org.

  • getfedora.org.

  • cloud.fedoraproject.org.

The other domains are edited only when we add/subtract a host or move it to a new ip. Not much else.

If you need to edit a domain that is NOT In the above list:

  • change to the 'master' subdir, edit the domain as usual (remember to update the serial), save it.

If you need to edit one of the domains in the above list: (replace fedoraproject.org with the domain from above):

  • if you need to add/change a host in fedoraproject.org that is not '@' or 'wildcard' then:

    • edit fedoraproject.org.template

    • make your changes

    • do not edit the serial or anything surrounded by \{\{ }} unless you REALLY know what you are doing.

  • if you need to only add/remove a proxy during an outage or due to networking issue then run:

    • ./zone-template fedoraproject.org.cfg disable ip [ip] [ip]:: to disable the ip of the proxy you want removed.

    • ./zone-template fedoraproject.org.cfg enable ip [ip] [ip]:: reverses the disable

    • ./zone-template fedoraproject.org.cfg reset:: will reset to all ips enabled.

  • if you want to add an all new proxy as '@' or 'wildcard' for fedoraproject.org:

    • edit fedoraproject.org.cfg

    • add the ip to the correct section of the ipv4 or ipv6 in the config.

    • save the file

    • check the file for validity by running: python fedoraproject.org.cfg looking for errors or tracebacks.

When complete run:

git add . git commit -a -m 'description of your change here'

It is important to commit this before running the do-domains script as it makes it easier to track the changes.

In all cases then run:

  • ./do-domains

  • if that completes successfully then run:

    git add .
git commit -a -m 'description of your change here'
git push

  • nameservers update from dns via cron every 10minutes.

The above git process can be achieved with the below bash function where the commit message is passed as an arg when running.:

dnscommit()
{
  local args=$1
  cd ~/dns;
  git commit -a -m "${args}"
  git pull --rebase && ./do-domains && git add built && git commit -a -m "Signed DNS" && git push
}

If you need an update to be live more quickly:

and then run this on all of the nameservers (as root):

/usr/local/bin/update-dns

To run this via ansible from batcave do:

$ sudo rbac-playbook update_dns.yml

this will pull from the git tree, update all of the zones and reload the name server.

DNS update

DNS config files are ansible managed on batcave01.

From your local machine run:

git clone ssh://git@pagure.io/fedora-infra/ansible.git
cd ansible/roles/dns/files/
...make changes needed...
git commit -m "What you did"
git push

It should update within a half hour. You can test the new configs with dig:

dig @ns01.fedoraproject.org fedoraproject.org

Adding a new zone

First name the zone and generate new set of keys for it. Run this on ns01. Note it could take SEVERAL minutes to run:

/usr/sbin/dnssec-keygen -a RSASHA1 -b 1024 -n ZONE c.fedoraproject.org
/usr/sbin/dnssec-keygen -a RSASHA1 -b 2048 -n ZONE -f KSK c.fedoraproject.org

Then copy the created .key and .private files to the private git repo (You need to be sysadmin-main to do this). The directory is private/private/dnssec.

  • add the zone in zones.conf in ansible/roles/dns/files/zones.conf

  • save and commit - but do not push

  • Add zone file to the master subdir in this repo

  • git add and commit the file

  • check the zone by running check-domains

  • if you intend to have this be a dnssec signed zone then you must

    • create a new key:

      /usr/sbin/dnssec-keygen -a RSASHA1 -b 1024 -n ZONE $domain.org
/usr/sbin/dnssec-keygen -a RSASHA1 -b 2048 -n ZONE -f KSK $domain.org

- put the files this generates into /srv/privatekeys/dnssec on batcave01
  - edit the do-domains file in this dir and your domain to the
  signed_domains entry at the top
  - edit the zone you just created and add the contents of the .key
  files to the bottom of the zone

If this is a subdomain of fedoraproject.org:

  • run dnssec-dsfromkey on each of the .key files generated

  • paste that output into the bottom of fedoraproject.org.template

  • commit everything to the dns tree

  • push your changes

  • push your changes to the ansible repo

  • test

If you add a new child zone, such as c.fedoraproject.org or vpn.fedoraproject.org you will also need to add the contents of dsset-childzone.fedoraproject.org (for example), to the main fedoraproject.org zonefile, so that DNSSEC has a valid trust path to that zone.

You also must set the NS delegation entries near the top of fedoraproject.org zone file these are necessary to keep dnssec-signzone from whining with this error msg:

dnssec-signzone: fatal: 'xxxxx.example.com': found DS RRset without NS RRset

Look for the: "vpn IN NS" records at the top of fedoraproject.org and copy them for the new child zone.

GeoDNS

As part of our Content Distribution Network we use geodns for certain zones. At the moment just fedoraproject.org and *.fedoraproject.org zones. We’ve got proxy servers all over the US and in Europe. We are now sending users to proxy servers that are near them. The current list of available 'zone areas' are:

  • DEFAULT

  • EU

  • NA

DEFAULT contains all the zones. So someone who does not seem to be in or near the EU, or NA would get directed to any random set. (South Africa for example doesn’t get directed to any particular server).

Don’t forget to increase the serial number in the fedoraproject.org zone file. Even if you’re making a change to one of the geodns IPs. There is only one serial number for all setups and that serial number is in the fedoraproject.org zone.

Non geodns fedoraproject.org IPs If you’re adding as server that is just in one location, and isn’t going to get geodns balanced. Just add that host to the fedoraproject.org zone.

Adding and removing countries

Our setup actually requires us to specify which countries go to which servers. To do this, simply edit the named.conf file in ansible. Below is an example of what counts as "NA" (North America).:

view "NA" {
       match-clients { US; CA; MX; };
       recursion no;
       zone "fedoraproject.org" {
               type master;
               file "master/NA/fedoraproject.org.signed";
       };
       include "etc/zones.conf";
};

IP Country Mapping

The IP → Location mapping is done via a config file that exists on the dns servers themselves (it’s not ansible controlled). The file, located at /var/named/chroot/etc/GeoIP.acl is generated by the GeoIP.sh script (that script is in ansible).

This is known to be a less efficient means of doing geodns than the patched version from kernel.org. We’re using this version at the moment because it’s in Fedora and works. The level of DNS traffic we see is generally low enough that the inefficiencies aren’t that noticed. For example, average load on the servers before this geodns was .2, now it’s around .4

resolv.conf

In order to make the network more transparent to the admins, we do a lot of search based relative names. Below is a list of what a resolv.conf should look like.

Any machine that is not on our vpn or has not yet joined the vpn should NOT have the vpn.fedoraproject.org search until after it has been added to the vpn (if it ever does)
iad2
search iad2.fedoraproject.org vpn.fedoraproject.org fedoraproject.org
iad2 in the QA network
search qa.fedoraproject.org vpn.fedoraproject.org iad2.fedoraproject.org fedoraproject.org
Non-iad2
search vpn.fedoraproject.org fedoraproject.org

The idea here is that we can, when need be, setup local domains to contact instead of having to go over the VPN directly but still have sane configs. For example if we tell the proxy server to hit "app1" and that box is in iad2, it will go directly to app1, if its not, it will go over the vpn to app1.

Want to help? Learn how to contribute to Fedora Docs ›