If we were to ask a dozen senior system admins to check all their mission-critical systems, most of them would start with checking their DNS servers. Regardless of the size of the server farm, if the DNS fails, everything will go down in a matter of seconds. In this guide, we will start with the basics and explore the behaviors of the DNS system. This guide is about designing a highly available and secure DNS cluster to support millions of users. The guide includes related configuration parameters but does not include step-by-step instructions.

This documentation is focused on the BIND DNS server, yet most of the practices apply to all DNS servers.

1. Domain Name Resolution System

Before we dive into technicality, we must get a few concepts out of the way. This section explains the basics of the DNS system and could be safely skipped.

1.1 Glossary and Concepts

• DNS server: A server that resolves DNS queries. It translates domain names to IP addresses. Sometimes this lookup can be in reverse.
• Primary DNS server: Master server, you can read and write DNS records on this server.
• Secondary DNS server: Slave server, you can only read from this server, updates are replicated from the primary server.
• Fully Qualified Domain Name (FQDN): This is also known as the absolute domain name. This is a complete domain address like mail.google.com, www.root101.net or subdomain.domain.net. Domain needs to specify an exact location to be FQDN.
• Forward Lookup: Client asks the DNS server for a domain name, the server responds with IP. One to one relationship.
• Reverse Lookup: Client asks the DNS server for an IP, server responds with domain name(s). One to many relationships. This is also the reason for the slow SSH login times. (LINK)

• DNS Zone: A distinct part of the domain namespace, separated for easier management.

  • Primary DNS Zone: Master zone, which you can read from and write to
  • Secondary DNS Zone: Slave zone, which you can read from but can’t write to
  • Stub Zone: This is often used to make DNS servers of different organizations resolve
• Top Level Domain (TLD): Last segments of domains, .com, .org, .net, .gov, .edu are most popular ones.

1.2 How Does DNS Protocol Works?

DNS is the spine of the World Wide Web (WWW). Thousands of DNS servers around the world work together to respond to your DNS queries. To make every server in the system play nice with each other, a hierarchy has been developed. This is also sometimes called the DNS tree. Typical flow of a DNS query is like:

1. When an application wants to interact with a web URL, it needs to translate the domain name to an IP. Application makes a DNS request
2. Local cache is checked for a possible reply, no-cache found. The request goes to the DNS resolver server.
3. Request is received by a DNS resolver server, cache is checked for the answer.
4. Resolver follows the domain hierarchy( aka chain of authority ) to find the authoritative server.
5. Resolver asks the query to the auth server. Authoritative server it responds resolver and resolver responds to client.

But how does the DNS resolver server finds the correct authoritative server for every domain?

Let's assume that you wish to visit subdomain.root101.net. (Assume this is a completely new domain name, there no-cache on your computer or your DNS server) Your computer asks your DNS server for the IP of the website.

• Your browser asks your DNS resolver where is subdomain.root101.net. ? (the last dot is not a typo, even though you don’t type this, it gets appended to end).

• Since there is no cache resolver start the search for the authoritative server from left to right. When split into parts this domain becomes: subdomain - root101 - net - .

  • Resolver asks authority server of . : What is the address authority of net
  • Resolver asks authority server of net : What is the address authority of root101
  • Resolver asks authority server of root101 : What is the address of subdomain
• Resolver repeats this response to the client.

As an interesting side note, there are only 13 IP addresses for the dot zone ( usually called the root name servers). This is because of the system design and IPv4 limitations. But this does not mean 13 servers, these are load balancer IP addresses and they are backed by thousands of servers.

1.3 DNS Server Types / DNS Roles

Productions systems often have two types of DNS servers, one for internal users and one for external users.

These are more like roles, any virtual server can carry all the roles(don't do this), or each of the roles could be distributed over a dozen bare-metals if the system requires such power.

1.3.1 Recursive DNS Server

This is also called internal DNS or internal NS. Duty of this role is the solve DNS queries of the internal clients. When a client is configured to use this DNS server, it makes all of the domain name queries to this server. In the example below, this is the first server that the local machine asks for DNS if it is not in the local cache.

When a request arrives, these servers look its cache for possible responses. If a response not found in its cache, it asks the domain to related domain authority and caches this response. It has a pre-configured TTL (time-to-live) value when TTL expires cache gets purged.

1.3.2 Authoritative DNS Server

This is also known as external DNS/NS. This server contains the master zone records for certain domains. For a given domain name, this server is considered the most accurate source of DNS information. All caches around the world are done according to the reading from this server.

2. System Design

In this section, we will go over tips and best practices.

2.1 Topology

2.1.1 Use Dedicated Servers

As aforementioned, DNS is the most critical asset of usual ecosystems. Even though DNS servers are not very resource hungry they should be isolated from other software. There is no saving resources when it comes to DNS. Use dedicated virtual servers or bare-metal depending on your needs.

2.1.2 Run Separate Authoritative and Recursive DNS Servers

Even though bind can support hybrid modes, it is advised to split authoritative and recursive servers. If we put both roles on the same server, external attacks could affect your internal users and vice versa.

2.1.3 Split the Master Servers From Slaves

Most of the DNS solutions use master-slave design for High Availability. We chose a single node as the master, and we update the DNS records on the master. The changes get replicated to slaves with zone transfers.

Master can resolve DNS requests just like slaves. A common mistake is to use the master for resolving queries. This approach saves a few vCPUs, yet it can easily cost a good night's sleep. Don't allow the master to respond to daily requests. Master has more critical tasks than serving clients. All-access to the master server should be heavily restricted.

2.1.3 Don't Run Open Resolvers

Don’t run open resolvers, they could be used to attack other DNS servers. Use access control to allow only certain clients.

2.2 Software

2.2.1 Chose Supported Versions

Bind has many versions with different EOL dates. Choose wisely for your update cycle. Bind has a system for its releases.

• Major branches go like 9.12, 9.13, 9.14.
• Minor versions go like 9.12.1, 9.12.2, 9.12.3.
• Odd-numbered major versions are development versions. 9.13 was the first of this trend.
• Even-numbered major versions are stable versions like 9.12.

Stable versions have a yearly update cycle. Systems with longer update cycles usually prefer Extended Support Version. The current ESV is 9.11 (this is not a development version, development version rule came after 9.13) and has planned support until 2021. Next ESV is planned to be 9.16. Versions ending with -S like 9.10-S, 9.11-S are stable preview versions, which have special paid support options. “These releases provide a deployable, supported vehicle for our support customers to have early access to new features in development”

2.3 Routing and High Availability

There are two methods which are often used for these purposes.

2.3.1 Load balancers

For most systems, we use load balancers to scale our system and add high availability. They are fine for mid-size ecosystems, yet if your clients are geographically distributed you might benefit from providing them with optimal routing.

More capable load balancer solutions support geographic load balancing, which utilizes methods like geofencing or geotargeting to provide clients with better routes based on their locations. These routes are based on rules and they usually cannot detect changes in these routes without admin interaction. In this manner, anycast is slightly better at routing. Yet modern load balancers are packed with a lot of features that could not be matched with anycast. They are also easier to manage.

2.3.2 Anycast

This is a special traffic routing algorithm used for speedy delivery. This simply finds the fastest route between the client and the server. Using Anycast :

• Round-trip-time will be minimized. The number of hops is reduced. Better than load balancers
• High availability , multiple IPs will be advertised. Auto failover at the network level.
• DDOS is mitigated to a degree.
• Downtimes are mitigated much better.
• Simplified server-side configuration, yet harder to maintain on the network side.

2.4 Performance

2.4.1 Configure IPv6

All of us will be using it sooner or later, but we are not there yet. Check to make sure there is no IPv6 configuration on the device unless it is required. This configuration comes default with most distributions. If you don't do this, BIND will also try to work with IPv6. This attempt will fail quietly yet, named waits for this fail, which slows down slows responses.

2.4.2 Check network support

Make sure your infrastructure supports EDNS0 and large UDP packet sizes. Don’t let firewalls/packet filters slow down the outgoing queries of recursive servers.

2.4.3 minimal-responses

This option can reduce the amount of work required to respond to a query.

2.4.4 firewalls

Firewalls are essential and they keep us safe. But pushing every byte of traffic trough the firewall is not a good idea. Choose what should go through the firewall and what should not. Unnecessary traffic through the firewall means more latency in queries. Having even little extra latency in DNS means every application in your server pool with work little slower. Also, high end firewall licenses are not cheap.

2.4.5 Traffic Buffers

It is a good idea to allow EDNS and large UDP responses up to 4096 If you are dealing with large query responses, ensure that your network infrastructure allows reassembly of fragmented UDP packets.

3. Security

3.1 Attack Vectors

If you were to defend your systems from mischiefs of scripts kiddies and surgical attacks of the hackers we must learn about possible attack vectors.

These are only DNS specific attacks, universal attacks like buffer overflows and MITM are not considered in this list.

3.1.1 DNS flood

This is the most basic DNS attack. Just send a lot of requests to the DNS server. If you can send enough, you can overload the server. There is not a lot you can do on the DNS server to prevent this kind of attacks. You can put a security appliance or use a specialized DDOS prevention service to protect your servers from such attacks.

3.1.2 DNS reflection attack

This is a type of DNS flood. Perpetrator uses misconfigured servers of the company A to attack company B. Even though company B is the main target, company A will also suffer performance lose. This why we should always disable the recursion of public-facing servers.

When a perpetrator starts this attack, they send forged DNS queries to an open DNS resolver. Fake thing about this package is the return address. When a client asks a DNS response is sent back to the client. In this method, responses are sent to another server.

3.1.3 DNS amplification attack

This is a type of reflection attack. Special thing about this, perpetrators use protocol extensions like EDNS0 or DNSSEC to increase the size of the reply. When a perpetrator leverages these extensions, a 70-80 byte request can create a 3000-4000 byte response which is used to flood the target server in company B. This is why its called amplification attack. DNS rate limiting can be used to prevent attackers from using your DNS servers to attack other DNS servers.

3.1.4 DNS cache poisoning

When an attacker infects a recursive DNS server and manipulates its cache with wrong entries it's called cache poisoning. When a client uses these recursive servers, the client will be redirected into a fake site. Attackers can gather information and grab credentials using this method. DNSSEC is developed to sign DNS records. This makes it harder to poison DNS caches.

3.2 Security Best Practices

3.2.1 Hide BIND version

If an attacker can figure out the version of your DNS server, finding possible exploits and buffer overflow would be very easy for them. So we must hide our DNS server version from the world.

To do this just go to the named.conf. In the options { … }; set version "xxx"; to something like version "denied";. I usually set the version to an older version. This creates a rabbit hole for the enemy.

3.2.2 Restrict Zone Transfers

Zone transfer is the method used by the slaves to update their records. This is essential if you plan to run more than one server. Yet you must be very strict about the name transfer permissions. When a hacker attacks your ecosystem, most of the time they start with enumeration. They would put serious effort into discovering your DNS records. They might listen to the traffic passing through the system. They might utilize brute-forcing and URL enumeration which might trigger one of your security appliances. But if an attacker can complete even a single zone transfer request, he or she will have the whole map of your ecosystem with very little effort. This is harder to detect than brute-forcing.

3.2.3 Block everything that is not required.

If an authoritative server is configured correctly, it only responds to queries. It never asks anything to external servers. Enforce this behavior with firewall rules. Monitor

4. Stress Testing

Best way to make sure our server works fine is to put some stress on it. To make a proper test, you need to use an extra server. If you run the stress test on the same server, resources used by the test will slow down the Bind. To create stress on a DNS server, there two popular tools.

How to get the list:

/usr/sbin/rndc dumpdb -zones
cd /var/named/data/
cat cache_dump.db  | cut -f1  | grep gov.tr | grep -v Zone | sort | uniq | cut -f 1 -d" " | sed -r 's/\.$/ A/g'

4.1 dnsperf

dnsperf is much older than the other alternative. It can create decent stress on your servers and send dynamic update messages instead of queries if required to test dynamic update performance. This tool is usually enough to test an installation.

4.1.1 Compilation

Sadly, its missing in the official repos of both Centos 7 and Debian 10. To compile on debian 10 following commands were used.

apt-get install -y  libbind-dev libkrb5-dev libssl-dev libcap-dev libxml2-dev libjson-c-dev libgeoip-dev libprotobuf-c-dev libfstrm-dev liblmdb-dev libssl-dev pkg-config dh-autoreconf
git clone https://github.com/DNS-OARC/dnsperf
cd dnsperf/
./autogen.sh
./configure
make
make install
dnsperf -h
DNS Performance Testing Tool
Version 2.3.2

By default, dnsperf accepts queries from standard input. To make proper test, you need to create a file full of queries. Following file format is accepted:

google.com A
facebook.com A
root101.net A
test.it A

4.1.2 Usage

dnsperf -s 192.168.1.1 -d /home/tester/queries.txt -c 100 -l 60 -S 10
output here

There are few interesting parameters for the dnsperf:

• -d: path the the input file
• -s : address of the target DNS server
• -c : clients, number of different clients to simulate when doing the requests
• -l : how many seconds the test will last. Unless specified, file is queries once.
• -n: repeat the file certain times if time is not specified
• -Q : limit the maximum requests per second
• -S : statistics will be show every time this many seconds pass. Great for monitoring the test

4.2 flamethrower

flamethrower is a newer alternative to the dnsperf. It offer much more advanced tests and extra capabilities.

• supports alternative source ports and source port distribution
• better fit for CI/CD work-flow
• batch query support
• random and rule based query generator
• random binary garbage generation ( much like fuzzing )
• source address spoofing
• query jitter capability
• json and pcap exports

This tool is better suited for custom test.