Consul
Forward DNS for Consul service discovery
By default, DNS is served from port 53. On most operating systems, this requires elevated privileges. Rather than running Consul with an administrative or root account, you can forward appropriate queries to Consul (running on an unprivileged port) from another DNS server or port redirect.
In this tutorial, you will learn how to configure the following DNS servers for DNS forwarding.
After configuring forwarding, you will test the configuration. Finally, you will learn how to troubleshoot some common errors.
Prerequisites
To complete this tutorial, you will need a running Consul agent. Consul should be running with default settings and serving DNS on port 8600.
By default, Consul does not resolve DNS records outside the .consul.
zone unless the
recursors configuration option
has been set. For example, if a Consul DNS reply includes a CNAME record pointing outside the .consul
top level domain (TLD), then the DNS reply will only include CNAME records by default. When recursors
is set and the upstream resolver is functioning correctly, however, Consul will try to resolve CNAMEs and include any records (e.g., A, AAAA, PTR) for them in its DNS reply.
For BIND, dnsmasq and Unbound, you will need to configure /etc/resolv.conf
with a nameserver entry poining to localhost
. This configures the operating system to forward the request to the chosen DNS daemon.
/etc/resolv.conf
nameserver 127.0.0.1
systemd-resolved setup
You can use the systemd-resolved
service to resolve local application names in your network. To use the service, configure systemd-resolved
to send .consul
domain queries to Consul by creating consul.conf
file located in the /etc/systemd/resolved.conf.d/ directory.
Add the following settings to your resolved configuration file:
/etc/systemd/resolved.conf.d/consul.conf
[Resolve]
DNS=127.0.0.1
DNSSEC=false
Domains=~consul
The main limitation with this configuration is that systemd 245 and older does not support configuring port numbers in the DNS field. You can either configure Consul to listen on port 53
instead of 8600, or map port 53 to 8600 using iptables
.
Binding to port 53 usually requires running systemd-resolved
as a privileged user or running Linux with the
CAP_NET_BIND_SERVICE capability. If you are using the Consul Docker image, then you will need to add the following to the environment to allow Consul to use the port: CONSUL_ALLOW_PRIVILEGED_PORTS=yes
The following iptables
commands are sufficient to map the ports.
iptables --table nat --append OUTPUT --destination localhost --protocol udp --match udp --dport 53 --jump REDIRECT --to-ports 8600
iptables --table nat --append OUTPUT --destination localhost --protocol tcp --match tcp --dport 53 --jump REDIRECT --to-ports 8600
The above configuration assumes Consul's DNS server is listening on the loopback address. If Consul is not listening on the loopback IP, replace the references to 'localhost' and '127.0.0.1' with the appropriate IP address for your environment.
Note
PTR record queries will still be sent out to the other configured resolvers, in addition to Consul.
After creating the resolved configuration, restart systemd-resolved
.
# systemctl restart systemd-resolved
Validate the systemd-resolved configuration
Validate that systemd-resolved has restarted and is configured to forward queries to Consul.
# systemctl is-active systemd-resolved
active
# resolvectl domain
Global: ~consul
Link 2 (eth0):
# resolvectl query consul.service.consul
consul.service.consul: 127.0.0.1
-- Information acquired via protocol DNS in 6.6ms.
-- Data is authenticated: no
Confirm that /etc/resolv.conf
points to the stub-resolv.conf
file managed by
systemd-resolved, and that the IP address for systemd-resolved's stub resolver
is the configured nameserver
.
$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 37 Aug 20 22:50 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0
Ensure that the operating system can resolve DNS queries to the .consul
domain.
$ host consul.service.consul
consul.service.consul has address 127.0.0.1
Using any local resolver with systemd
By default, the local resolver stub in the resolved.conf
file is configured to listen for UDP and TCP requests at 127.0.0.53:53, but you can set the DNSStubListener
option to false
, which disables the stub. As a result, your system will be able to use any DNS configuration as long as it loads earlier than resolved
.
/etc/systemd/resolved.conf
DNSStubListener=false
Disabling the local resolver stub can also solve other DNS configuration issues. See Troubleshooting for additional information.
BIND setup
In this example, BIND and Consul are running on the same machine.
Note
If your distribution uses systemd, you may need to disable systemd-resolved
prior to following this example. Refer to the guidance in the systemd-resolved instructions for additional information.
First, disable DNSSEC so that Consul and BIND can communicate. The following example shows a BIND configuration with DNSSEC disabled.
/etc/named.conf
options {
listen-on port 53 { 127.0.0.1; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
allow-query { localhost; };
recursion yes;
dnssec-enable no;
dnssec-validation no;
/* Path to ISC DLV key */
bindkeys-file "/etc/named.iscdlv.key";
managed-keys-directory "/var/named/dynamic";
};
include "/etc/named/consul.conf";
Zone file
Next, set up a zone for your Consul managed records in consul.conf
.
/etc/named/consul.conf
zone "consul" IN {
type forward;
forward only;
forwarders { 127.0.0.1 port 8600; };
};
Dnsmasq setup
Note
If your distribution uses systemd, you may need to disable systemd-resolved
prior to following this example. Refer to the guidance in the systemd-resolved instructions for additional information.
Dnsmasq is typically configured via a dnsmasq.conf
or a series of files in
the /etc/dnsmasq.d
directory. Add the following settings to your dnsmasq configuration file
(e.g., /etc/dnsmasq.d/10-consul
):
/etc/dnsmasq.d/10-consul
# Enable forward lookup of the 'consul' domain:
server=/consul/127.0.0.1#8600
# Uncomment and modify as appropriate to enable reverse DNS lookups for
# common netblocks found in RFC 1918, 5735, and 6598:
#rev-server=0.0.0.0/8,127.0.0.1#8600
#rev-server=10.0.0.0/8,127.0.0.1#8600
#rev-server=100.64.0.0/10,127.0.0.1#8600
#rev-server=127.0.0.1/8,127.0.0.1#8600
#rev-server=169.254.0.0/16,127.0.0.1#8600
#rev-server=172.16.0.0/12,127.0.0.1#8600
#rev-server=192.168.0.0/16,127.0.0.1#8600
#rev-server=224.0.0.0/4,127.0.0.1#8600
#rev-server=240.0.0.0/4,127.0.0.1#8600
Restart the dnsmasq
service after creating the configuration.
You can configure additional settings in dnsmasq
that may be useful in your environments. See
dnsmasq(8)
for additional details.
/etc/dnsmasq.conf
# Accept DNS queries only from hosts whose address is on a local subnet.
#local-service
# Don't poll /etc/resolv.conf for changes.
#no-poll
# Don't read /etc/resolv.conf. Get upstream servers only from the command
# line or the dnsmasq configuration file (see the "server" directive below).
#no-resolv
# Specify IP address(es) of other DNS servers for queries not handled
# directly by consul. There is normally one 'server' entry set for every
# 'nameserver' parameter found in '/etc/resolv.conf'. See dnsmasq(8)'s
# 'server' configuration option for details.
#server=1.2.3.4
#server=208.67.222.222
#server=8.8.8.8
# Set the size of dnsmasq's cache. The default is 150 names. Setting the
# cache size to zero disables caching.
#cache-size=65536
Unbound setup
Note
If your distribution uses systemd, you may need to disable systemd-resolved
prior to following this example. Refer to the guidance in the systemd-resolved instructions for additional information.
Unbound is typically configured in a unbound.conf
file or a series of files in
the /etc/unbound/unbound.conf.d
directory. Add the following settings to your Unbound configuration file
(e.g., /etc/unbound/unbound.conf.d/consul.conf
):
/etc/unbound/unbound.conf.d/consul.conf
#Allow insecure queries to local resolvers
server:
do-not-query-localhost: no
domain-insecure: "consul"
#Add consul as a stub-zone
stub-zone:
name: "consul"
stub-addr: 127.0.0.1@8600
You may have to add the following line to the bottom of your
/etc/unbound/unbound.conf
file for the new configuration to be included.
/etc/unbound/unbound.conf
include: "/etc/unbound/unbound.conf.d/*.conf"
iptables setup
The rules must be set on the same host as the Consul instance. Relay hosts should not be on the same host, otherwise the redirects will intercept the traffic.
On Linux systems that support it, incoming requests and requests to
the local host can use iptables
to forward to ports on the same machine
without using a secondary service. The recursors
flag is especially important for resolving other domains with iptables
.
This is because Consul only resolves the .consul
TLD by default.
Recursors should not include the local host address because the iptables
redirects would intercept the requests.
The iptables
method is suitable for situations where an external DNS
service is already running in your infrastructure and is used as the
recursor. It is also a suitable method if you want to use an existing DNS server as your query
endpoint and forward requests for the consul
domain to the Consul
server. In both cases, you may want to query the Consul server without the overhead of running a separate service on the Consul host.
iptables -t nat -A PREROUTING -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d localhost -p udp -m udp --dport 53 -j REDIRECT --to-ports 8600
iptables -t nat -A OUTPUT -d localhost -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 8600
macOS setup
On macOS systems, you can use the macOS system resolver to point all .consul
requests to Consul.
Documentation for this feature is available via: man 5 resolver
.
You must add a resolver entry in /etc/resolver/
to point at Consul. If you do not have this folder, create it.
To configure a resolver entry, you will need sudo/root access. Create a new file /etc/resolver/consul
,
and add the following text to the file:
/etc/resolver/consul
nameserver 127.0.0.1
port 8600
The configuration informs the macOS resolver daemon to forward all .consul
TLD requests to 127.0.0.1 on port 8600.
Query Consul DNS and BIND
First, add a service registration to your Consul agent. You can use the following example service configuration file and registration command.
redis.json
{
"service": {
"name": "redis",
"port": 80
}
}
$ curl --data @redis.json http://127.0.0.1:8500/v1/catalog/register
To ensure that Consul DNS and your DNS server are configured correctly, use a dig
query for the service you've registered.
$ dig @localhost -p 8600 primary.redis.service.dc1.consul. A
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.32.amzn1 <<>> @localhost primary.redis.service.dc1.consul. A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11536
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;primary.redis.service.dc1.consul. IN A
;; ANSWER SECTION:
primary.redis.service.dc1.consul. 0 IN A 172.31.3.234
;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Apr 9 17:36:12 2014
;; MSG SIZE rcvd: 76
Next, run the same query against your BIND instance and make sure you get a valid result:
$ dig @localhost -p 53 primary.redis.service.dc1.consul. A
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.32.amzn1 <<>> @localhost primary.redis.service.dc1.consul. A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11536
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;primary.redis.service.dc1.consul. IN A
;; ANSWER SECTION:
primary.redis.service.dc1.consul. 0 IN A 172.31.3.234
;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Apr 9 17:36:12 2014
;; MSG SIZE rcvd: 76
If desired, verify reverse DNS using the same method:
$ dig @127.0.0.1 -p 8600 133.139.16.172.in-addr.arpa. PTR
; <<>> DiG 9.10.3-P3 <<>> @127.0.0.1 -p 8600 133.139.16.172.in-addr.arpa. PTR
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3713
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;133.139.16.172.in-addr.arpa. IN PTR
;; ANSWER SECTION:
133.139.16.172.in-addr.arpa. 0 IN PTR consul1.node.dc1.consul.
;; Query time: 3 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sun Jan 31 04:25:39 UTC 2016
;; MSG SIZE rcvd: 109
$ dig @127.0.0.1 -p 8600 +short -x 172.16.139.133
consul1.node.dc1.consul.
Troubleshooting
If you don't get an answer from your DNS server (e.g., BIND, Dnsmasq) but you do get an answer from Consul, turn on your DNS server's query log to check for errors.
For BIND:
rndc querylog
tail -f /var/log/messages
The log may show errors like this:
error (no valid RRSIG) resolving
error (no valid DS) resolving
This indicates that DNSSEC is not disabled properly.
If you see errors about network connections, verify that there are no firewall or routing problems between the servers running BIND and Consul.
For Dnsmasq, see the log-queries
configuration option and the USR1
signal.
DNS forwarding may fail if you use the default systemd-resolved
configuration and attempt to bind to 0.0.0.0. The default configuration uses a DNS stub that listens for UDP and TCP requests at 127.0.0.53. As a result, attempting to bind to 127.0.0.53 conflicts with the running stub. You can disable the stub as described in the Using any local resolver with systemd section to troubleshoot this problem.
Next steps
In this tutorial, you configured DNS forwarding for Consul DNS and your DNS server. Now that you have configured your environment, you can start using Consul for service discovery or service mesh.
- Ensure only healthy service instances are available in Consul DNS.
- Deploy services into a zero trust network with Consul service mesh.