DNS not replicating from master to slave

Installing, Configuring, Troubleshooting server daemons such as Web and Mail
codenjanod
Posts: 7
Joined: 2010/01/27 10:53:12
Contact:

DNS not replicating from master to slave

Post by codenjanod » 2010/01/27 11:17:59

Hi all,

I have set up a master BIND DNS server with 2 slaves.
All the services start up fine on all 3 servers, but zones and named.conf info is not being replicated.
The 3 servers are tentatively installed on a Xen virtual server for testing purposes.
All 3 servers are 64bit and installed with:
-> CentOS release 5.4 (Final) - 2.6.18-164.10.1.el5xen
-> BIND 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2

All 3 servers are fully updated.
All 3 servers have static IP addresses.

The services on all 3 servers start up without any apparent issues.

[b]Master server:[/b]
# /etc/init.d/named restart ; tail -f /var/log/messages
[quote]Stopping named: [ OK ]
Starting named: [ OK ]
Jan 27 13:02:29 ns1 named[6103]: using default UDP/IPv6 port range: [1024, 65535]
Jan 27 13:02:29 ns1 named[6103]: listening on IPv4 interface lo, 127.0.0.1#53
Jan 27 13:02:29 ns1 named[6103]: listening on IPv4 interface eth0, 10.2.37.202#53
Jan 27 13:02:29 ns1 named[6103]: command channel listening on 127.0.0.1#953
Jan 27 13:02:29 ns1 named[6103]: command channel listening on ::1#953
Jan 27 13:02:29 ns1 named[6103]: zone 0.168.192.in-addr.arpa/IN: loaded serial 100
Jan 27 13:02:29 ns1 named[6103]: zone yourdomain.com/IN: loaded serial 100
Jan 27 13:02:29 ns1 named[6103]: running
Jan 27 13:02:29 ns1 named[6103]: zone yourdomain.com/IN: sending notifies (serial 100)
Jan 27 13:02:29 ns1 named[6103]: zone 0.168.192.in-addr.arpa/IN: sending notifies (serial 100)[/quote]

[b]Slave 1:[/b]
# /etc/init.d/named restart ; tail -f /var/log/messages
[quote]Stopping named: [ OK ]
Starting named: [ OK ]
Jan 27 12:30:47 ns2 named[5965]: found 2 CPUs, using 2 worker threads
Jan 27 12:30:47 ns2 named[5965]: using up to 4096 sockets
Jan 27 12:30:47 ns2 named[5965]: loading configuration from '/etc/named.conf'
Jan 27 12:30:47 ns2 named[5965]: using default UDP/IPv4 port range: [1024, 65535]
Jan 27 12:30:47 ns2 named[5965]: using default UDP/IPv6 port range: [1024, 65535]
Jan 27 12:30:47 ns2 named[5965]: listening on IPv4 interface lo, 127.0.0.1#53
Jan 27 12:30:47 ns2 named[5965]: listening on IPv4 interface eth0, 10.2.37.214#53
Jan 27 12:30:47 ns2 named[5965]: command channel listening on 127.0.0.1#953
Jan 27 12:30:47 ns2 named[5965]: command channel listening on ::1#953
Jan 27 12:30:47 ns2 named[5965]: running[/quote]

[b]Slave 2:[/b]
# /etc/init.d/named restart ; tail -f /var/log/messages
[quote]Stopping named: [ OK ]
Starting named: [ OK ]
Jan 27 12:30:45 ns3 named[5874]: found 2 CPUs, using 2 worker threads
Jan 27 12:30:45 ns3 named[5874]: using up to 4096 sockets
Jan 27 12:30:45 ns3 named[5874]: loading configuration from '/etc/named.conf'
Jan 27 12:30:45 ns3 named[5874]: using default UDP/IPv4 port range: [1024, 65535]
Jan 27 12:30:45 ns3 named[5874]: using default UDP/IPv6 port range: [1024, 65535]
Jan 27 12:30:45 ns3 named[5874]: listening on IPv4 interface lo, 127.0.0.1#53
Jan 27 12:30:45 ns3 named[5874]: listening on IPv4 interface eth0, 10.2.37.225#53
Jan 27 12:30:45 ns3 named[5874]: command channel listening on 127.0.0.1#953
Jan 27 12:30:45 ns3 named[5874]: command channel listening on ::1#953
Jan 27 12:30:45 ns3 named[5874]: running[/quote]

The configuration files are like follows:

[b]Master server:[/b]
# cat /var/named/chroot/etc/named.conf
[code]options {
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
allow-transfer { 10.2.37.214; 10.2.37.225; };
};

key "rndc-key" {
algorithm hmac-md5;
secret "toS8KHNJ2p2pDDWnW5pDSA==";
};

zone "yourdomain.com" IN {
type master;
file "/var/named/yourdomain.com.zone";
allow-update { none; };
};

zone "0.168.192.in-addr.arpa" IN {
type master;
file "/var/named/0.168.192.rev";
allow-update { none; };
};[/code]


[b]Slave servers(both are the same):[/b]
# cat /var/named/chroot/etc/named.conf
[code]options {
allow-query { 10.2.37.202; 127.0.0/8; };
allow-transfer { 10.2.37.202; };
transfer-format many-answers;
notify no;

};[/code]



Any help or pointers will be appreciated.


Thanks

milosb
Posts: 661
Joined: 2009/01/18 00:39:15
Location: 44 49′14″N 20 27′44″E

Re: DNS not replicating from master to slave

Post by milosb » 2010/01/27 14:13:19

Well one thing you are clearly missing in [b]named.conf[/b] of your slave DNS servers, are slave zone declarations along with location where to commit this data (and retrieve it from afterward).

Your slaves' [b]named.conf[/b] need to contain zone declaration like:
[b]
zone "yourdomain.com" IN {
type slave;
masters { 10.2.37.202; };
file "slaves/yourdomain.com.zone";
};
[/b]
This way your slaves will create zone data-files in [b]/var/named/chroot/var/named/slaves/[/b] folder with appropriate SE Linux context. You can also check if your zone transfers really work from your slave servers with the following command:
[b]
dig -t axfr yourdomain.com 10.2.37.202
[/b]

BTW, your system is NOT fully updated - you're running an old kernel. The latest one is: [b]2.6.18-164.11.1.el5[/b].

EDIT: Also, it'd be a good idea to get a good reference on setting up BIND DNS server, like O'Reilly Media's: DNS & BIND.

codenjanod
Posts: 7
Joined: 2010/01/27 10:53:12
Contact:

Re: DNS not replicating from master to slave

Post by codenjanod » 2010/01/28 06:49:18

Thank you very much for the assistance.

I did what you suggested and the servers started replicating.
But it seems I am still missing something.

[b]Master Server:[/b]
# tail -f /var/log/messages
[code]
Jan 28 07:57:37 ns1 named[1465]: client 10.2.37.214#50135: transfer of 'yourdomain.com/IN': AXFR started
Jan 28 07:57:37 ns1 named[1465]: client 10.2.37.214#50135: transfer of 'yourdomain.com/IN': AXFR ended

Jan 28 07:57:44 ns1 named[1465]: client 10.2.37.225#52508: transfer of 'yourdomain.com/IN': AXFR started
Jan 28 07:57:44 ns1 named[1465]: client 10.2.37.225#52508: transfer of 'yourdomain.com/IN': AXFR ended
[/code]

But now the slave servers are complaining again.

[b]Slave 1:[/b]
# tail -f /var/log/messages
[code]
Jan 28 07:57:37 ns2 named[1445]: zone yourdomain.com/IN: Transfer started.
Jan 28 07:57:37 ns2 named[1445]: transfer of 'yourdomain.com/IN' from 10.2.37.202#53: connected using 10.2.37.214#50135
Jan 28 07:57:37 ns2 named[1445]: dumping master file: slaves/tmp-9PbqiSu0dz: open: file not found
Jan 28 07:57:37 ns2 named[1445]: transfer of 'yourdomain.com/IN' from 10.2.37.202#53: failed while receiving responses: file not found
Jan 28 07:57:37 ns2 named[1445]: transfer of 'yourdomain.com/IN' from 10.2.37.202#53: end of transfer
[/code]

[b]Slave 2:[/b]
# tail -f /var/log/messages
[code]
Jan 28 07:57:44 ns3 named[1466]: zone yourdomain.com/IN: Transfer started.
Jan 28 07:57:44 ns3 named[1466]: transfer of 'yourdomain.com/IN' from 10.2.37.202#53: connected using 10.2.37.225#52508
Jan 28 07:57:44 ns3 named[1466]: dumping master file: slaves/tmp-9r9qwhSwSj: open: file not found
Jan 28 07:57:44 ns3 named[1466]: transfer of 'yourdomain.com/IN' from 10.2.37.202#53: failed while receiving responses: file not found
Jan 28 07:57:44 ns3 named[1466]: transfer of 'yourdomain.com/IN' from 10.2.37.202#53: end of transfer
[/code]


When I test from the 2 slave server with your provided test command:
# dig -t axfr yourdomain.com 10.2.37.202

I get the below in the log on the Master server:

[code]
Jan 28 08:12:09 ns1 named[1465]: client 10.2.37.225#37852: transfer of 'yourdomain.com/IN': AXFR started
Jan 28 08:12:09 ns1 named[1465]: client 10.2.37.225#37852: transfer of 'yourdomain.com/IN': AXFR ended
Jan 28 08:12:09 ns1 named[1465]: client 10.2.37.225#49387: bad zone transfer request: '10.2.37.202/IN': non-authoritative zone (NOTAUTH)

Jan 28 08:12:15 ns1 named[1465]: client 10.2.37.214#39761: transfer of 'yourdomain.com/IN': AXFR started
Jan 28 08:12:15 ns1 named[1465]: client 10.2.37.214#39761: transfer of 'yourdomain.com/IN': AXFR ended
Jan 28 08:12:15 ns1 named[1465]: client 10.2.37.214#57885: bad zone transfer request: '10.2.37.202/IN': non-authoritative zone (NOTAUTH)
[/code]


When doing a "netstat -ntpl" on all 3 servers, I get the following:
[code]
tcp 0 0 10.2.37.225:53 0.0.0.0:* LISTEN 1466/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 1466/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 1466/named
tcp 0 0 :::22 :::* LISTEN 1136/sshd
tcp 0 0 ::1:953 :::* LISTEN 1466/named
[/code]


The permissions are as follows on all my directories and files starting from the "/" :
[b]# ll /[/b]
[quote]drwxr-xr-x 21 root root 4096 Jan 26 10:27 var[/quote]

[b]# ll /var/[/b]
[quote]drwxrwxr-x 5 named named 4096 Jan 26 10:27 named[/quote]

[b]# ll /var/named/[/b]
[quote]drwxrwxr-x 6 named named 4096 Jan 26 12:03 chroot
drwxrwxr-x 2 named named 4096 Jan 20 18:31 data
drwxrwxr-x 2 named named 4096 Jan 20 18:31 slaves[/quote]

[b]# ll /var/named/chroot/[/b]
[quote]drwxrwxr-x 2 root named 4096 Jan 26 10:27 dev
drwxrwxr-x 2 named named 4096 Jan 28 07:53 etc
drwxrwxr-x 70 root root 0 Jan 28 07:37 proc
drwxrwxr-x 6 named named 4096 Jan 26 10:27 var[/quote]

[b]# ll /var/named/chroot/etc/[/b]
[quote]-rw-r--r-- 1 root root 245 Jan 14 13:41 localtime
-rwxrwxr-x 1 named named 780 Jan 28 07:53 named.conf
-rwxrwxr-x 1 named named 76 Jan 27 09:54 rndc.key[/quote]

[b]# ll /var/named/chroot/var/[/b]
[quote]drwxrwxr-x 2 named named 4096 Jan 20 18:31 log
drwxrwxr-x 4 root named 4096 Jan 27 15:25 named
drwxrwxr-x 4 named named 4096 Jan 26 12:03 run
drwxrwxr-x 2 named named 4096 Mar 14 2003 tmp[/quote]

[b]# ll /var/named/chroot/var/named/[/b]
[quote]drwxrwxr-x 2 named named 4096 Aug 25 2004 data
-rwxrwxr-x 1 named named 351 Jan 28 08:03 dbs.192.168.0
-rwxrwxr-x 1 named named 413 Jan 28 07:25 dbs.yourdomain.com
drwxrwxr-x 2 named named 4096 Jan 28 08:32 slaves[/quote]


No files are located either in [b]/var/named/chroot/var/named/data[/b] or in [b]/var/named/chroot/var/named/slaves[/b].

I have also disabled the iptables firewall, as that is not difficult to set up and allow port access between the 3 servers.
I just wanted to cut that out of the equation to make sure DNS and replication is working.

Just another question,
If I create a new zone on the master server in "named.conf", as well as the corresponding file in [b]/var/named/chroot/var/named[/b], do I have to manually create it on both the slave servers as well ?
Or is it automated via replication ?
If not automatically created in the slave server's "named.conf", is there a way to automate it ?


Thanks again for the help and assistance.

BTW. I also did an upgrade of all 3 servers, thanks for that.

I will have a look into the literature that you have suggested.

codenjanod
Posts: 7
Joined: 2010/01/27 10:53:12
Contact:

Re: DNS not replicating from master to slave

Post by codenjanod » 2010/02/04 13:02:30

Hi all,

Seems I have a part of my issues resolved.
I carefully went through all my configs and fixed anything and everything that I could find that looked faulty or mistyped or even just left out.

So now the logs show that files are being transferred.

My configs are now as follows:
Master server:

# vi /var/named/chroot/etc/named.conf
[code]
options {
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
allow-transfer { 10.2.37.214; 10.2.37.225; };
notify yes;
};

key "rndc-key" {
algorithm hmac-md5;
secret "toS8KHNJ2p2pDDWnW5pDSA==";
};

controls {
inet 127.0.0.1 port 953
allow { 127.0.0.1; } keys { "rndc-key"; };
};

zone "yourdomain.com" IN {
type master;
file "/var/named/yourdomain.com.zone";
allow-update { 10.2.37.214; 10.2.37.225; };
};

zone "0.168.192.in-addr.arpa" IN {
type master;
file "/var/named/0.168.192.rev";
allow-update { 10.2.37.214; 10.2.37.225; };
};
[/code]


Both slave servers have the same config below:

# vi /var/named/chroot/etc/named.conf
[code]
options {
directory "/var/named";
allow-query { 10.2.37.202; 127.0.0/8; };
allow-transfer { 10.2.37.202; };
transfer-format many-answers;
notify yes;
};

key "rndc-key" {
algorithm hmac-md5;
secret "toS8KHNJ2p2pDDWnW5pDSA==";
};

controls {
inet 127.0.0.1 port 953
allow { 127.0.0.1; } keys { "rndc-key"; };
};

zone "yourdomain.com" IN {
type slave;
file "/var/named/dbs.yourdomain.com";
masters { 10.2.37.202; };
};

zone "0.168.192.in-addr.arpa" {
type slave;
file "/var/named/dbs.192.168.0";
masters { 10.2.37.202; };
};
[/code]


So, I tried to run a test.
So on my Master server, I added the following entry:

# vi /var/named/chroot/etc/named.conf
[code]
zone "test1.com" IN {
type master;
file "/var/named/test1.com.zone";
allow-update { 10.2.37.214; 10.2.37.225; };
};
[/code]


I also added the zone file under:

# ll /var/named/chroot/var/named/test1.com.zone
[quote]
-rw-r--r-- 1 root root 325 Feb 4 14:30 /var/named/chroot/var/named/test1.com.zone
[/quote]


I update the serial of the above zone file.
I restart the service on all the DNS servers.

But the new "test1" entry is not transfered to either of the slave servers.
The zone file [b]/var/named/chroot/var/named/test1.com.zone[/b] is also not transferred.

The zone entry in the [b]named.conf[/b] file is not transfered either.

There are no errors in the logs.



Log on the Master server:

# /etc/init.d/named restart ; tail -f /var/log/messages
[quote]
Feb 4 14:53:49 ns1 named[10157]: command channel listening on 127.0.0.1#953
Feb 4 14:53:49 ns1 named[10157]: zone 0.168.192.in-addr.arpa/IN: loaded serial 101
Feb 4 14:53:49 ns1 named[10157]: zone test1.com/IN: loaded serial 101
Feb 4 14:53:49 ns1 named[10157]: zone yourdomain.com/IN: loaded serial 101
Feb 4 14:53:49 ns1 named[10157]: running
Feb 4 14:53:49 ns1 named[10157]: zone 0.168.192.in-addr.arpa/IN: sending notifies (serial 101)
Feb 4 14:53:49 ns1 named[10157]: zone yourdomain.com/IN: sending notifies (serial 101)
Feb 4 14:53:49 ns1 named[10157]: zone test1.com/IN: sending notifies (serial 101)
[/quote]


Neither of the Slave server show anything in their logs with regards to the new [b]test1[/b] entry.


I need this entire process to be automated.
As currently I am testing on only 3 BIND DNS servers, but we will end up having close to 8 servers at all our branches.
And to do such a task manually is really not viable.

There must be an easier way to accomplish this.

Please help


Thanks

SimonBillis
Posts: 5
Joined: 2010/02/03 16:48:52

Re: DNS not replicating from master to slave

Post by SimonBillis » 2010/02/04 14:03:23

As far as I am aware you need to write scripts to automate the configuration transfer between the master and the slaves. It is a relatively simple task to write a script that reads the master config and turns it into a config suitable for the slaves. Once you have this config you can then scp it to the slaves and then reload bind.

THT
Simon.

codenjanod
Posts: 7
Joined: 2010/01/27 10:53:12
Contact:

Re: DNS not replicating from master to slave

Post by codenjanod » 2010/02/05 07:11:59

Hi there,

I am also just glad that it seems my configuration is good.

Thanks for that tip with regards to the scripts.
I must admit though that my scripting/programming skills are not all that good.

Could you maybe point me into a direction where I can find a script that can convert this:
[code]
zone "yourdomain.com" IN {
type master;
file "/var/named/yourdomain.com.zone";
allow-update { trusted-servers; };
allow-transfer { trusted-servers; };
};
[/code]


to this:
[code]
zone "yourdomain.com" IN {
type slave;
file "/var/named/dbs.yourdomain.com";
masters { 10.2.37.202; };
};
[/code]


Many thanks

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: DNS not replicating from master to slave

Post by TrevorH » 2010/02/06 00:40:53

I think you might need to have an

also-notify {x.x.x.x; x.x.x.x; };

line in your options stanza on the master server (with x.x.x.x being the IPs of your slave servers). Also I have 'notify explicit' so that only those servers named in the also-notify get notified.

I do not think that this will handle setting up [b]new[/b] zones but it will propagate changes made in the zones that are defined on both master and slave servers. We use a mixture of nsupdate and dhcp to update our internal zones and those updates do get sent to the slaves.

codenjanod
Posts: 7
Joined: 2010/01/27 10:53:12
Contact:

Re: DNS not replicating from master to slave

Post by codenjanod » 2010/02/11 11:29:12

Hi all,

Thanks for all the replies and help, but I managed to find a script and make it work so that it works 100% for my needs.

This is it, so I hope this might be able to help someone else:
[code]
#! /bin/bash
cd /var/named/chroot/etc/

#store number of lines in conf file
lines=`cat named.conf | wc -l`

# bc calculates how many lines to tail - header
modifylines=`echo "$lines-27" | bc`

#write the header to a file (first 27 lines)
head -27 named.conf > named.conf.head

# just modify the zone file lines, the first 27 lines do not need to be changed
tail -$modifylines named.conf > named.conf.tail

sed -e "s/type master/type slave/g" named.conf.tail > named.conf.slave
sed -e "s/file \"pri\/rev\//file \"sec\/rev\//g" named.conf.slave > named.conf.rev
sed -e "s/file \"pri\/fwd\//file \"sec\/fwd\//g" named.conf.rev > named.conf.fwd

awk '{print $0} /\";/ {found=1} /\";/ && found {print "\tmasters { SLAVE1_IP; };"}' named.conf.fwd > named.conf.tailnew
awk '{print $0} /\";/ {found=1} /\";/ && found {print "\tmasters { SLAVE2_IP; };"}' named.conf.fwd > named.conf.tailnew2

cat named.conf.head named.conf.tailnew > /var/named/chroot/etc/named_slave/named.conf
cat named.conf.head named.conf.tailnew2 > /var/named/chroot/etc/named_slave/named.conf2

rm named.conf.tail named.conf.head named.conf.fwd named.conf.slave named.conf.tailnew named.conf.rev

# rsync it over to secondary
rsync -e ssh -avzlP /var/named/chroot/etc/named_slave/named.conf root@ns2.server:/var/named/chroot/etc/.
ssh ns2.server "service named reload"

rsync -e ssh -avzlP /var/named/chroot/etc/named_slave/named.conf2 root@ns3.server:/var/named/chroot/etc/named.conf
ssh ns3.server "service named reload"

service named reload
[/code]


All changes are made to all the [b]"master"[/b] entries and replaced by [b]"slave"[/b]
And as you can see the [b]named.conf[/b] file, is then rsync'd over to the initial 2 slave servers.
Also makes things much easier to add more slave servers.


Hope this helps someone.

amonamarth
Posts: 40
Joined: 2010/01/06 07:06:29
Location: Los Angeles

Re: DNS not replicating from master to slave

Post by amonamarth » 2010/03/04 07:48:43

WOW! That's a complex script :-D ... I think there is an easier way and please correct me if I'm wrong:

According to the theory of DNS, authoritative DNS servers, that is master DNS servers, have TTL(time-to-live) and serial number values specified in the SOA record for each zone. When the TTL for the zone expires, the slave server(s), provided that they have been specified on the master's /etc/named.conf as slaves and allowed to transfer, will contact the master server to check the serial number for the zone. If the zone has changed(new serial number), the slave will initiate a zone transfer on its own without the need to run any scripts. I believe that is in the specifications of the DNS protocol itself, therefore DNS servers have been design to do so on its own.

On your /etc/named.conf and zone files I see all necessary options for a successful AFXR transfer.

Here is the SOA record for my zone:

$TTL 86400 ; 1 day
domain.com IN SOA ns1.domain.com. admin.domain.com. (
2010013001 ; serial, I use date and a 2 digit number
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)

On the /etc/named.conf on the master server you have all the relevant options I have:
notify yes;
allow transfer;
etc.

On the /etc/named.conf on the slave servers you specify the master server for the zone.

Anyways ... it works for me without running scripts. I'm happy to post full conf files, minus private info. Let me know

:-(

codenjanod
Posts: 7
Joined: 2010/01/27 10:53:12
Contact:

Re: DNS not replicating from master to slave

Post by codenjanod » 2010/03/04 08:21:25

Thanks for that ... took a bit to tweak the script to change the master entries in the "named.conf" file to slave for the slave servers, but so far it is working fine.

It will be great if all this can happen on the fly and we never have to really touch any of the slave server's configuration(named.conf) files.

So if you telling me that your named.conf file replicates to the slave server 100%, as well as any changed or new zone files, that is great.

Then I would like to have a look at your config files if you do not mind, minus your private info of course.

The only reason for the script is that I now have 3 slave servers, soon to be 4, and I do not want to edit all 3 slave server's "named.conf" files with any changed or new entries, so the script makes it easier.

And after the notifies the zone files are copied without any hassles.


Thanks again

Post Reply