This is a disclaimer: 
Using the notes below is dangerous for both your sanity and peace of mind.  
If you still want to read them beware of the fact that they may be "not even wrong".

Everything I write in there is just a mnemonic device to give me a chance to
fix things I badly broke because I'm bloody stupid and think I can tinker with stuff
that is way above my head and go away with it. It reminds me of Gandalf's warning: 
"Perilous to all of us are the devices of an art deeper than we ourselves possess."

Moreover, a lot of it I blatantly stole on the net from other obviously cleverer 
persons than me -- not very hard. Forgive me. My bad.

Please consider it and go away. You have been warned!

(:toc:)

DNS/Bind Setup, Config and Management at the BIC

The authority on the BIC domain zone bic.mni.mni.mcgill.ca has been delegated to us by McGill NCS a while ago, I think back in ‘98 IIRC.

  • The master DNS server is shadow.bic.mni.mcgill.ca [132.206.178.7]
  • One slave is configured as a caching server, agrippa.bic.mni.mcgill.ca [132.206.178.186]
  • Bind9 is used.
  • 20160401. Note that McGill DNS has been completely modified.
  • moka/kona/oolong are now gone and have been replaced.
  • kona/moka/oolong were acting as both resolvers and SOA.
  • In this configuration kona/moka/oolong were secondaries and maps were local on our master and xferred to kona/moka/oolong.
  • For this to work, we have to allow queries not only to out ‘trusted’ ACLS, but to the world at large.
  • 2 new authoritative servers have been configure pens1/pens2.bic.mni.mcgill.ca.
  • The new configuration has pirns1/pirns2/pirns3 as internal resolvers.
  • All things that were secondary, were given delegation records

BIND9 Config

  • Things are little bit non-standard and iffy.
  • Back in the days I had to compile BIND9 on an Irix machine (it’s name was shadow too!) as the SGI distro didn’t contain a recent enough version of BIND.
  • Lacking experience I scattered stuff around needlessly.
  • I’m slowly consolidating files content, statements and their location to conform to Debian files layout.

Master (shadow)

Bind9 is installed along with other related utilities and librairies:

ii  bind9                                 1:9.8.4.dfsg.P1-6+nmu2+deb7u2~bpo60+1 Internet Domain Name Server
ii  bind9-host                            1:9.7.3.dfsg-1~squeeze11              Version of 'host' bundled with BIND 9.X
ii  bind9utils                            1:9.8.4.dfsg.P1-6+nmu2+deb7u2~bpo60+1 Utilities for BIND
ii  libbind9-60                           1:9.7.3.dfsg-1~squeeze11              BIND9 Shared Library used by BIND
ii  libbind9-80                           1:9.8.4.dfsg.P1-6+nmu2+deb7u2~bpo60+1 BIND9 Shared Library used by BIND
  • The named configuration file named.conf is located in /etc/bind, so that all static configuration files relating to bind are in one place.
  • Zone data files for the root servers, and the forward and reverse localhost zones are also provided in /etc/bind.
  • The working directory for named is /var/cache/bind.
  • The main config file, /etc/bind/named.conf:
(:source:)
// This is the primary configuration file for the BIND DNS server named.
//
// Please read /usr/share/doc/bind/README.Debian for information on the 
// structure of BIND configuration files in Debian for BIND versions 8.2.1 
// and later, *BEFORE* you customize this configuration file.
//

include "/etc/bind/named.conf.options";

// prime the server with knowledge of the root servers
zone "." {
        type hint;
        file "/etc/bind/db.root";
};

// be authoritative for the localhost forward and reverse zones, and for
// broadcast zones as per RFC 1912

zone "localhost" {
        type master;
        file "/etc/bind/db.local";
};

zone "127.in-addr.arpa" {
        type master;
        file "/etc/bind/db.127";
};

zone "0.in-addr.arpa" {
        type master;
        file "/etc/bind/db.0";
};

zone "255.in-addr.arpa" {
        type master;
        file "/etc/bind/db.255";
};

// add local zone definitions here
include "/etc/bind/named.conf.local";
(:sourceend:)
  • /etc/bind/named.conf refers to a few zone files that reside in /etc/bind that come bundled with the bind9 Debian package.
  • /etc/bind/named.conf includes 2 files, /etc/bind/named.conf.options and /etc/bind/named.conf.local.
  • The option file /etc/bind/named.conf.options defines a few ACLs for security purposes, the working directory, log files and level, statistics file, control port, etc,
  • The local file /etc/bind/named.conf.local contains the stuff that’s is dependent on our local environment.
  • etc/bind/named.conf.options
(:source:)
// This ACL defined trusted networks for which we allow recursives queries.
acl "trusted" {
    // BIC public IP domains and private networks
    132.206.178.0/24; 
    132.206.201.0/24;
    172.16.10.0/24; 
    172.16.50.0/24;
    172.16.100.0/24; 
    192.168.86.0/24; 
    192.168.1.0/24; 
    // McGill authoritative nameservers
    132.216.44.69;
    132.206.25.70;

    localhost;
    localnets;
};

options {
    directory "/var/cache/bind";

    // Only allow queries from the ACL "trusted" defined above.
    // allow-query { trusted; }; // trusted could be replaced by any only if necessary
    // JF. 20160614. McGill has changed their NS and what were secondaries 
    // JF. 20160614. before (moka/kona/oolong) has been changed to internal resolvers  
    // JF. 20160614. (pirns1/pirns2/pirns3) and have been given delegations records
    // JF. 20160614. and so we must now allow for queries.
    allow-query { any; }; 
    allow-transfer { trusted; };
    allow-recursion { trusted;} ;
    allow-query-cache { trusted; }; // for BIND 9.4+
    additional-from-cache no;

    // from bind 9:
    //  [fetch-glue] is obsolete. In BIND 8, fetch-glue yes caused the
    //  server to attempt to fetch glue resource records it didn't have
    //  when constructing the additional data section of a response.
    //  This is now considered a bad idea and BIND 9 never does it.

    // fetch-glue no;

    // If there is a firewall between you and nameservers you want
    // to talk to, you might need to uncomment the query-source
    // directive below.  Previous versions of BIND always asked
    // questions using port 53, but BIND 8.1 and later use an unprivileged
    // port by default.

    // query-source address * port 53;

    // If your ISP provided one or more IP addresses for stable
    // nameservers, you probably want to use them as forwarders.
    // Uncomment the following block, and insert the addresses replacing
    // the all-0's placeholder.

    // forwarders {
    //  0.0.0.0;
    //
    //};

    //========================================================================
    // If BIND logs error messages about the root key being expired,
    // you will need to update your keys.  See https://www.isc.org/bind-keys
    //========================================================================
    dnssec-enable yes;
    dnssec-validation auto;
    dnssec-lookaside auto;
    //bindkeys-file "/etc/bind/bind.keys";

    statistics-file "named.stats";
    zone-statistics yes;
};

logging {
    channel my_syslog {
    syslog daemon;
    severity info;
    };

    channel logfile {
        file "/var/log/named.log";
        severity info;
        print-category yes;
        print-severity yes;
        print-time yes;
    };

    channel "querylog" {
        file "query.log" versions 5 size 100m;
        print-time yes;
    };

    category default    { logfile; };
    category config     { logfile; };
    //category statistics { logfile; };
    category queries    { querylog; };
    //reduce log verbosity on issues outside our control
    category lame-servers { null; };
    //category cname { null; };
};

key "rndc-key" {
      algorithm hmac-md5;
      secret "ur023ZC/6vrCVHe/fNRrJA==";
};

controls {
      inet 127.0.0.1 port 953
              allow { 127.0.0.1; } keys { "rndc-key"; };
};
(:sourceend:)

Things to notice for the options file:

  • allow-recursion { trusted;}; only allows recursive queries from the trusted hosts/networks ACL’s. This disables amplification DDoS attacks.

You can verify if recursive queries are enabled or not using the following dig or nmap tricks:

    dig ANY iweb.com @<IP>
    (flags: qr rd ra; means open recursive, 'ra' == recursion available)

    ~$ dig ANY iweb.com @132.206.178.7
    ; <<>> DiG 9.8.1-P1 <<>> ANY iweb.com @132.206.178.7
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 30243
    ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
    ;; WARNING: recursion requested but not available

    ;; QUESTION SECTION:
    ;iweb.com.			IN	ANY

    ;; Query time: 19 msec
    ;; SERVER: 132.206.178.7#53(132.206.178.7)
    ;; WHEN: Wed Jan 27 11:06:15 2016
    ;; MSG SIZE  rcvd: 26

or use nmap (this example is done from one of the trusted acl's so recursive queries are allowed)

    nmap -PN --script=dns-recursion -sU -p53 132.206.178.7

    Starting Nmap 6.00 ( http://nmap.org ) at 2016-01-27 10:54 EST
    Nmap scan report for shadow.bic.mni.mcgill.ca (132.206.178.7)
    Host is up (0.00092s latency).
    PORT   STATE SERVICE
    53/udp open  domain
    |_dns-recursion: Recursion appears to be enabled
    MAC Address: 00:16:3E:C3:E7:F5 (Xensource)

    Nmap done: 1 IP address (1 host up) scanned in 5.75 seconds
  • The statements dnssec-enable yes; and dnssec-validation auto; enable DNSSEC on BIND9.
  • dnssec-lookaside auto; line makes named reads the DLV (DNSSEC Look-aside Validation) key from bind.keys the first time it executes. This is the dlv.isc.org key.
  • The bindkeys-file line is needed only if the bind.keys file is in a location other than /etc/bind/bind.keys - if it’s /etc/bind/bind.keys, it’s loaded by default.
  • statistics-file “named.stats”; in the options section. This is where the stats accumulate when rndc stats is ran.
  • This is to gather statistics for Munin.
  • This file will grow continuously as rndc will just append to it.
  • A logrotate file is shown below that will rotate it in an orderly fashion.
  • channel logfile { file “/var/log/named.log”;…};. This is where bind logs everything.
  • A logrotate script rotate it on a regular basis as it can grow a large.
  • channel “querylog” { file “query.log” versions 5 size 100m; print-time yes; }; in the logging sections. This is for Munin to plot the queries that bind receives.
  • The file query.log lives in the working directory /var/cache/bind and is rotated away when it grows bigger that 100MB. 5 versions will be kept.
  • etc/bind/named.conf.local
(:source:)
//
// Do any local configuration here
//

// Consider adding the 1918 zones here, if they are not used in your
// organization
include "/etc/bind/zones.rfc1918";

zone "bic.mni.mcgill.ca" {
    type master;
    file "/etc/bind/db.bic";
    allow-transfer { 
                    132.206.178.7; 
                    132.206.178.186;
                    132.206.44.21; 
                    132.206.44.22; 
                    132.216.44.21; 
                    132.216.44.22; 
                    132.206.25.21;
                    132.206.25.22;
                    132.204.8.207;
    };
};

zone "brain-child.org" {
    type master;
    file "/etc/bind/db.brain-child.org";
    allow-transfer {
                    132.206.178.7;
                    132.206.178.186;
                    132.206.44.21; 
                    132.206.44.22; 
                    132.216.44.21; 
                    132.216.44.22; 
                    132.206.25.21;
                    132.206.25.22;
                    132.204.8.207;
    };

};

zone "canada-meg-consortium.org" {
    type master;
    file "/etc/bind/db.canada-meg-consortium.org";
    allow-transfer {
                    132.206.178.7;
                    132.206.178.186;
                    132.206.44.21; 
                    132.206.44.22; 
                    132.216.44.21; 
                    132.216.44.22; 
                    132.206.25.21;
                    132.206.25.22;
                    132.204.8.207;
    };

};

zone "178.206.132.IN-ADDR.ARPA" {
    type master;
    file "/etc/bind/db.132.206.178";
    allow-transfer {
                    132.206.178.7; 
                    132.206.178.186;
                    132.206.44.21; 
                    132.206.44.22; 
                    132.216.44.21; 
                    132.216.44.22; 
                    132.206.25.21;
                    132.206.25.22;
                    132.204.8.207;
    };
};

zone "201.206.132.IN-ADDR.ARPA" {
    type master;
    file "/etc/bind/db.132.206.201";
    allow-transfer {
                    132.206.178.7; 
                    132.206.178.186;
                    132.206.44.21; 
                    132.206.44.22; 
                    132.216.44.21; 
                    132.216.44.22; 
                    132.206.25.21;
                    132.206.25.22;
                    132.204.8.207;
    };
};
(:sourceend:)

The local definition file contains statements about who is allowed the transfer the master zone files content. Those include the master server itself, the secondary caching server and the DNS servers inside the McGill domain.

Logs rotation

The /var/log/named.log logfile is rotated away on a weekly basis using the logrotate daemon facility. Same goes for the munin stat file /var/cache/bind/named.stats.

/etc/logrotate.d/bind

(:source:)
/var/log/named.log {
  weekly
  missingok
  rotate 52
  compress
  delaycompress
  notifempty
  create 640 bind bind
  postrotate
    if [ -f /var/run/bind/named.pid ]; then
      /etc/init.d/bind9 restart > /dev/null
    fi
  endscript
}
/var/cache/bind/named.stats {
  weekly
  missingok
  rotate 4
  compress
  delaycompress
  notifempty
  create 640 bind bind
}
(:sourceend:)

Secondary (Caching) DNS Server

  • BIND9 software installation base:
ii  bind9                                 1:9.8.4.dfsg.P1-6+nmu2+deb7u9      amd64        Internet Domain Name Server
ii  bind9-host                            1:9.8.4.dfsg.P1-6+nmu2+deb7u9      amd64        Version of 'host' bundled with BIND 9.X
ii  bind9utils                            1:9.8.4.dfsg.P1-6+nmu2+deb7u9      amd64        Utilities for BIND
ii  libbind9-80                           1:9.8.4.dfsg.P1-6+nmu2+deb7u9      amd64        BIND9 Shared Library used by BIND
  • Extra stuff needed for DNSSEC: apt-get install dnssec-tools libnet-dns-sec-perl libmailtools-perl libcrypt-openssl-random-perl:
ii  dnssec-tools                        1.7-3                               DNSSEC tools, applications and wrappers
ii  libcrypt-openssl-random-perl        0.04-1+b1                           Access to the OpenSSL pseudo-random number generator
ii  libmailtools-perl                   2.06-1                              Manipulate email in perl programs
ii  libnet-dns-sec-perl                 0.16-1                              DNSSEC extension to NET::DNS
  • /etc/bind/named.conf
(:source:)     
// This is the primary configuration file for the BIND DNS server named.
//
// Please read /usr/share/doc/bind9/README.Debian.gz for information on the 
// structure of BIND configuration files in Debian, *BEFORE* you customize 
// this configuration file.
//
// If you are just adding zones, please do that in /etc/bind/named.conf.local

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";
(:sourceend:)
  • /etc/bind/named.conf
(:source:) 
options {
    directory "/var/cache/bind";

    // If there is a firewall between you and nameservers you want
    // to talk to, you may need to fix the firewall to allow multiple
    // ports to talk.  See http://www.kb.cert.org/vuls/id/800113

    // If your ISP provided one or more IP addresses for stable 
    // nameservers, you probably want to use them as forwarders.  
    // Uncomment the following block, and insert the addresses replacing 
    // the all-0's placeholder.

    // forwarders {
    //  0.0.0.0;
    // };

    //========================================================================
    // If BIND logs error messages about the root key being expired,
    // you will need to update your keys.  See https://www.isc.org/bind-keys
    //========================================================================
    dnssec-enable yes;
    dnssec-validation auto;
    dnssec-lookaside auto;
    //bindkeys-file "/etc/bind/bind.keys";

    auth-nxdomain no;    # conform to RFC1035
    listen-on-v6 { any; };
    statistics-file "named.stats";
    zone-statistics yes;

};

logging {
//    channel "querylog" { file "query.log"; };
    channel "querylog" { file "query.log" versions 5 size 100m; print-time yes; };
    category queries    { querylog; };
    category lame-servers { null; };
};
(:sourceend:)
  • The statements dnssec-enable yes; and dnssec-validation auto; enable DNSSEC on BIND9.
  • dnssec-lookaside auto; line makes named reads the DLV (DNSSEC Look-aside Validation) key from bind.keys the first time it executes. This is the dlv.isc.org key.
  • The bindkeys-file line is needed only if the bind.keys file is in a location other than /etc/bind/bind.keys - if it’s /etc/bind/bind.keys, it’s loaded by default.
  • /etc/bind/named.conf.default-zones
(:source:)
// prime the server with knowledge of the root servers
zone "." {
	type hint;
	file "/etc/bind/db.root";
};

// be authoritative for the localhost forward and reverse zones, and for
// broadcast zones as per RFC 1912

zone "localhost" {
	type master;
	file "/etc/bind/db.local";
};

zone "127.in-addr.arpa" {
	type master;
	file "/etc/bind/db.127";
};

zone "0.in-addr.arpa" {
	type master;
	file "/etc/bind/db.0";
};

zone "255.in-addr.arpa" {
	type master;
	file "/etc/bind/db.255";
};
(:sourceend:)

Parenting And Sub-Domain Delegation to the Isilon Cluster

  • We now have an Isilon Cluster and it requires to be authoritative for its sub-domain.
  • This is achieved by stitching the following bits to the end of the BIC zone file:
  • /etc/bind/db.bic
(:source:)
; Delegatation for Isilon Cluster

;$ORIGIN isi.bic.mni.mcgill.ca.
; Define two name servers for sub-domain isi.bic.mni.mcgill.ca.
;@       IN      NS      sip.bic.mni.mcgill.ca.
; next nameserver points to bic network dns master
;@       IN      NS      shadow.bic.mni.mcgill.ca.

; Records above could have been written without the $ORIGIN as
isi.bic.mni.mcgill.ca. IN NS sip.bic.mni.mcgill.ca.
;isi.bic.mni.mcgill.ca. IN NS shadow.bic.mni.mcgill.ca.

; OR as simply
;      IN NS   sip
;      IN NS   shadow.bic.mni.mcgill.ca.

; sub-domain address records for name server only - glue record
;sip     IN      A       132.206.178.232 ; 'glue' record
; or as
;sip.bic.mni.mcgill.ca. IN A 132.206.178.232
;
(:sourceend:)
  • The following IP addresses are used by the external and management networks of the Isilon Cluster.
  • The internal (InfiniBand) network is not relevant to this discussion.
  • It is completely hidden from our public and private networks.
  • sip.isi.bic.mni.mcgill.ca is the SOA for the sub-domain isi.bic.mni.mcgill.ca.
  • 132.206.178.233/234/235/236/237 are the dynamic pool of IPs used to offer services.
  • nfs.isi.bic.mni.mcgill.ca. is the dynamically generated DNS name to provive NFS services.
  • mgmt.isi.bic.mni.mcgill.ca. is the dynamically generated DNS name to the management network.
(:source:)
132.206.178.232  sip.isi.bic.mni.mcgill.ca       sip             #CC - BUNKER - 132.206.178.232-239 reserved for Isilon cluster nodes
132.206.178.233  ambas.isi.bic.mni.mcgill.ca     ambas           #CC - BUNKER - 132.206.178.232-239 reserved for Isilon cluster nodes
132.206.178.234  egeus.isi.bic.mni.mcgill.ca     egeus           #CC - BUNKER - 132.206.178.232-239 reserved for Isilon cluster nodes
132.206.178.235  angus.isi.bic.mni.mcgill.ca     angus           #CC - BUNKER - 132.206.178.232-239 reserved for Isilon cluster nodes
132.206.178.236  caius.isi.bic.mni.mcgill.ca     caius           #CC - BUNKER - 132.206.178.232-239 reserved for Isilon cluster nodes
132.206.178.237  titus.isi.bic.mni.mcgill.ca     titus           #CC - BUNKER - 132.206.178.232-239 reserved for Isilon cluster nodes
132.206.178.238  bates.bic.mni.mcgill.ca         bates           #CC - BUNKER - 132.206.178.232-239 reserved for Isilon cluster nodes
132.206.178.239  node9.bic.mni.mcgill.ca         node9           #CC - BUNKER - 132.206.178.232-239 reserved for Isilon cluster nodes
132.206.178.249  zeus.bic.mni.mcgill.ca          zeus                # Xen virtualized CentOs 7 host on loginx2 / Isilon InsightIQ license manager
172.16.10.4     sw-vdx-1        # Brocade VDX Isilon Cluster
172.16.10.5     sw-vdx-2        # Brocade VDX Isilon Cluster
172.16.10.6     sw-vdx-v        # Brocade VDX (Virtual) Isilon Cluster
172.16.10.20                    # 172.16.10.20-24 reserved for Isilon mgmt zone
172.16.10.21                    # 172.16.10.20-24 reserved for Isilon mgmt zone
172.16.10.22                    # 172.16.10.20-24 reserved for Isilon mgmt zone
172.16.10.23                    # 172.16.10.20-24 reserved for Isilon mgmt zone
172.16.10.24                    # 172.16.10.20-24 reserved for Isilon mgmt zone
(:sourceend:)