Prerequisites:

o Familiarity with DNS, /etc/hosts and /etc/nsswitch.conf http://en.wikipedia.org/wiki/Name_Service_Switch o Basic understanding of iptables http://www.netfilter.org/ http://iptables-tutorial.frozentux.net/

Lazy QA/Staging

In the staging of production systems it is often desirable to override DNS to point at a local staging server instead of the production one. For example, you may have a large number of scripts that call "www.xyz.com"; changing all of these to "wwwtest.xyz.com" would be a pain in the ass.

Sure, there are good ways around this if you have a decent QA environment. You can use a special QA network with its own DNS server. To some extent you can use rule-based proxies.

A large number of solutions involve editing your local hosts file (/etc/hosts under UNIX, \system32\drivers\etc\hosts on Windows) to override DNS.

This works due to a file named /etc/nsswitch.conf. This very useful file contains the rules that programs use to resolve host names and other directory services (sendmail aliases, autofs, etc).

Basically, when a program wants to connect to "www.xyz.com" it makes a call to gethostbyname(), which consults /etc/nsswitch.conf.

A common /etc/nsswitch.conf might contain the following line:

Listing 1:

hosts: files dns

This tells gethostbyname() to consult local files first (ie., /etc/hosts). If it can't find what it's looking for there it will consult DNS.

NOTE: Not all programs respect /etc/resolv.conf. For example, the "nslookup", "dig" and "host" commands will return DNS information regardless of what is in /etc/hosts.

Usually this works fine. You'll resolve www.xyz.com normally with the entry commented out in /etc/hosts and resolve it to the staging IP with the entry present.

The problem comes in when you want certain users to use DNS, and certain users to use the overrides in /etc/hosts.

Per-user DNS overrides

The important thing to remember about configuration files in /etc (/etc/hosts, /etc/nsswitch.conf, etc.) is that they are considered system-level confs. That is, they apply to all users. gethostbyname() doesn't care which user called it--it just blindly returns results based on the system-level rules in /etc.

The trick to making this work is understanding how /etc/nsswitch.conf works. Let's take a closer look at our above example, which specifies the search order as "files dns". This means it consults /etc/hosts first, and only if it fails or doesn't find an entry will it try DNS.

The key is, "if it fails."

It's highly unlikely /etc/hosts will ever fail--it's just a flat file. It is possible for DNS to fail, however. In fact, since this requires network access, we can make it fail.

Our goal is for some users to resolve www.xyz.com publicly, while forcing other users to hit staging. We can do this by providing public information via DNS and the staging IPs via /etc/hosts.

Change /etc/nsswitch.conf to consult DNS first, only falling back on /etc/hosts if that fails:

Listing 2:

hosts: dns files

Now any entry we put into /etc/hosts for "www.xyz.com" will be ignored unless the DNS server fails to respond. The next step is to force DNS to fail for certain users.

Let's say user jdoe has a uid of 503. We want Joe to hit staging. using the "-m owner" feature of iptables we can block him from hitting udp/tcp port 53 (DNS).

Listing 3:

/sbin/iptables -A OUTPUT -p tcp --dport 53 -m owner --uid-owner 503 -j REJECT /sbin/iptables -A OUTPUT -p udp --dport 53 -m owner --uid-owner 503 -j REJECT

User jdoe will now be unable to make DNS requests since udp/tcp port 53 outgoing is blocked. gethostbyname() will fail and thus fall back to /etc/hosts due to the options in /etc/nsswitch.conf. We can similarly add iptables rules for any user we want to hit staging.

You may need to restart your nscd service depending on how it's configured. nscd is a caching daemon that runs by default on some systems (Solaris, SuSE, etc.). More information can be found below:

http://linuxmanpages.com/man8/nscd.8.php

There is one problem with this method--all hosts must be in /etc/hosts. If jdoe gets bored of QAing and wants to visit slashdot.org, he can't do so unless this is in /etc/hosts or he removes the iptables rule.

Packet filtering with iptables is very powerful and robust. To learn more about it visit the two links provided above in the prerequisites section, as it is a must-have utility for any Linux professional.

Prerequisites:

1) Ability to build a custom kernel (Refer to your specific distro's documentation) 2) SCSI device addressing http://www.tldp.org/HOWTO/SCSI-2.4-HOWTO/scsiaddr.html 3) The sg3_utils package http://sg.torque.net/sg/sg3_utils.html

System administrators often wonder why Linux can't use a sane naming convention for its disks like Solaris does--/dev/dsk/c0t1d0s2 is simple to understand--scsi host 0, bus 1, device 0, lun 2. Simple. Elegant.

On Linux 2.4 we get /dev/sdd1 (Under 2.6 we get sysfs and udev, neither of which is much better in my opinion).

Thankfully the sg3_utils package contains the "sg_map" command, which is insanely helpful. Let's take a basic system that has five disks:

Listing 1:

[root@host1 root]# sg_map -x /dev/sg0 0 0 0 0 8 /dev/sg1 0 0 1 0 1 /dev/st0 /dev/sg2 1 0 0 0 0 /dev/sda /dev/sg3 1 0 1 0 0 /dev/sdb /dev/sg4 1 0 2 0 0 /dev/sdc /dev/sg5 1 0 3 0 0 /dev/sdd /dev/sg6 1 0 4 0 0 /dev/sde /dev/sg7 1 0 6 0 3

The -x flag is required to print out the numbers in the middle. We only care about the first four numbers, which are, in order, "host, bus, device, lun" (the fifth digit is scsi type; read the sg_map man page for more info).

On this particular system we have a tape drive on scsi host 0; our five disks are on scsi host 1. They all share the same bus (0) and each disk has its own device id. All disks use only LUN 0, even though each disk is separated into different partitions (/dev/sda1, /dev/sda2, etc). Under Linux different partitions on the same disk do not become different LUNs.

When you're only dealing with individual disks this is annoying but straightforward. Problems can arise, however, If you have a SAN or direct-attached storage. Many of these will arrange their disk volumes by LUN.

Problem is, by default the 2.4 Linux kernel will *only* scan LUN 0.

To change this you must modify your kernel settings. Install your kernel sources as per your distro's documentation (compiling your own kernel is beyond the scope of this article). When you're at the configuration step (whichever you use: make menuconfig, make xconfig, etc) go down to the SCSI section and put a check in the line for "enable scanning of multiple LUNs" Save your .config and finish compiling the kernel as usual.

In your grub or lilo conf file, add "max_scsi_luns=128" to the end of your kernel line:

Listing 2:

title Red Hat Enterprise Linux WS (2.4.21-20.ELsmp-san) root (hd0,0) kernel /vmlinuz-2.4.21-20.ELsmp-san ro root=LABEL=/ console=tty0 console=ttyS0,9600n8 max_scsi_luns=128

Reboot onto your new kernel and you should now be able to see all your LUNs.

Listing 3:

[root@host2 root]# sg_map -x /dev/sg0 0 0 1 0 0 /dev/sda /dev/sg1 2 0 0 0 0 /dev/sdb /dev/sg2 2 0 0 1 0 /dev/sdc /dev/sg3 2 0 0 2 0 /dev/sdd /dev/sg4 2 0 0 3 0 /dev/sde /dev/sg5 2 0 0 4 0 /dev/sdf /dev/sg6 2 0 0 5 0 /dev/sdg

This particular server has a single internal disk (/dev/sda on scsi host 0) and serveral SAN volumes (/dev/sdb-/dev/sdg on scsi host 0). Note that all these SAN volumes have the same bus and device identifiers; they are separated only by LUN.

This should be the end of this article, however due to a quirk in how Linux 2.4 handles LUN scanning one last important note must be addressed:

Linux 2.4 scans LUNs in order and will stop scanning if it finds a non-existent LUN.

This is particularly troublesome if you have a SAN and only want certain hosts to see certain volumes. Let's say you have six LUNs numbered 0-5. If you want one host to only see LUNs 0, 1 and 2 this is easy enough--disableing luns 3, 4 and 5 from either your HBA or the SAN itself will cause Linux no problems.

However, let's say you want a host to only see LUN 4. This will cause a problem--Linux will scan LUN 0, find nothing, and move on. It won't even attempt to scan LUNs 1-5. If you want a host to see LUNs 0, 1, 2, 4 and 5 it will only see the first three (if Linux doesn't see LUN 3 it won't bother trying LUNs 4 or 5).

This can lead to messy SAN setups and poses the risk of one machine accidentally mounting a disk it's not supposed to. I'm not aware of a clean solution to this at this time with Linux 2.4 other than separating SAN volumes at a device level instead of using LUNs. If anyone knows a good solution I'd be quite keen on learning about it.