Prerequisites:
System administrators often wonder why Linux can't use a sane naming convention for its disks like Solaris does--/dev/dsk/c0t1d0s2 is simple to understand--scsi host 0, bus 1, device 0, lun 2. Simple. Elegant.
On Linux 2.4 we get /dev/sdd1 (Under 2.6 we get sysfs and udev, neither of which is much better in my opinion).
Thankfully the sg3_utils package contains the "sg_map" command, which is insanely helpful. Let's take a basic system that has five disks:
Listing 1:
[root@host1 root]# sg_map -x
/dev/sg0 0 0 0 0 8
/dev/sg1 0 0 1 0 1 /dev/st0
/dev/sg2 1 0 0 0 0 /dev/sda
/dev/sg3 1 0 1 0 0 /dev/sdb
/dev/sg4 1 0 2 0 0 /dev/sdc
/dev/sg5 1 0 3 0 0 /dev/sdd
/dev/sg6 1 0 4 0 0 /dev/sde
/dev/sg7 1 0 6 0 3 |
The -x flag is required to print out the numbers in the middle. We only care about the first four numbers, which are, in order, "host, bus, device, lun" (the fifth digit is scsi type; read the sg_map man page for more info).
On this particular system we have a tape drive on scsi host 0; our five disks are on scsi host 1. They all share the same bus (0) and each disk has its own device id. All disks use only LUN 0, even though each disk is separated into different partitions (/dev/sda1, /dev/sda2, etc). Under Linux different partitions on the same disk do not become different LUNs.
When you're only dealing with individual disks this is annoying but straightforward. Problems can arise, however, If you have a SAN or direct-attached storage. Many of these will arrange their disk volumes by LUN.
Problem is, by default the 2.4 Linux kernel will *only* scan LUN 0.
To change this you must modify your kernel settings. Install your kernel sources as per your distro's documentation (compiling your own kernel is beyond the scope of this article). When you're at the configuration step (whichever you use: make menuconfig, make xconfig, etc) go down to the SCSI section and put a check in the line for "enable scanning of multiple LUNs" Save your .config and finish compiling the kernel as usual.
In your grub or lilo conf file, add "max_scsi_luns=128" to the end of your kernel line:
Listing 2:
title Red Hat Enterprise Linux WS (2.4.21-20.ELsmp-san)
root (hd0,0)
kernel /vmlinuz-2.4.21-20.ELsmp-san ro root=LABEL=/ console=tty0
console=ttyS0,9600n8 max_scsi_luns=128 |
Reboot onto your new kernel and you should now be able to see all your LUNs.
Listing 3:
[root@host2 root]# sg_map -x
/dev/sg0 0 0 1 0 0 /dev/sda
/dev/sg1 2 0 0 0 0 /dev/sdb
/dev/sg2 2 0 0 1 0 /dev/sdc
/dev/sg3 2 0 0 2 0 /dev/sdd
/dev/sg4 2 0 0 3 0 /dev/sde
/dev/sg5 2 0 0 4 0 /dev/sdf
/dev/sg6 2 0 0 5 0 /dev/sdg |
This particular server has a single internal disk (/dev/sda on scsi host 0) and serveral SAN volumes (/dev/sdb-/dev/sdg on scsi host 0). Note that all these SAN volumes have the same bus and device identifiers; they are separated only by LUN.
This
should be the end of this article, however due to a quirk in how Linux 2.4 handles LUN scanning one last important note must be addressed:
Linux 2.4 scans LUNs in order and will stop scanning if it finds a non-existent LUN.
This is particularly troublesome if you have a SAN and only want certain hosts to see certain volumes. Let's say you have six LUNs numbered 0-5. If you want one host to only see LUNs 0, 1 and 2 this is easy enough--disableing luns 3, 4 and 5 from either your HBA or the SAN itself will cause Linux no problems.
However, let's say you want a host to only see LUN 4. This
will cause a problem--Linux will scan LUN 0, find nothing, and move on. It won't even attempt to scan LUNs 1-5. If you want a host to see LUNs 0, 1, 2, 4 and 5 it will only see the first three (if Linux doesn't see LUN 3 it won't bother trying LUNs 4 or 5).
This can lead to messy SAN setups and poses the risk of one machine accidentally mounting a disk it's not supposed to. I'm not aware of a clean solution to this at this time with Linux 2.4 other than separating SAN volumes at a device level instead of using LUNs. If anyone knows a good solution I'd be quite keen on learning about it.
wth...
eh, spam happens =P