When our InternetServer was a physical machine, liberty.gnhlug.org, this documented the mass storage configuration.

It had twin 73 GB SCSI hard drives. More at ServerHardware and ServerLiberty.

Mirroring

Consensus was that disk mirroring (RAID level 1) was appropriate. There is no service contract or tech support, physical access is limited, and we have limited manpower and no budget. A disk failure could knock us offline for weeks. At the same time, our current disk usage is small.

The server did not come with a hardware RAID controller, and we have no money to buy one. So, that means software RAID. It works pretty well on Linux. Many installers can set it up for you, including the ones that come with Red Hat derived OSes.

Software RAID also has some advantages over hardware controllers:

• The disks are portable without having to worry about the controller
• It's very well tested
• There are no proprietary innards we can't see or fix ourselves
• We have CPU power to spare for RAID overhead

Booting from RAID 1 works well with GRUB. The boot partition has to be a simple mirror (no LVM, RAID 5, etc.). It thus appears to GRUB like an ordinary partition. Since GRUB never writes to the filesystem, it doesn't matter that the partition is actually a member of a mirror set. By installing GRUB on to both disks this way, either disk becomes bootable. This is best done by booting GRUB from floppy or CD, as differences between how the BIOS (and thus GRUB) and the Linux kernel see disks can matter.

On the the kernel boot command line, specify the RAID/LVM device (/dev/md0 or whatever) for the root filesystem to mount. Do not specify physical devices here. You can use whatever kind of device you like. The kernel uses an initial ramdisk to do things like activate RAID devices and LVM volumes; it's only the boot partition that has to be a simple mirror.

Example GRUB stanza:

title CentOS (2.6.18-92.1.17.el5)
   root (hd0,0)
   kernel /vmlinuz-2.6.18-92.1.17.el5 ro root=LABEL=ROOT vga=6
   initrd /initrd-2.6.18-92.1.17.el5.img

LVM

We're using LVM to create containers for each filesystem. That gives us the flexibility to re-organize our filesystems and such later on. Given that our plans for the future are always very much up in the air, that is a real benefit. Say we make /var/log too small and need to make it bigger, or /home too big or small, or whatever. LVM makes it trivial to change, with minimal downtime. Without LVM, you have to take the filesystems offline and possibly spend forever doing a risky shift-in-place operation.

One drawback is that this is an extra layer of complexity. On the plus side, it's pretty well automated in most distros.

Layout

Each disk is partitioned into a small boot partition and a large LVM PV (Physical Volume) partition. Linux RAID (md device) is used to mirror each pair of partitions. Thus, the partition type for each is Linux RAID auto-detect.

Disk /dev/sda: 8924 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+     64      65-    522081   fd  Linux raid autodetect
/dev/sda2         65    8923    8859   71159917+  fd  Linux raid autodetect
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty

LVM runs on top of the mirrored PV partition; LVM sees a single PV (/dev/md1). A single volume group liberty_vg exists. It contains several different LVs (Logical Volumes), each with a different purpose in the filesystem.

  LV          Size
  home_lv     4.00 G
  local_lv    4.00 G
  log_lv      1.00 G
  mailman_lv  2.00 G
  recovery_lv 4.00 G
  root_lv     1.00 G
  sites_lv    4.00 G
  spool_lv    1.00 G
  swap_lv     1.00 G
  tmp_lv      1.00 G
  usr_lv      4.00 G
  var_lv      1.00 G

  * 39.84 GB unallocated

Not everyone on the server team was in agreement, so it mainly came down to "the guy doing most of the work at the time picked what he liked best". The guy at the time was Ben Scott. His reasons for his approach:

• A place for everything and everything in its place.
• It keeps runaway output from killing the system (/var/log and /var/spool especially).
• It makes any kind of OS reinstall/etc much easier.
• It localizes filesystem fragmentation.
• It localizes any filesystem corruption.

Drawbacks to this approach include:

• More complicated
• Resizing filesystems generally involves at least a partial service interruption
• Resizing a root filesystem is rather difficult
  • However, by using separate LVs, the root filesystem should "never" need to be resized