• The disks are portable without having to worry about the controller
• It's very well tested
• There are no proprietary innards we can't see or fix ourselves
• We have CPU power to spare for RAID overhead

title CentOS (2.6.18-92.1.17.el5)
   root (hd0,0)
   kernel /vmlinuz-2.6.18-92.1.17.el5 ro root=LABEL=ROOT vga=6
   initrd /initrd-2.6.18-92.1.17.el5.img

Layout

Disk /dev/sda: 8924 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  LV          Size
  home_lv     4.00 G
  local_lv    4.00 G
  log_lv      1.00 G
  mailman_lv  2.00 G
  recovery_lv 4.00 G
  root_lv     1.00 G
  sites_lv    4.00 G
  spool_lv    1.00 G
  swap_lv     1.00 G
  tmp_lv      1.00 G
  usr_lv      4.00 G
  var_lv      1.00 G

• A place for everything and everything in its place.
• It keeps runaway output from killing the system (/var/log and /var/spool especially).
• It makes any kind of OS reinstall/etc much easier.
• It localizes filesystem fragmentation.
• It localizes any filesystem corruption.

Drawbacks to this approach include:

• More complicated
• Resizing filesystems generally involves at least a partial service interruption
• Resizing a root filesystem is rather difficult
  • However, by using separate LVs, the root filesystem should "never" need to be resized

-- BenScott - 16 Feb 2006

By the way, it is not true that IDE driver and RAID support needs to be compiled into the kernl. You can use an initrd (initial RAM disk). The initrd contains the needed modules, and is loaded (along with the kernel image) by GRUB. This is how we are actually doings things. Works fine.

-- BenScott - 15 Oct 2006

If we have all the paritions divided up we'll need to plan on downtime as you're not going to be resizing filesystems on the fly even if you do resize LVM on the fly. I don't like resizing root filesystems remotely without a terminal server. If someone wants to volunteer to do this even if it means going on site with an SLA, fine, but that can't be me. I agreee with Ben's strategy on systems with lots of disk and local or terminal access. -- BillMcGonigle - 17 Feb 2006

I mostly agree - it might be worth considering a separate /boot partition if we're going to have terminal server access to this machine and the / filesystem becomes corrupted, we at least stand a chance of fsck'ing it back to life remotely. We certainly shouldn't try to predict usage of /home vs /usr/local vs /opt vs /var as the specs are too loosely defined. -- BillMcGonigle - 11 Feb 2006

Well, I'm from the "divide things up into a bunch of partitions" camp. This ties in with why I favor LVM -- it means we don't have to predict allocation. We can guess and change things later. (If we're going to go with only one partition, then I would say there is no reason to use LVM, either -- plain RAID should be enough.) As for why I like multiple partitions: Party, it's just my nature -- a place for everything and everything in its place and all that. It keeps runaway output from killing the system (/var/log and /var/spool on separate partitions). It makes an OS reinstall/etc much easier. It localizes fragmentation. All that being said, if everyone else is in the "one big partition" camp, I guess we can do that (awwwww ). -- BenScott - 16 Feb 2006

title Fedora Core (2.6.15-1.1831_FC4)
        root (hd0,0)
        kernel /vmlinuz-2.6.15-1.1831_FC4 ro root=/dev/md1
        initrd /initrd-2.6.15-1.1831_FC4.img

-- BillMcGonigle - 11 Feb 2006

You can't boot from LVM and LVM usually implies you'll be adding disks. Since the server is maxed out on disks, this is just an extra layer of complexity. If we're going to just increase the disk size in one of the two slots it's easy enough to copy the data from the 18GB disk to whatever we wind up with, then re-sync the mirrors. I'm a fan of LVM but where it's appropriate. -- BillMcGonigle - 11 Feb 2006

The boot problem is easily solved with a small /boot partition protected by plain RAID (md). As for why LVM, as I said, this gives us flexibility in our partition layout. Say we make /var/log too small and need to make it bigger, or /home too big or small, or whatever. LVM makes it trivial to change, with minimal downtime. Without LVM, you have to take the filesystems offline and possibly spend forever doing a risky shift-in-place operation. While LVM does mean some added complexity, as you pointed out for mirroring, it's pretty well automated in most distros. Other then complexity (which I feel is a valid point, just perhaps not an overriding one), are there objections to LVM? -- BenScott - 16 Feb 2006

• Software RAID on linux is now faster and has always been more servicable than hardware RAID. Most RH-based installers handle install to RAID just fine, given enough mouse clicks - converting after the fact is a big exercise so let's not do that. Sample GRUB below. -- BillMcGonigle - 11 Feb 2006

Configuration Directives/Snippits

• Root on RAID grub example

title Fedora Core (2.6.15-1.1831_FC4)
        root (hd0,0)
        kernel /vmlinuz-2.6.15-1.1831_FC4 ro root=/dev/md1
        initrd /initrd-2.6.15-1.1831_FC4.img

• You can't boot from LVM and LVM usually implies you'll be adding disks. Since the server is maxed out on disks, this is just an extra layer of complexity. If we're going to just increase the disk size in one of the two slots it's easy enough to copy the data from the 18GB disk to whatever we wind up with, then re-sync the mirrors. I'm a fan of LVM but where it's appropriate. -- BillMcGonigle - 11 Feb 2006

• I mostly agree - it might be worth considering a separate /boot partition if we're going to have terminal server access to this machine and the / filesystem becomes corrupted, we at least stand a chance of fsck'ing it back to life remotely. We certainly shouldn't try to predict usage of /home vs /usr/local vs /opt vs /var as the specs are too loosely defined. -- BillMcGonigle - 11 Feb 2006

-- BenScott - 10 Feb 2006

-- DrewVanZandt - 10 Feb 2006

Mirroring

I (BenScott) think we should mirror the two disks (RAID level 1). There is no service contract or tech support, physical access is limited, and we have limited manpower and no budget. A disk failure could knock us offline for weeks. At the same time, our current disk usage is small.

While I always prefer hardware RAID for the boot drive, the server does not have a controller, and we have no money to buy one. So, that means software RAID, which, as far as I know, does work pretty well on Linux. However I haven't done one of these in a long time, and never with GRUB. Do others have recent experience?

-- BenScott - 10 Feb 2006

LVM

I'm learning towards using LVM to create containers for each partition. That gives us the flexibility to re-organize our partitions and such later on. Given that our plans for the future are very much up in the air, that is a real benefit. Anyone have any objections or suggestions with LVM? -- BenScott - 10 Feb 2006

Partition and directory layout

This topic tends to invite holy wars. That's usually a sign that the topic lacks objective, one-size-fits-all criteria. Please keep that in mind as we all put forth our preferences and opinions. -- BenScott - 10 Feb 2006