<<O>>  Difference Topic ServerStorage (r1.14 - 22 Feb 2009 - BenScott)

META TOPICPARENT InternetServer
Changed:
<
<
Our InternetServer has two SCSI hard drives. They are both roughly 18 gigs, but are not the same model or speed. Details at ServerHardware.
>
>
Our InternetServer has twin 73 GB SCSI hard drives. Details at ServerHardware.

Mirroring

Changed:
<
<
I (BenScott) think we should mirror the two disks (RAID level 1). There is no service contract or tech support, physical access is limited, and we have limited manpower and no budget. A disk failure could knock us offline for weeks. At the same time, our current disk usage is small.
>
>
Consensus was that disk mirroring (RAID level 1) was appropriate. There is no service contract or tech support, physical access is limited, and we have limited manpower and no budget. A disk failure could knock us offline for weeks. At the same time, our current disk usage is small.

Changed:
<
<
While I always prefer hardware RAID for the boot drive, the server does not have a controller, and we have no money to buy one. So, that means software RAID, which, as far as I know, does work pretty well on Linux. However I haven't done one of these in a long time, and never with GRUB. Do others have recent experience?
>
>
The server did not come with a hardware RAID controller, and we have no money to buy one. So, that means software RAID. It works pretty well on Linux. Many installers can set it up for you, including the ones that come with Red Hat derived OSes.

Changed:
<
<
-- BenScott - 10 Feb 2006
>
>
Software RAID also has some advantages over hardware controllers:

Changed:
<
<
Software RAID on linux is now faster and has always been more servicable than hardware RAID. Most RH-based installers handle install to RAID just fine, given enough mouse clicks - converting after the fact is a big exercise so let's not do that. Sample GRUB:
>
>
  • The disks are portable without having to worry about the controller
  • It's very well tested
  • There are no proprietary innards we can't see or fix ourselves
  • We have CPU power to spare for RAID overhead

Changed:
<
<
title Fedora Core (2.6.15-1.1831_FC4)
        root (hd0,0)
        kernel /vmlinuz-2.6.15-1.1831_FC4 ro root=/dev/md1
        initrd /initrd-2.6.15-1.1831_FC4.img

-- BillMcGonigle - 11 Feb 2006

>
>
Booting from RAID 1 works well with GRUB. The boot partition has to be a simple mirror (no LVM, RAID 5, etc.). It thus appears to GRUB like an ordinary partition. Since GRUB never writes to the filesystem, it doesn't matter that the partition is actually a member of a mirror set. By installing GRUB on to both disks this way, either disk becomes bootable. This is best done by booting GRUB from floppy or CD, as differences between how the BIOS (and thus GRUB) and the Linux kernel see disks can matter.

Changed:
<
<
I second the SW RAID 1 suggestion, and have used SW RAID extensively including for boot. grub can deal with booting from either disk, though you do have to install it twice (once on each MBR.) Are there details on space available in the machine? (I'm assuming there's not much left in a 1U...) - If there's room, I can volunteer an 80 GB IDE for backups/bulk storage etc. Note that both the RAID personality and the IDE controller driver must be compiled into the kernel for this.
>
>
On the the kernel boot command line, specify the RAID/LVM device (/dev/md0 or whatever) for the root filesystem to mount. Do not specify physical devices here. You can use whatever kind of device you like. The kernel uses an initial ramdisk to do things like activate RAID devices and LVM volumes; it's only the boot partition that has to be a simple mirror.

Changed:
<
<
-- DrewVanZandt - 10 Feb 2006
>
>
Example GRUB stanza:

Changed:
<
<
The only physical space not consumed is around the two open PCI slots. The ServerHardware page has been updated with pix, more details, and OEM docs.

-- BenScott - 16 Feb 2006

By the way, it is not true that IDE driver and RAID support needs to be compiled into the kernl. You can use an initrd (initial RAM disk). The initrd contains the needed modules, and is loaded (along with the kernel image) by GRUB. This is how we are actually doings things. Works fine.

-- BenScott - 15 Oct 2006

>
>
title CentOS (2.6.18-92.1.17.el5)
   root (hd0,0)
   kernel /vmlinuz-2.6.18-92.1.17.el5 ro root=LABEL=ROOT vga=6
   initrd /initrd-2.6.18-92.1.17.el5.img

LVM

Changed:
<
<
I'm learning towards using LVM to create containers for each partition. That gives us the flexibility to re-organize our partitions and such later on. Given that our plans for the future are very much up in the air, that is a real benefit. Anyone have any objections or suggestions with LVM? -- BenScott - 10 Feb 2006
>
>
We're using LVM to create containers for each filesystem. That gives us the flexibility to re-organize our filesystems and such later on. Given that our plans for the future are always very much up in the air, that is a real benefit. Say we make /var/log too small and need to make it bigger, or /home too big or small, or whatever. LVM makes it trivial to change, with minimal downtime. Without LVM, you have to take the filesystems offline and possibly spend forever doing a risky shift-in-place operation.

Changed:
<
<
You can't boot from LVM and LVM usually implies you'll be adding disks. Since the server is maxed out on disks, this is just an extra layer of complexity. If we're going to just increase the disk size in one of the two slots it's easy enough to copy the data from the 18GB disk to whatever we wind up with, then re-sync the mirrors. I'm a fan of LVM but where it's appropriate. -- BillMcGonigle - 11 Feb 2006
>
>
One drawback is that this is an extra layer of complexity. On the plus side, it's pretty well automated in most distros.

Changed:
<
<
The boot problem is easily solved with a small /boot partition protected by plain RAID (md). As for why LVM, as I said, this gives us flexibility in our partition layout. Say we make /var/log too small and need to make it bigger, or /home too big or small, or whatever. LVM makes it trivial to change, with minimal downtime. Without LVM, you have to take the filesystems offline and possibly spend forever doing a risky shift-in-place operation. While LVM does mean some added complexity, as you pointed out for mirroring, it's pretty well automated in most distros. Other then complexity (which I feel is a valid point, just perhaps not an overriding one), are there objections to LVM? -- BenScott - 16 Feb 2006
>
>

Layout


Changed:
<
<
Oh, and I do believe that LVM does make any potential future disk expansion easier. We can easily do a staggered disk replacement (replace one disk, remirror, replace next disk, remirror), add new partitions and make a mirror of them, put an LVM PV on that, and then grow filesystems. Without LVM, how would we do that? Can you resize an md physical device? -- BenScott - 16 Feb 2006
>
>
Each disk is partitioned into a small boot partition and a large LVM PV (Physical Volume) partition. Linux RAID (md device) is used to mirror each pair of partitions. Thus, the partition type for each is Linux RAID auto-detect.

Changed:
<
<

Partition and directory layout

This topic tends to invite holy wars. That's usually a sign that the topic lacks objective, one-size-fits-all criteria. Please keep that in mind as we all put forth our preferences and opinions. -- BenScott - 10 Feb 2006

>
>
Disk /dev/sda: 8924 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Changed:
<
<
Here's the first "holy salvo"... Put everything (except swap) in one partition. Why? There's no good reason not too (unless we're going to use dump/restore to tapes). -- BruceDawson - 10 Feb 2006
>
>
Device Boot Start End #cyls #blocks Id System /dev/sda1 * 0+ 64 65- 522081 fd Linux raid autodetect /dev/sda2 65 8923 8859 71159917+ fd Linux raid autodetect /dev/sda3 0 - 0 0 0 Empty /dev/sda4 0 - 0 0 0 Empty

Changed:
<
<
I mostly agree - it might be worth considering a separate /boot partition if we're going to have terminal server access to this machine and the / filesystem becomes corrupted, we at least stand a chance of fsck'ing it back to life remotely. We certainly shouldn't try to predict usage of /home vs /usr/local vs /opt vs /var as the specs are too loosely defined. -- BillMcGonigle - 11 Feb 2006
>
>
LVM runs on top of the mirrored PV partition; LVM sees a single PV (/dev/md1). A single volume group liberty_vg exists. It contains several different LVs (Logical Volumes), each with a different purpose in the filesystem.

Changed:
<
<
Well, I'm from the "divide things up into a bunch of partitions" camp. This ties in with why I favor LVM -- it means we don't have to predict allocation. We can guess and change things later. (If we're going to go with only one partition, then I would say there is no reason to use LVM, either -- plain RAID should be enough.) As for why I like multiple partitions: Party, it's just my nature -- a place for everything and everything in its place and all that. It keeps runaway output from killing the system (/var/log and /var/spool on separate partitions). It makes an OS reinstall/etc much easier. It localizes fragmentation. All that being said, if everyone else is in the "one big partition" camp, I guess we can do that (awwwww smile ). -- BenScott - 16 Feb 2006
>
>
  LV          Size
  home_lv     4.00 G
  local_lv    4.00 G
  log_lv      1.00 G
  mailman_lv  2.00 G
  recovery_lv 4.00 G
  root_lv     1.00 G
  sites_lv    4.00 G
  spool_lv    1.00 G
  swap_lv     1.00 G
  tmp_lv      1.00 G
  usr_lv      4.00 G
  var_lv      1.00 G

Changed:
<
<
KISS. -- BillMcGonigle - 17 Feb 2006
>
>
* 39.84 GB unallocated

Changed:
<
<
If we have all the paritions divided up we'll need to plan on downtime as you're not going to be resizing filesystems on the fly even if you do resize LVM on the fly. I don't like resizing root filesystems remotely without a terminal server. If someone wants to volunteer to do this even if it means going on site with an SLA, fine, but that can't be me. I agreee with Ben's strategy on systems with lots of disk and local or terminal access. -- BillMcGonigle - 17 Feb 2006
>
>
Not everyone on the server team was in agreement, so it mainly came down to "the guy doing most of the work at the time picked what he liked best". The guy at the time was Ben Scott. His reasons for his approach:

Changed:
<
<
Well, assuming one does use multiple partitions and mount points and such, it's fairly easy to avoid ever having to worry about the the root filesystem. The point about downtime costs (due to taking a filesystem offline to shrink it) can't be so easily dismissed. While EXT2/3 doesn't take too long to shrink (and can be grown while mounted), it's still downtime. Hmmmm. /me ponders. -- BenScott - 18 Feb 2006
>
>
  • A place for everything and everything in its place.
  • It keeps runaway output from killing the system (/var/log and /var/spool especially).
  • It makes any kind of OS reinstall/etc much easier.
  • It localizes filesystem fragmentation.
  • It localizes any filesystem corruption.

Drawbacks to this approach include:

  • More complicated
  • Resizing filesystems generally involves at least a partial service interruption
  • Resizing a root filesystem is rather difficult
    • However, by using separate LVs, the root filesystem should "never" need to be resized

Revision r1.13 - 14 Oct 2006 - 23:47 - BenScott
Revision r1.14 - 22 Feb 2009 - 22:07 - BenScott