« Software development resources
Microsoft Forefront antivirus standalone installation »

Upgrading Linux software RAID-1 array

I just finished upgrading my Debian Etch fileserver from 2×200GB IDE disks to 2×500GB SATA disks. I managed to keep the server running for nearly the entire time, by failing and hot-adding disks to the RAID-1 arrays. If I had room in the case for more than two disks it would have been even easier.

Here is the configuration BEFORE:

  • /dev/hda partitioned into hda1 (10GB), hda2 (1GB), hda3 (175GB)
  • /dev/hdc partitioned into hdc1 (10GB), hdc2 (1GB), hdc3 (175GB)
  • RAID-1 array md0 composed of hda1 and hdc1, mounted as /
  • RAID-1 array md1 composed of hda2 and hdc2, mounted as swap
  • RAID-1 array md2 composed of hda3 and hdc3, mounted as /home

I started the ball rolling by failing one partition from each RAID array:

mdadm –fail /dev/md0 /dev/hdc1
mdadm –fail /dev/md1 /dev/hdc2
mdadm –fail /dev/md2 /dev/hdc3

Then I powered down the server, disconnected and removed hdc and added a new 500GB SATA disk to the SATA PCI card. It booted up fine with all three RAID arrays degraded. I used fdisk to partition the new SATA disk (/dev/sda) with identical sized partitions 1 and 2, and with the third partition taking up the remainder of the disk. I set all partition types to fd (linux raid auto-detect):

  • sda1 (10GB), sda2 (1GB), sda3 (454GB)

Then one at a time I hot-added these partitions to the running RAID arrays. This causes a background reconstruction, so it’s worth waiting for each to finish before starting the next:

mdadm –add /dev/md0 /dev/sda1
mdadm –add /dev/md1 /dev/sda2
mdadm –add /dev/md2 /dev/sda3

When all three were completely synced (cat /proc/mdstat to see the progress), I edited /etc/mdadm/mdadm.conf to change all references from /dev/hdcx to /dev/sdax. I then re-built the initramfs so it knew how to start the arrays at boot time:

update-initramfs -k all -c -t

I then powered down the server again, removed the last IDE disk (hda) and added the second SATA disk (sdb). At this point the system is unbootable, so I started from a rescue CD (actually the Debian Etch netinst cd, starting with the “rescue” boot option). Once I got a command prompt (Alt-F2 and Alt-F3 virtual consoles), I installed grub:

mount /dev/sda1 /mnt
chroot /mnt /bin/bash
nano /boot/grub/device.map

I edited the device.map so it looked like this:

(hd0) /dev/sda
(hd1) /dev/sdb

Then installed grub on the first SATA disk:

grub-install /dev/sda

I rebooted and grub succesfully booted the server. As expected, all RAID arrays were in degraded mode. I used fdisk to re-partition the second SATA disk to match the first, then hot-added the mirrors to the RAID arrays (waiting for each re-sync to complete before starting the next):

mdadm –add /dev/md0 /dev/sdb1
mdadm –add /dev/md1 /dev/sdb2
mdadm –add /dev/md2 /dev/sdb3

Then I edited /etc/mdadm/mdadm.conf to update the partitions to (for example) sda1,sdb1. I re-build the initramfs again as above, and rebooted to test everything booted up cleanly. Also I checked /proc/mdstat after the reboot to check all arrays were fully functional.

So now the new disks are installed, but there’s no extra storage available because the ext3 partition is still set to the old size! I rebooted into single user mode, unmounted /home, then used resize2fs to expand the filesystem to use the whole partition:

e2fsck -f /dev/md2
resize2fs /dev/md2

One reboot later and voila, 455GB usable in /home.

Then I followed this guide to install grub on the second RAID disk in a bootable way.

Tags: ,

This entry was posted on Wednesday, October 24th, 2007 at 9:40 pm and is filed under Debian, Work. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

2 Responses to “Upgrading Linux software RAID-1 array”

  1. Darrell Vermaak Says:
    May 15th, 2008 at 10:20 am

    Thanks for the article – very useful!

    I was using a Reiserfs on RAID1 – had to resize the raid after repartitioning the disks and hot adding them to the raid by using
    mdadm –grow /dev/md1 –size=max
    then unmount the /dev/md1
    then run resize_reiserfs /dev/md1
    this resyncs the disks/raid – to speed up the process I used
    echo 200000 > /proc/sys/dev/speed_limit_max
    echo 200000 > /proc/sys/dev/speed_limit_min

  2. Daniel Kahn Gillmor Says:
    February 9th, 2009 at 11:09 pm

    It seems to me like the above process is more painful than it needs to be today.

    * you can run grub-install /dev/sda && grub-install /dev/sdb before the second reboot. that way, if your BIOS is capable of booting from the second disk, you shouldn’t need to wrestle with the rescue CD

    * you can resize ext3 filesystems while they’re mounted today (as long as they’re growing, not shrinking) — no need to drop to single-user mode.

Leave a Reply