Home > Solaris, Storage > Expanding ZFS zpool RAID

Expanding ZFS zpool RAID

I’m big fan of ZFS and all the volume managing options it offers. It’s often that ZFS makes hard things easy and impossible things possible. In an era of ever growing data sets, sysadmins are regularly pressed with the need to expand volumes. While this may be easy to accomplish in an enterprise environment with IBM or Hitachi storage solutions, problems arise on mid and low level servers. Most often expanding volumes means online rsync to new data pool, then another rsync while the production system is down and finally putting new system to production. ZFS makes this process a breeze.

Here is one example where ZFS really shines. Take a look at the following pool:

# zfs list | grep "tank "
tank                        1.75T  31.4G  40.0K  /tank

and it’s geometry:

# zpool status tank
  pool: tank
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c2t0d0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
          raidz1-1  ONLINE       0     0     0
            c2t3d0  ONLINE       0     0     0
            c2t4d0  ONLINE       0     0     0
            c2t5d0  ONLINE       0     0     0

errors: No known data errors

1.75TiB pool is slowly getting filled. As you see, it’s 6 disk pool consisted of two RAID-Z’s in stripe. It is approximate of RAID 50, in normal RAID nomenclature. It’s a lot of data to rsync over, isn’t it? Well, ZFS offers one neat solution. We could replace single disk with a bigger one, rebuild RAID and repeat the procedure 6 times. Finally, after last rebuild, we could ‘grow’ the pool to new size. In this particular case I decided to replace 500GB Seagate’s with 2TB Western Digital drives.

This is how the pool looks like after disk c2t5d0 is replaced with 2TB drive:

# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scan: resilvered 449G in 7h9m with 0 errors on Mon Dec 24 20:58:51 2012
config:

        NAME        STATE     READ WRITE CKSUM
        tank        DEGRADED     0     0     0
          raidz1-0  ONLINE       0     0     0
            c2t0d0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
          raidz1-1  DEGRADED     0     0     0
            c2t3d0  ONLINE       0     0     0
            c2t4d0  ONLINE       0     0     0
            c2t5d0  UNAVAIL      0     0     0  cannot open

errors: No known data errors

Now we need to tell ZFS to rebuild the pool:

# zpool replace tank c2t5d0 c2t5d0

After this command, rebuild process will start. Few hours later, state of the system is:

# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Tue Jan  1 14:43:22 2013
    91.6M scanned out of 2.63T at 3.52M/s, 217h26m to go
    91.6M scanned out of 2.63T at 3.52M/s, 217h26m to go
    14.5M resilvered, 0.00% done
config:

        NAME              STATE     READ WRITE CKSUM
        tank              DEGRADED     0     0     0
          raidz1-0        ONLINE       0     0     0
            c2t0d0        ONLINE       0     0     0
            c2t1d0        ONLINE       0     0     0
            c2t2d0        ONLINE       0     0     0
          raidz1-1        DEGRADED     0     0     0
            c2t3d0        ONLINE       0     0     0
            c2t4d0        ONLINE       0     0     0
            replacing-2   DEGRADED     0     0     0
              c2t5d0/old  FAULTED      0     0     0  corrupted data
              c2t5d0      ONLINE       0     0     0  (resilvering)

errors: No known data errors

After the process is finished, pool will look something like this:

# zpool status tank
  pool: tank
 state: ONLINE
 scan: resilvered 449G in 7h1m with 0 errors on Tue Jan  1 21:44:37 2013
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c2t0d0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
          raidz1-1  ONLINE       0     0     0
            c2t3d0  ONLINE       0     0     0
            c2t4d0  ONLINE       0     0     0
            c2t5d0  ONLINE       0     0     0

errors: No known data errors

Once when all disks are replaced, only thing needed to grow the pool is to set autoexpand to on. If it was already on, then first turn it to off and then turn it again to on to grow the pool:

# zfs list | grep "tank "
tank                        1.75T  31.4G  40.0K  /tank
# zpool set autoexpand=off tank
# zpool set autoexpand=on  tank
# zfs list | grep "tank "
tank                        1.75T  5.40T  40.0K  /tank

And that’s it! We’ve grown striped 2x RAID-Z configuration from 500GB drives to 2TB drives, growing the total size from 1.7 TB to 7.10 TB. Enjoy the wonders of ZFS!

Advertisements
Categories: Solaris, Storage Tags: , , , , ,
  1. Mark
    September 15, 2015 at 5:13 am

    Not really a breeze compared to say Linux software RAID where you can just add new drive(s) to the array, grow the filesystem, and you’re done. ZFS can’t add new drives to an existing pool, which is a pretty serious limitation. This is especially limiting when you already have a pool of large drives (say 4TB) and need to expand.

    • September 19, 2015 at 10:02 pm

      While it’s true you can’t expand number of disks in zpool vdev, ZFS still has other benefits over other solutions, like send/recv, more usable snapshot/clones, etc.

  2. UX-admin
    October 2, 2015 at 7:31 am

    “ZFS can’t add new drives to an existing pool,”
    “While it’s true you can’t expand number of disks in zpool,”

    Both of these assertions are incorrect: ZFS can add an unlimited number of devices to a pool, AND the newly added devices will automatically form a vdev, AND this vdev will be striped along with the rest of the vdevs, AND it is possible to mix and match different types of vdevs, AND this functionality has been available in ZFS since its inception, back in 2005.

    • October 2, 2015 at 11:39 am

      I missed the “zpool vdev”. So matter of speach.

  3. June 23, 2016 at 2:10 pm

    Say I have 2 mirrored vdevs in a pool. Can I expand 1 vdev with larger disks and get more space in my pool?

    • NeBot
      July 20, 2017 at 10:26 pm

      No. The reason for this is that the mirror has the vdev size of the smallest disk in the set. When you have, say, a 500gb drive pair as a mirror and replace one of them with a 2tb drive, the disk will behave as a 500gb drive until both disks are 2tb in size. then the mirror will grow to the smallest disk in the mirror set. ZFS will not allow storage to a portion of a disk that violates the rules of the set, in this case using a portion on disk A that cannot be mirrored to the other drive due to it being smaller.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: