Direct Attached Storage

From Indie IT Wiki
Jump to: navigation, search
eSATA DAS

Introduction

XCASE: Mini Tower Raid System. Ideal for home server or workplace storage. 5 Hotswap Bays.

http://www.xcase.co.uk/Cfi-B8253ER-5-Drive-Hotswap-External-Raid-System-p/storage-cfi-b8253er.htm

This is 5 bay DAS (Direct Attached Storage) enclosure for high capacity storage needs. With built-in ventilating protective screen front panel, the unit utilizes cable-less backplane design to support 5 hard drives. Hard drives are installed on an easy access tray for simply insertion and removal. Hard drive cooling fan located on the back of unit. It is a compact RAID tower use only one cable to access 5 SATA hard drives utilizing the SATA II port multiplier technology and also support RAID 0, 1, 5, 10.

I have decided to use this device in a Linux Software RAID Level 10 (RAID-10) array set up with four 1.5TB hard disk drives - giving me a total of 3TB of file storage space.

Linux Software MD RAID Level 10

The Linux kernel software RAID driver (called md, for "multiple device") can be used to build a classic RAID 1+0 array...

http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10

8 x HDD RAID 10 means striping 4 pairs of RAID 1. As long as any two failed drives are not in the same paired RAID 1, the RAID 10 will just keep on going without any sign of slowing down. Thus, you can have up to 4 failed drives in 8 x HDD RAID 10, and the array remains intact. However, you don't really want to play russian roulette, and you ought to replace any failed drive as soon as you can. A degraded RAID 10 won't suffer performance loss, but a degraded RAID 5 will slow down by ~50%.

4 x HDD

mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
 a=b    RAID 1
  +               RAID 0
 c=d    RAID 1

8 x HDD

mdadm --create /dev/md0 --level=10 --raid-devices=8 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
 a=b    RAID 1
  +
 c=d    RAID 1
  +               RAID 0
 e=f    RAID 1
  +
 g=h    RAID 1

In my 8 drive DAS...

[8|/dev/sde|WD-WMAZA1547403]\
            =                RAID 1
[7|/dev/sdd|WD-WMAZA1455936]/
            +
[6|/dev/sdc|WD-WMAZA1455763]\
            =                RAID 1
[5|/dev/sdb|WD-WMAZA1461618]/
            +                        RAID 0
[4|/dev/sdi|WD-WMAZA1263294]\
            =                RAID 1
[3|/dev/sdh|WD-WMAZA1447312]/
            +
[2|/dev/sdg|WD-WCAZA6573412]\
            =                RAID 1          (which is why it was only using these 2 drives when recovering)
[1|/dev/sdf|WD-WMAZA1548052]/

Installation In Ubuntu Linux

  1. Add hard drives
  2. Plug in esata and power cables
  3. Plug into Ubuntu Linux PC
  4. Partition each drive as type 'fd'
  5. Use 'mdadm' to create RAID10 array
  6. Format with chosen file system
  7. Use and enjoy!

http://www.youtube.com/watch?v=fIWLaqT0-DQ

More to come. For now, command line history... :-)

cat /proc/partitions 
modprobe raid10
mknod /dev/md0 b 9 0
fdisk /dev/sdb
sfdisk -d /dev/sdb | sfdisk /dev/sdc
sfdisk -d /dev/sdb | sfdisk /dev/sdd
sfdisk -d /dev/sdb | sfdisk /dev/sde
apt-get install mdadm
cat /proc/mdstat
mdadm --create /dev/md0 --verbose --raid-devices=4 --level=10 --assume-clean /dev/sdb1 /dev/sdc /dev/sdd1 /dev/sde1
cat /proc/mdstat
mkfs.ext4 /dev/md0
reboot

fdisk output - one hard disk drive

# fdisk -l /dev/sdb
Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x97837bce
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      182401  1465136001   fd  Linux raid autodetect

Partitions output

# cat /proc/partitions 
major minor  #blocks  name
  8        0 1465138584 sda
  8        1 1454002956 sda1
  8        2          1 sda2
  8        5   11133013 sda5
  8       16 1465138584 sdb
  8       17 1465136001 sdb1
  8       32 1465138584 sdc
  8       33 1465136001 sdc1
  8       64 1465138584 sde
  8       65 1465136001 sde1
  8       48 1465138584 sdd
  8       49 1465136001 sdd1
  9        5 2930271872 md0

Kernel output

[   74.873945] md: bind<sdb1>
[   74.875586] md: bind<sde1>
[   74.877133] md: bind<sdd1>
[   75.075389] md: bind<sdc1>
[   75.086003] raid10: raid set md0 active with 4 out of 4 devices
[   75.086021] md0: detected capacity change from 0 to 3000598396928

MD output

# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid10 sdc1[1] sdd1[2] sde1[3] sdb1[0]
      2930271872 blocks 64K chunks 2 near-copies [4/4] [UUUU]
unused devices: <none>

df output

# df -H -T -x tmpfs
Filesystem    Type     Size   Used  Avail Use% Mounted on
/dev/sda1     ext4     1.5T   314G   1.1T  23% /
/dev/md0      ext4     3.0T   2.0T   805G  72% /mnt/3TB

EDGE10 DAS 801 t

Edge10 DAS 801t

http://www.edge10.com/digital_storage.php

This is the listing with ONE of my drives which has failed...

[8|/dev/sde|WD-WMAZA1547403]
[7|/dev/sdd|WD-WMAZA1455936]
[6|/dev/sdc|WD-WMAZA1455763]
[5|/dev/sdb|WD-WMAZA1461618]
[4|/dev/sdh|WD-WMAZA1263294]
[3|/dev/sdg|WD-WMAZA1447312]
[2| empty  |               ]
[1|/dev/sdf|WD-WMAZA1548052]
cat /proc/partitions /proc/mdstat 

major  minor   #blocks  name
  8        0 1465138584 sda
  8        1 1457032242 sda1
  8        2    8106310 sda2
  8       16 1953514584 sdb
  8       17 1953514552 sdb1
  8       32 1953514584 sdc
  8       33 1953514552 sdc1
  8       48 1953514584 sdd
  8       49 1953514552 sdd1
  8       64 1953514584 sde
  8       65 1953514552 sde1
  8       80 1953514584 sdf
  8       81 1953514552 sdf1
  8       96 1953514584 sdg
  8       97 1953514552 sdg1
  8      112 1953514584 sdh
  8      113 1953514552 sdh1
254      384 7814057728 md_d6

md_d6 : active raid10 sdc1[1] sdf1[4] sde1[3] sdb1[0] sdh1[7] sdd1[2] sdg1[6]
      7814057728 blocks 64K chunks 2 near-copies [8/7] [UUUUU_UU]

UPDATE: Tuesday, 23 August 2011

Partition the new drive, using one of the other drives to copy...

sudo sfdisk -d /dev/sdc |sfdisk --force /dev/sdg

Check the drive has partitioned correctly...

sudo fdisk -l /dev/sdg

Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
   Device Boot      Start         End      Blocks   Id  System
/dev/sdg1               1      243202  1953514552   fd  Linux raid autodetect

Hot add the new drive to the RAID10 array...

sudo mdadm --manage /dev/md_d6 --add /dev/sdg1

Check the kernel log...

dmesg |tail -n 15

[  983.627653] md: bind<sdg1>
[  984.075096] RAID10 conf printout:
[  984.075104]  --- wd:7 rd:8
[  984.075111]  disk 0, wo:0, o:1, dev:sdb1
[  984.075117]  disk 1, wo:0, o:1, dev:sdc1
[  984.075122]  disk 2, wo:0, o:1, dev:sdd1
[  984.075127]  disk 3, wo:0, o:1, dev:sde1
[  984.075132]  disk 4, wo:0, o:1, dev:sdf1
[  984.075136]  disk 5, wo:1, o:1, dev:sdg1
[  984.075140]  disk 6, wo:0, o:1, dev:sdh1
[  984.075145]  disk 7, wo:0, o:1, dev:sdi1
[  984.075259] md: recovery of RAID array md_d6
[  984.075264] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  984.075269] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[  984.075289] md: using 128k window, over a total of 1953514432 blocks.

Which means...

wd == working disks
rd == raid disks
wo == write-only
 o == online

Check the RAID status...

cat /proc/mdstat

md_d6 : active raid10 sdg1[8] sdf1[4] sdc1[1] sdh1[6] sdi1[7] sdd1[2] sde1[3] sdb1[0]
      7814057728 blocks 64K chunks 2 near-copies [8/7] [UUUUU_UU]
      [>....................]  recovery =  1.8% (35372032/1953514432) finish=493.3min speed=64795K/sec

This is the listing with all of my drives...

[8|/dev/sde|WD-WMAZA1547403]
[7|/dev/sdd|WD-WMAZA1455936]
[6|/dev/sdc|WD-WMAZA1455763]
[5|/dev/sdb|WD-WMAZA1461618]
[4|/dev/sdi|WD-WMAZA1263294]
[3|/dev/sdh|WD-WMAZA1447312]
[2|/dev/sdg|WD-WCAZA6573412]
[1|/dev/sdf|WD-WMAZA1548052]

UPDATE: Wednesday, 1 May 2012

Today, I went to mount my DAS and found that all the drives were marked as SPARE. Panic set in (obviously).

$ cat /proc/partitions /proc/mdstat 
major minor  #blocks  name
           0 1465138584 sda
  8        1 1457032242 sda1
  8        2    8106310 sda2
  8       16 1953514584 sdb
  8       17 1953514552 sdb1
  8       32 1953514584 sdc
  8       33 1953514552 sdc1
  8       48 1953514584 sdd
  8       49 1953514552 sdd1
  8       64 1953514584 sde
  8       65 1953514552 sde1
  8       80 1953514584 sdf
  8       81 1953514552 sdf1
  8       96 1953514584 sdg
  8       97 1953514552 sdg1
  8      112 1953514584 sdh
  8      113 1953514552 sdh1
  8      128 1953514584 sdi
  8      129 1953514552 sdi1
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md_d6 : inactive sdg1[5](S) sdb1[0](S) sdi1[7](S) sdh1[6](S) sdf1[4](S) sde1[3](S) sdc1[1](S)
     13674601024 blocks
unused devices: <none>

The solution? Power everything off. Wait 5 minutes. Try again. If all OK, put the correct info into mdadm.conf:-

sudo mdadm --scan --detail >> /etc/mdadm/mdadm.conf
sudo service mdadm restart

The system had a Funny Five Minutes... but it was not funny for me.