Tuesday, September 1, 2009

Cheap Man's 40-Disk Storage Cluster


Table of Contents

Introduction

This is a computer case design that fits 40 disks, 4 motherboards, 5 power supplies, a bunch of fans and a gigabit switch.

The main goal of this design is to use the cheapest parts with the least effort to assemble everything. So you could say this is also Poor/Lazy Man's Storage Cluster!
The area used is 60x60x50 (WxDxH) cm only.

Keep in mind that this is a case not a whole system. I've only factored the price of the pieces used to put the case together.

Parts and Prices

Metal table with 2 net-like surfaces7 KDIKEA
2x Wooden CD rack that fits 35 CDs2x3 KDIKEA
Plastic drawer mat1.75 KDIKEA
Rubber grooved floor mat (3mm thick)1.5 KD for 0.5 meterTrueValue
Nylon Cable Ties (203x3.2mm)0.5 KDFamily Hardware Store
Total16.75 KD

Tools

  • Hands
  • Foot
  • Long nose pliers
  • Scissors
  • Hammer (or anything that hammers)

Design Diagram


In this diagram you see the measurements of each component and how they fit. When we put the disks inside the rack, there was an empty space of 4mm; we took care of that by using the rubber mat, which is 3mm thick on each side, totaling to 6mm, which helps holding the disks and serve as a shock absorber.

The disk rack is made of wood and we have rubber mats inside, so you'd expect for heat to be trapped. Our work around is this:
  • Choosing a rubber mat with grooves
  • Inserting the disks heads down having the 2.5cm edges touching the rubber
  • Leaving space between disks
  • Pushing the disks down till touching the metal table
  • Installing fans on the lower part of the table, blowing at the disks

Note that I have used Layout #1. Layout #2 was too cluttered and I didn't really think it through properly so I don't know if it's even possible. If you are able to squeeze more than 4 motherboards in that same table (or same dimensions), let me know!

Assembly

  1. Assemble lower shelf of table
  2. Use foot to break back-panel of CD racks
  3. Hammer the metal pins of the CD racks inwards
  4. Point the side that has the metal pins towards table surface (keeps wood fragments away from you)
  5. Tie the rack to the table using the cable ties
    Note: We assembled the upper surface but worked on the lower one later and kept the upper free for future motherboards.
  6. Cut 11 lines (in a group) of the rubber mat
  7. Attach the rubber mat to a side and tie it down. Do the same for the other side
    We have cut out the extra edges shown in the first picture to reduce heat contraption.
  8. If you want to have a separate power supply unit (PSU) for fans, attach it to the bottom of the lower surface
    Note: I'll tell you later how to turn on the PSU without a motherboard (jump-starting).If you're going to run anything at the bottom, now is the time to attach them. You won't be able to do it later on!
  9. Put the power supplies in place and tie them (make sure it's touching the metal table)
  10. Cut plastic mat to fit rest of table area and tie it to the table. Use the nail to punch holes
  11. Punch extra holes for the motherboards and don't tie the motherboards too hard!
  12. Tie fans below the disk rack(s) and connect them in serial to a PSU
    Note: We made a mistake above and tied the fans in the opposite direction of the motherboard and were too lazy to reposition them.
  13. Slap in the hard disk drives (HDDs) and hook them to the motherboard(s)
  14. Powering on
    To power on a motherboard, you could either use a power switch (or make one), or enable Wake-up On LAN (WOL) from the BIOS (assuming your motherboard supports it. You'd need to know the MAC address of your LAN port.
    We enabled WOL but it didn't work for some reason. Crappy MSI board.
  15. If your motherboard doesn't have a built-in video/graphics card, you'll need to bend the tip of the graphics card you're about to attach
  16. Almost done. Attach the upper surface and make sure that the rack is facing the opposite side to the one on the lower surface, so that the fans don't hit the cables

    Make sure to double check on the HDD cables after attaching the upper surface.
  17. Jump-starting a PSU
    I stripped a cable wrapper that had a metal piece inside it and stuck it in the proper pins. This way, I control the fan PSU using the ON/OFF switch at the back; no need for a separate power switch. Unfortunately we didn't take pictures of that, but here are some references:

To Cluster Or Not To Cluster

Now that you've had your motherboards all hooked up most likely to a gigabit switch, there are different ways to use all this storage capacity:
  1. Make them appear as a single storage unit
    This can be done through the use of iSCSI. It allows you to expose either each single hard disk or a whole RAID array as a single storage device to another machine over Ethernet. This way you can combine all the disks/arrays under one machine and create a LV (Logical Volume not Louis Vuitton) then the filesystem on top of the LV.
    I don't know how to do this on Windows. I can help you do this on Linux though. If you did this on Windows, drop me an email and I'll link your page.
  2. Use them separately
    Well this is a no-brainer: Just assign a different IP for each machine and expose each storage through Samba (on Linux) or share the directories on Windows.
  3. A mix between the above two
    Using iSCSI puts a lot of risk on the data because if one motherboard, or multiple disks fail (in case of using RAID5) then you lose all your data. For good. And since we have such a good history of increasing and managing the storage smoothly (NOT!), we decided to not use iSCSI. Maybe if we had a better history, we'd gone with it.

    What we're going to do is keep the existing Samba share and move the Anime directory (2.4TB) to another machine. We then mount the other machine using NFS over the existing Anime directory. Mounting a directory over another is called shadowing.
    For this to work properly, you need to create the usernames on all systems with the same IDs, otherwise you'll have a heck of a time with permissions.

    Now, the users still access the same old single IP and can still access all data, though distributed cross systems. If a machine's disks died, at least we won't lose all the data.

    We don't yet have the 2nd motherboard, so I'll write about this in detail when we get it and do the setup.

Post Assembly

After assembling and running the machine, it's been put under heavy load and these are the temperature readings:
/dev/sda: WDC WD10EACS-00D6B1: 27°C
/dev/sdb: WDC WD10EACS-00D6B1: 29°C
/dev/sdc: WDC WD10EACS-00ZJB0: 29°C
/dev/sdd: WDC WD10EADS-65L5B1: 28°C
/dev/sde: WDC WD10EADS-65L5B1: 29°C
/dev/sdf: MAXTOR STM31000340AS: 29°C
/dev/sdg: WDC WD10EACS-00ZJB0: 27°C
/dev/sdh: WDC WD10EADS-00L5B1: 26°C

This is way much better than before! They used to be in the 40's range!!

Last Words

Our baby is running fine now, and for the first time we haven't faced problems, thank God!

If you have any questions or comments, let us know. I suggest you subscribe via email when commenting, or leave a blank comment to just subscribe to stay posted on updates when adding the 2nd motherboard.

Good luck and don't blame us if you get electrocuted ^_^'

10 comments:

Unknown said...

you didn't mention the HDD cost..
also you should mention how ugly it is and let us know when your mother will through it out @@

MBH said...

Abdilhadi,
From the introduction: "Keep in mind that this is a case not a whole system. I've only factored the price of the pieces used to put the case together."

Throw not through.

This baby sits in the dewaniya not home ;)

mario said...

that is one crazy cluster. super job!

MBH said...

mario,
Thanks :D

Bloggylife said...

LOL AWESOME :D

you couldn't resist the typo, could you :P

MBH said...

BloggyLife,
Glad you liked it :D

I'm afraid not, but my buddy Abdilhadi knows pretty well how much of a Spell Nazi I am... he's been suffering since college.

Actually, come to think of it, I used to correct people even when talking (Arabic or English).

I think most are happy that I'm not around anymore. OK OK ... they're ALL happy :p

Pyyth4 said...

lol this is a mad one...

just make sure you put it up in such a way you dont short circuit anything or force sparks to come someway or the other....wood is flammable...

btw i like the shirt in first picture, "die n00b" :P

MBH said...

Pyyth4,
No worries! That's what the plastic mat is for! I like festivals, but not with my hardware :p

That's my favorite t-shirt. I wear it for special occasions :D

Bashar said...

Just where on God's earth did you find this insane idea, and how could you make it so ugly :)

This should go on Digg

MBH said...

Bashar,
I was walking around in IKEA with 3baid, saw the table and got the idea. Instantly.

It's NOT UGLY!!! I call it abstract looks :p
Beauty is in the eyes of the beholder, or beholding the eyes of beauty, or whomever is holding an eye...