Software RAID vs. LVM: Quick Speed Test

Table of Contents

1. Introduction
2. System Setup
3. Speed Test Methods
   1. hdparm
   2. dd
4. Speed Test #1: RAID vs. LVM
5. Speed Test #2: Disks Separately
6. Conclusion

Introduction

Currently, I have a fileserver that is setup this way:

Filesystem
      ^
Logical Volume Manager
      ^
Software RAID Arrays
      ^
Physical Disks

In my case, the LVM is an extra layer and it's not useful since I only have one physical entity that belongs to a Volume Group: A single RAID5 array.
So you could put your filesystem on top of a Logical Volume, or directly on the RAID array device. It depends on how you want to manage your data and devices.

So, is this hampering performance? The tables below will do the talking, but first: the setup.

System Setup

Processor	Intel Pentium Dual CPU E2160 @ 1.80GHz
Motherboard	MSI (MS-7514) P43 Neo3-F
	North Bridge: Intel P43
	South Bridge: Intel ICH10
SATA Controller 1	JMicron 20360/20363 AHCI Controller
	AHCI Mode: Enabled
	Ports: 6-7
Sata Controller 2	82801JI (ICH10 Family) SATA AHCI Controller
	Ports: 0-5
RAM	1GB @ CL 5
Video Card	GeForce 7300 GS
Disk sda	WDC WD10EACS-00D6B1
Disk sdb	WDC WD10EACS-00D6B1
Disk sdc	WDC WD10EACS-00ZJB0
Disk sdd	WDC WD10EADS-65L5B1
Disk sde	WDC WD10EADS-65L5B1
Disk sdf	MAXTOR STM31000340AS
Disk sdg	WDC WD10EACS-00ZJB0
Disk sdh	WDC WD10EADS-00L5B1
Disk sdi	Hitachi HDS721680PLAT80 (OS)
Chunk size	256kB
LVM: Physical Extent Size	1GB
LVM: Read ahead sectors	Auto (set to 256)

Speed Test Methods

A quick and easy way to run a speed test is by using a tool called hdparm and another called dd.
Note that these two utilities don't take the filesystem performance into account, as they read directly from the device, not a certain file. It doesn't matter in this case, as I'm about to show comparisons to show the magnitude of difference speed only, not show very exact results ;)

hdparm

hdparm -tT /dev/xxx
-t: Perform timings of device reads for benchmark and comparison purposes.
Displays  the  speed of reading through the buffer cache to the disk without any prior caching of data.
This measurement is an indication of how fast the drive can sustain sequential data reads under Linux, without any filesystem overhead.

-T: Perform timings of cache reads for benchmark and comparison purposes.
This displays the speed of reading directly from the Linux buffer cache without disk access.
This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.

dd

dd if=/dev/xxx of=/dev/null bs=10M count=400
This will read from the device and dump the data to a null device (just reading). Block size=10 Megabytes (2^20).
This will read 4GB of data. I specified 4GB to make sure that it surpasses the RAM size.

Before running dd, I flushed the read cache by entering: hdparm -f /dev/sd[a-h], which flushes the cache of all RAID disks.

Speed Test #1: RAID vs. LVM

LVM	root@Adam:~/mdadm-3.0# dd if=/dev/mapper/arrays-storage of=/dev/null bs=10M count=400 2097152000 bytes (2.1 GB) copied, 41.1147 s, 43.0 MB/s
	root@Adam:~/mdadm-3.0# hdparm -tT /dev/mapper/arrays-storage  Timing cached reads:   1926 MB in  2.00 seconds = 962.65 MB/sec  Timing buffered disk reads:  146 MB in  3.00 seconds =  48.62 MB/sec
RAID	root@Adam:~/mdadm-3.0# dd if=/dev/md0 of=/dev/null bs=10M count=400 2097152000 bytes (2.1 GB) copied, 10.9341 s, 125 MB/s
	root@Adam:~/mdadm-3.0# hdparm -tT /dev/md0 Timing cached reads:   1998 MB in  2.00 seconds = 998.73 MB/sec Timing buffered disk reads:  538 MB in  3.01 seconds = 178.98 MB/sec

The above numbers are the average of 3 runs.

Speed Test #2: Disks Separately

root@Adam:~# for i in {a,b,c,d,e,f,g,h}; do dd if=/dev/sd"$i"1 of=/dev/null bs=10M count=400; done
root@Adam:~# for i in {a,b,c,d,e,f,g,h}; do hdparm -I /dev/sd"$i" | grep Firmware; done

Disk	Model	Firmware	Speed Test Result
sda	WDC WD10EACS-00D6B1	01.01A01	46.3106 s, 90.6 MB/s
sdb	WDC WD10EACS-00D6B1	01.01A01	48.6391 s, 86.2 MB/s
sdc	WDC WD10EACS-00ZJB0	01.01B01	70.8184 s, 59.2 MB/s
sdd	WDC WD10EADS-65L5B1	01.01A01	46.9733 s, 89.3 MB/s
sde	WDC WD10EADS-65L5B1	01.01A01	44.2861 s, 94.7 MB/s
sdf	MAXTOR STM31000340AS	MX15	77.1797 s, 54.3 MB/s
sdg	WDC WD10EACS-00ZJB0	01.01B01	50.5498 s, 83.0 MB/s
sdh	WDC WD10EADS-00L5B1	01.01A01	46.747 s, 89.7 MB/s

As you can see, though sdc & sdg have the same model and firmware, their speed differs! I have no clue why and I searched in Western Digital's website for firmwares to download, but their site leads no where to any firmware download link.

The Maxtor disk has a newer firmware released. I'll checkout its changelog before installing it. Also, as a precaution, I'll clone the Maxtor disk to sdg since it's not being used now; just in case the new firmware doesn't play nice!

Conclusion

From the above numbers, it's clear that LVM, in my setup, has crippled the performance by a huge margin (~66%). So for my next setup, I'm going to skip LVM and slap the filesystem directly on top of the RAID5 array.

On one of my PCs (Adrenalin), I already have XFS filesystem running on top of the RAID array and LVM is not being used. I get double the speed of hard disks out of the array (140 MB/s) when tested it last year with hdparm.

I don't claim that this is a typical problem of LVM. I did a quick search and didn't find numbers. I'm too lazy right now to find anything really. But I have the numbers on that MSI crap board (caused me so many problems with the SATA ports), and I'll skip LVM on that board. If I keep the board & not smash it to smithereens.

Irrelevant note: I'm loving posting to my blog through Google Docs.

Data Recovery: Hard Disk Surgery

A friend of mine brought a disk that wouldn't work when connected to a machine.

As soon as I hooked the disk to my box, it restarted and refused to start, with all lights blinking madly. Seems like a short-circuit caused that to heappen, so I replaced his case and put the disk in my case and this time: No short-circuit.

The OS booted, but running "lsusb" command showed nothing on the USB bus. When I replaced the case, the pins of his disk interface were a bit bent and I had to straighten them to be able to hook my case's PATA-to-USB (PATA = Parallel ATA) interface.

So, I thought, maybe the controller board itself, which has the PATA interface is damaged somewhere. Lucky for him, I had a disk of the same brand & model number, so I took mine out and put it on his disk. After doing that, I hooked it to the USB port on my machine, and still nothing on the USB bus.

After all easy & safe attempts were futile, the next step was to open the disk's case and look inside at what's really going on. During the previous attempts, I could hear a tiny sound of an attempt to read then failure, so hopefully now I'd be able to inspect further more.

After unscrewing all these tiny pesky screws with proper tools, it turned out that the disk's mechanical head was jammed. I tried nudging it and moving it with the screwdriver, but it seems it was stuck for good. This means, the whole disk housing is useless now, and the only way to get the data is by moving the disks from one hard disk to another. This was much tougher than I had anticipated and took me around 2 hours to carefully move the parts.

The disk heads rest at the edge of the disk, barely touching it, but they don't allow you to take the disks out. In order to do that, there's a white plastic piece, which has grooves where disks lay in, that should be removed first, in order to properly remove the disks. The heads, also partially rest on this plastic piece, and to take the piece out without damaging the heads, the heads must be moved to the inner-side of the disks -- This is by no means an easy task!!!

Pushing the heads by finger also got them bent. To be exact: The heads have small buds that hang by a very tiny piece of wire. These buds are the parts that touch the disks and read/write the data; the heads only work on moving these buds around the disks while the disks spin. I pushed one of the buds a bit to the top, to be able to push the heads safely to the inner part of the disk -- This was a major mistake. Pushing the bud inside rendered it dead and MY disk was no longer readable!!!

So, the whole salvation operation failed and resulted in 2 deaths. The only thing gained from this was experience on how to deal with these in the future, and the fact that I need a magnifier to be able to deal with the heads.

I should mention that I got my screwdrivers from: Al-Nisif warehouse, in Canada-Dry street, Shuwaikh. I'll get their exact info later.

Below are pictures I took while performing the operation.