Software RAID vs. LVM: Quick Speed Test

Table of Contents

1. Introduction
2. System Setup
3. Speed Test Methods
   1. hdparm
   2. dd
4. Speed Test #1: RAID vs. LVM
5. Speed Test #2: Disks Separately
6. Conclusion

Introduction

Currently, I have a fileserver that is setup this way:

Filesystem
      ^
Logical Volume Manager
      ^
Software RAID Arrays
      ^
Physical Disks

In my case, the LVM is an extra layer and it's not useful since I only have one physical entity that belongs to a Volume Group: A single RAID5 array.
So you could put your filesystem on top of a Logical Volume, or directly on the RAID array device. It depends on how you want to manage your data and devices.

So, is this hampering performance? The tables below will do the talking, but first: the setup.

System Setup

Processor	Intel Pentium Dual CPU E2160 @ 1.80GHz
Motherboard	MSI (MS-7514) P43 Neo3-F
	North Bridge: Intel P43
	South Bridge: Intel ICH10
SATA Controller 1	JMicron 20360/20363 AHCI Controller
	AHCI Mode: Enabled
	Ports: 6-7
Sata Controller 2	82801JI (ICH10 Family) SATA AHCI Controller
	Ports: 0-5
RAM	1GB @ CL 5
Video Card	GeForce 7300 GS
Disk sda	WDC WD10EACS-00D6B1
Disk sdb	WDC WD10EACS-00D6B1
Disk sdc	WDC WD10EACS-00ZJB0
Disk sdd	WDC WD10EADS-65L5B1
Disk sde	WDC WD10EADS-65L5B1
Disk sdf	MAXTOR STM31000340AS
Disk sdg	WDC WD10EACS-00ZJB0
Disk sdh	WDC WD10EADS-00L5B1
Disk sdi	Hitachi HDS721680PLAT80 (OS)
Chunk size	256kB
LVM: Physical Extent Size	1GB
LVM: Read ahead sectors	Auto (set to 256)

Speed Test Methods

A quick and easy way to run a speed test is by using a tool called hdparm and another called dd.
Note that these two utilities don't take the filesystem performance into account, as they read directly from the device, not a certain file. It doesn't matter in this case, as I'm about to show comparisons to show the magnitude of difference speed only, not show very exact results ;)

hdparm

hdparm -tT /dev/xxx
-t: Perform timings of device reads for benchmark and comparison purposes.
Displays  the  speed of reading through the buffer cache to the disk without any prior caching of data.
This measurement is an indication of how fast the drive can sustain sequential data reads under Linux, without any filesystem overhead.

-T: Perform timings of cache reads for benchmark and comparison purposes.
This displays the speed of reading directly from the Linux buffer cache without disk access.
This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.

dd

dd if=/dev/xxx of=/dev/null bs=10M count=400
This will read from the device and dump the data to a null device (just reading). Block size=10 Megabytes (2^20).
This will read 4GB of data. I specified 4GB to make sure that it surpasses the RAM size.

Before running dd, I flushed the read cache by entering: hdparm -f /dev/sd[a-h], which flushes the cache of all RAID disks.

Speed Test #1: RAID vs. LVM

LVM	root@Adam:~/mdadm-3.0# dd if=/dev/mapper/arrays-storage of=/dev/null bs=10M count=400 2097152000 bytes (2.1 GB) copied, 41.1147 s, 43.0 MB/s
	root@Adam:~/mdadm-3.0# hdparm -tT /dev/mapper/arrays-storage  Timing cached reads:   1926 MB in  2.00 seconds = 962.65 MB/sec  Timing buffered disk reads:  146 MB in  3.00 seconds =  48.62 MB/sec
RAID	root@Adam:~/mdadm-3.0# dd if=/dev/md0 of=/dev/null bs=10M count=400 2097152000 bytes (2.1 GB) copied, 10.9341 s, 125 MB/s
	root@Adam:~/mdadm-3.0# hdparm -tT /dev/md0 Timing cached reads:   1998 MB in  2.00 seconds = 998.73 MB/sec Timing buffered disk reads:  538 MB in  3.01 seconds = 178.98 MB/sec

The above numbers are the average of 3 runs.

Speed Test #2: Disks Separately

root@Adam:~# for i in {a,b,c,d,e,f,g,h}; do dd if=/dev/sd"$i"1 of=/dev/null bs=10M count=400; done
root@Adam:~# for i in {a,b,c,d,e,f,g,h}; do hdparm -I /dev/sd"$i" | grep Firmware; done

Disk	Model	Firmware	Speed Test Result
sda	WDC WD10EACS-00D6B1	01.01A01	46.3106 s, 90.6 MB/s
sdb	WDC WD10EACS-00D6B1	01.01A01	48.6391 s, 86.2 MB/s
sdc	WDC WD10EACS-00ZJB0	01.01B01	70.8184 s, 59.2 MB/s
sdd	WDC WD10EADS-65L5B1	01.01A01	46.9733 s, 89.3 MB/s
sde	WDC WD10EADS-65L5B1	01.01A01	44.2861 s, 94.7 MB/s
sdf	MAXTOR STM31000340AS	MX15	77.1797 s, 54.3 MB/s
sdg	WDC WD10EACS-00ZJB0	01.01B01	50.5498 s, 83.0 MB/s
sdh	WDC WD10EADS-00L5B1	01.01A01	46.747 s, 89.7 MB/s

As you can see, though sdc & sdg have the same model and firmware, their speed differs! I have no clue why and I searched in Western Digital's website for firmwares to download, but their site leads no where to any firmware download link.

The Maxtor disk has a newer firmware released. I'll checkout its changelog before installing it. Also, as a precaution, I'll clone the Maxtor disk to sdg since it's not being used now; just in case the new firmware doesn't play nice!

Conclusion

From the above numbers, it's clear that LVM, in my setup, has crippled the performance by a huge margin (~66%). So for my next setup, I'm going to skip LVM and slap the filesystem directly on top of the RAID5 array.

On one of my PCs (Adrenalin), I already have XFS filesystem running on top of the RAID array and LVM is not being used. I get double the speed of hard disks out of the array (140 MB/s) when tested it last year with hdparm.

I don't claim that this is a typical problem of LVM. I did a quick search and didn't find numbers. I'm too lazy right now to find anything really. But I have the numbers on that MSI crap board (caused me so many problems with the SATA ports), and I'll skip LVM on that board. If I keep the board & not smash it to smithereens.

Irrelevant note: I'm loving posting to my blog through Google Docs.

DB2 Container Rebalancing: Choosing the right filesystems

Our ERP software's database spans over multiple filesystems for better load balancing, but mixing high-load and low-load database tables on the same filesystem. Also, filesystem are grouped in volume groups that reside on different physical disks.

To better understand this, here's how our production SAN is devised:

Array 2 is one of two database data arrays, and is created over 6 hard disks in RAID level 5.
Array 2 contains two Logical Drives: SAPappShared and SAPdataDrv2. These will be mapped to the operating system (OS) as physical disks.

 

Here you can see that the 3rd array spans over different disks. This is to optimize data load times when requesting data from the database.
The two red/white disks are hot-spares that will substitute any failed disk immediately.
You can also see that this array reside on a different controller (B), while the previous is on (A). The controller handles I/O requests from the OS.

 

Database logs are written to a RAID level 1 array spanned over 2 disks. Logs are written always so it's better to keep them away from data to not hamper the performance.






From what you saw above, you can count 6 Logical Drives that relate to the database. The database uses only 4 of them: Log, Data1-1, Data1-2 and Data2. The Heartbeat drives are used by the cluster services of the OS.



Now, moving to the OS: IBM's AIX. We run version 5.3L.

On AIX, you cannot create a filesystem on a physical disk directly. The physical disk has to belong to a Volume Group. A volume group can house multiple physical disks, whether internal disks or attached through storage or network. A volume group can contain multiple filesystems. This all falls under the topic of the Logical Voume Manager, which I will not talk about here.

Back to our setup above, from AIX's point of view, the Logical Drives are shown as physical disks:

# lspv
hdisk0          00c3b3f0feb38826                    rootvg          active
hdisk1          00c3b3f0d832abbc                    rootvg          active
hdisk2          00c3b3e0fbad39e6                    hrtvg1
hdisk3          00c3b3e0fbb076c8                    saplogsvg       active
hdisk4          00c3b3e0fbef90aa                    sapdatavg1      active
hdisk5          00c3b3e0fbf2e9bf                    sapdatavg2      active
hdisk6          00c3b3f0168bedbb                    sapdatavg3      active

You can see from the list above that each Logical Drive is a physical disk and moreover, each has been put to a separate volume group.

These are filesystems spanned over the volume group on hdisk4:

# lspv -l hdisk4
hdisk4:
LV NAME               LPs   PPs   DISTRIBUTION          MOUNT POINT
db2lv                 2     2     00..02..00..00..00    /db2/EHP/db2ehp
db2dumplv             24    24    00..01..00..00..23    /db2/EHP/db2dump
sapdata6lv            280   280   123..00..00..135..22  /db2/EHP/sapdata6
sapdata3lv            340   340   13..132..135..00..60  /db2/EHP/sapdata3
loglv2                1     1     00..01..00..00..00    N/A

Before continuing, let's talk about database usage, to be able to justify the next moves.

Since the database tables have been split over multiple filesystems, some of the filesystems have high growth rate. Our database (IBM's DB2) can span the contents of a table over multiple filesystems; a load balancing technique. It does this through its concept of rebalancing containers. (In DB2, tables reside in Tablespaces and a tablespace consists of one or more containers).

So, if we create a second container of a high growth table, it will take half of it and dump it over the new container and in the future, it will write to both in a load balancing form.

Now, to be able to achieve a true sense of load balancing, in filesystem utilization and database performance, ideally these two filesystems (that house the high growth table) should be on different controllers, so that data requested residing on both filesystems can be served in parallel by the controllers.

If you're doing the rebalancing after going live with the system (which is the case mostly), then you'd need to know which physical disk on the OS goes to which controller.

Here, you see that each Logical Drive has a unique identifier called Logical Unit Number (LUN).

The LUN is represented as a hex value on AIX.





From the shell, type "lspv" to list the physical volumes, then "lsattr -El " to list the attributes of one of the disks. The value "lun_id" is what you want. See the output below.

# lspv
hdisk0          00c3b3f0feb38826                    rootvg          active
hdisk1          00c3b3f0d832abbc                    rootvg          active
hdisk2          00c3b3e0fbad39e6                    hrtvg1
hdisk3          00c3b3e0fbb076c8                    saplogsvg       active
hdisk4          00c3b3e0fbef90aa                    sapdatavg1      active
hdisk5          00c3b3e0fbf2e9bf                    sapdatavg2      active
hdisk6          00c3b3f0168bedbb                    sapdatavg3      active

# lsattr -El hdisk4
PR_key_value   none                             Persistant Reserve Key Value           True
cache_method   fast_write                       Write Caching method                   False
ieee_volname   600A0B80003260360000A92E472997CA IEEE Unique volume name                False
lun_id         0x0005000000000000               Logical Unit Number                    False
max_transfer   0x100000                         Maximum TRANSFER Size                  True
prefetch_mult  1                                Multiple of blocks to prefetch on read False
pvid           00c3b3e0fbef90aa0000000000000000 Physical volume identifier             False
q_type         simple                           Queuing Type                           False
queue_depth    10                               Queue Depth                            True
raid_level     5                                RAID Level                             False
reassign_to    120                              Reassign Timeout value                 True
reserve_policy single_path                      Reserve Policy                         True
rw_timeout     30                               Read/Write Timeout value               True
scsi_id        0x10400                          SCSI ID                                False
size           173670                           Size in Mbytes                         False
write_cache    yes                              Write Caching enabled                  False

# lsattr -El hdisk6
PR_key_value   none                             Persistant Reserve Key Value           True
cache_method   fast_write                       Write Caching method                   False
ieee_volname   600A0B80003260360000A9AA472EA49C IEEE Unique volume name                False
lun_id         0x000c000000000000               Logical Unit Number                    False
max_transfer   0x100000                         Maximum TRANSFER Size                  True
prefetch_mult  1                                Multiple of blocks to prefetch on read False
pvid           00c3b3f0168bedbb0000000000000000 Physical volume identifier             False
q_type         simple                           Queuing Type                           False
queue_depth    10                               Queue Depth                            True
raid_level     5                                RAID Level                             False
reassign_to    120                              Reassign Timeout value                 True
reserve_policy single_path                      Reserve Policy                         True
rw_timeout     30                               Read/Write Timeout value               True
scsi_id        0x10500                          SCSI ID                                False
size           294225                           Size in Mbytes                         False
write_cache    yes                              Write Caching enabled                  False

Using the above commands, finding which disk belongs to which controller on a SAN on AIX becomes like baby steps ;)

We did a rebalance on one of the containers today. The container's size was about 50GB and it took about an hour on an IBM DS4800 SAN hooked with fiber to the database node residing on an IBM pSeries (570) machine: 4 POWER5 processors allocated and 30GB of RAM.

I tried looking up time estimation for rebalancing containers on DB2 on Google, but didn't find anything useful. Luckily, things went on smoothly.
The error file "db2diag.log" had the following with regards to rebalancing:

2009-05-18-01.05.59.552039+180 E106271503A342     LEVEL: Warning
PID     : 799110               TID  : 1           PROC : db2rebal 0
INSTANCE: db2ehp               NODE : 000
FUNCTION: DB2 UDB, buffer pool services, sqlb_rebalance, probe:2876
MESSAGE : ADM6062I  Rebalance for table space "EHP#BTABD" (ID "17") has been
 completed.

2009-05-18-01.05.59.552281+180 I106271846A295     LEVEL: Warning
PID     : 799110               TID  : 1           PROC : db2rebal 0
INSTANCE: db2ehp               NODE : 000
FUNCTION: DB2 UDB, buffer pool services, sqlb_rebalance, probe:2876
MESSAGE : PoolID 17: Last extent moved was #1866949

During the rebalancing period, the container being worked on was not showing in the list on the ERP software (SAP) and threw some errors. It was working fine on the database level however. After the process completed, SAP reported the new containers properly.