Tuesday, September 8, 2009

Software RAID vs. LVM: Quick Speed Test

Introduction
System Setup
Speed Test Methods
1. hdparm
2. dd
Speed Test #1: RAID vs. LVM
Speed Test #2: Disks Separately
Conclusion

Introduction

Currently, I have a fileserver that is setup this way:

Filesystem
      ^
Logical Volume Manager
      ^
Software RAID Arrays
      ^
Physical Disks

In my case, the LVM is an extra layer and it's not useful since I only have one physical entity that belongs to a Volume Group: A single RAID5 array.
So you could put your filesystem on top of a Logical Volume, or directly on the RAID array device. It depends on how you want to manage your data and devices.

So, is this hampering performance? The tables below will do the talking, but first: the setup.

System Setup

Processor	Intel Pentium Dual CPU E2160 @ 1.80GHz
Motherboard	MSI (MS-7514) P43 Neo3-F
	North Bridge: Intel P43
	South Bridge: Intel ICH10
SATA Controller 1	JMicron 20360/20363 AHCI Controller
	AHCI Mode: Enabled
	Ports: 6-7
Sata Controller 2	82801JI (ICH10 Family) SATA AHCI Controller
	Ports: 0-5
RAM	1GB @ CL 5
Video Card	GeForce 7300 GS
Disk sda	WDC WD10EACS-00D6B1
Disk sdb	WDC WD10EACS-00D6B1
Disk sdc	WDC WD10EACS-00ZJB0
Disk sdd	WDC WD10EADS-65L5B1
Disk sde	WDC WD10EADS-65L5B1
Disk sdf	MAXTOR STM31000340AS
Disk sdg	WDC WD10EACS-00ZJB0
Disk sdh	WDC WD10EADS-00L5B1
Disk sdi	Hitachi HDS721680PLAT80 (OS)
Chunk size	256kB
LVM: Physical Extent Size	1GB
LVM: Read ahead sectors	Auto (set to 256)

Speed Test Methods

A quick and easy way to run a speed test is by using a tool called hdparm and another called dd.
Note that these two utilities don't take the filesystem performance into account, as they read directly from the device, not a certain file. It doesn't matter in this case, as I'm about to show comparisons to show the magnitude of difference speed only, not show very exact results ;)

hdparm

hdparm -tT /dev/xxx
-t: Perform timings of device reads for benchmark and comparison purposes.
Displays the speed of reading through the buffer cache to the disk without any prior caching of data.
This measurement is an indication of how fast the drive can sustain sequential data reads under Linux, without any filesystem overhead.

-T: Perform timings of cache reads for benchmark and comparison purposes.
This displays the speed of reading directly from the Linux buffer cache without disk access.
This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.

dd

dd if=/dev/xxx of=/dev/null bs=10M count=400
This will read from the device and dump the data to a null device (just reading). Block size=10 Megabytes (2^20).
This will read 4GB of data. I specified 4GB to make sure that it surpasses the RAM size.

Before running dd, I flushed the read cache by entering: hdparm -f /dev/sd[a-h], which flushes the cache of all RAID disks.

Speed Test #1: RAID vs. LVM

LVM	root@Adam:~/mdadm-3.0# dd if=/dev/mapper/arrays-storage of=/dev/null bs=10M count=400 2097152000 bytes (2.1 GB) copied, 41.1147 s, 43.0 MB/s
	root@Adam:~/mdadm-3.0# hdparm -tT /dev/mapper/arrays-storage Timing cached reads: 1926 MB in 2.00 seconds = 962.65 MB/sec Timing buffered disk reads: 146 MB in 3.00 seconds = 48.62 MB/sec

RAID	root@Adam:~/mdadm-3.0# dd if=/dev/md0 of=/dev/null bs=10M count=400 2097152000 bytes (2.1 GB) copied, 10.9341 s, 125 MB/s
	root@Adam:~/mdadm-3.0# hdparm -tT /dev/md0 Timing cached reads: 1998 MB in 2.00 seconds = 998.73 MB/sec Timing buffered disk reads: 538 MB in 3.01 seconds = 178.98 MB/sec

The above numbers are the average of 3 runs.

Speed Test #2: Disks Separately

root@Adam:~# for i in {a,b,c,d,e,f,g,h}; do dd if=/dev/sd"$i"1 of=/dev/null bs=10M count=400; done
root@Adam:~# for i in {a,b,c,d,e,f,g,h}; do hdparm -I /dev/sd"$i" | grep Firmware; done

Disk	Model	Firmware	Speed Test Result
sda	WDC WD10EACS-00D6B1	01.01A01	46.3106 s, 90.6 MB/s
sdb	WDC WD10EACS-00D6B1	01.01A01	48.6391 s, 86.2 MB/s
sdc	WDC WD10EACS-00ZJB0	01.01B01	70.8184 s, 59.2 MB/s
sdd	WDC WD10EADS-65L5B1	01.01A01	46.9733 s, 89.3 MB/s
sde	WDC WD10EADS-65L5B1	01.01A01	44.2861 s, 94.7 MB/s
sdf	MAXTOR STM31000340AS	MX15	77.1797 s, 54.3 MB/s
sdg	WDC WD10EACS-00ZJB0	01.01B01	50.5498 s, 83.0 MB/s
sdh	WDC WD10EADS-00L5B1	01.01A01	46.747 s, 89.7 MB/s

As you can see, though sdc & sdg have the same model and firmware, their speed differs! I have no clue why and I searched in Western Digital's website for firmwares to download, but their site leads no where to any firmware download link.

The Maxtor disk has a newer firmware released. I'll checkout its changelog before installing it. Also, as a precaution, I'll clone the Maxtor disk to sdg since it's not being used now; just in case the new firmware doesn't play nice!

Conclusion

From the above numbers, it's clear that LVM, in my setup, has crippled the performance by a huge margin (~66%). So for my next setup, I'm going to skip LVM and slap the filesystem directly on top of the RAID5 array.

On one of my PCs (Adrenalin), I already have XFS filesystem running on top of the RAID array and LVM is not being used. I get double the speed of hard disks out of the array (140 MB/s) when tested it last year with hdparm.

I don't claim that this is a typical problem of LVM. I did a quick search and didn't find numbers. I'm too lazy right now to find anything really. But I have the numbers on that MSI crap board (caused me so many problems with the SATA ports), and I'll skip LVM on that board. If I keep the board & not smash it to smithereens.

Irrelevant note: I'm loving posting to my blog through Google Docs.

Tuesday, September 1, 2009

Cheap Man's 40-Disk Storage Cluster

Table of Contents

Introduction

This is a computer case design that fits 40 disks, 4 motherboards, 5 power supplies, a bunch of fans and a gigabit switch.

The main goal of this design is to use the cheapest parts with the least effort to assemble everything. So you could say this is also Poor/Lazy Man's Storage Cluster!
The area used is 60x60x50 (WxDxH) cm only.

Keep in mind that this is a case not a whole system. I've only factored the price of the pieces used to put the case together.

Parts and Prices

Metal table with 2 net-like surfaces	7 KD	IKEA
2x Wooden CD rack that fits 35 CDs	2x3 KD	IKEA
Plastic drawer mat	1.75 KD	IKEA
Rubber grooved floor mat (3mm thick)	1.5 KD for 0.5 meter	TrueValue
Nylon Cable Ties (203x3.2mm)	0.5 KD	Family Hardware Store
Total	16.75 KD

Tools

Hands
Foot
Long nose pliers
Scissors
Hammer (or anything that hammers)

Design Diagram

In this diagram you see the measurements of each component and how they fit. When we put the disks inside the rack, there was an empty space of 4mm; we took care of that by using the rubber mat, which is 3mm thick on each side, totaling to 6mm, which helps holding the disks and serve as a shock absorber.

The disk rack is made of wood and we have rubber mats inside, so you'd expect for heat to be trapped. Our work around is this:

Choosing a rubber mat with grooves
Inserting the disks heads down having the 2.5cm edges touching the rubber
Leaving space between disks
Pushing the disks down till touching the metal table
Installing fans on the lower part of the table, blowing at the disks

Note that I have used Layout #1. Layout #2 was too cluttered and I didn't really think it through properly so I don't know if it's even possible. If you are able to squeeze more than 4 motherboards in that same table (or same dimensions), let me know!

Assembly

Assemble lower shelf of table
Use foot to break back-panel of CD racks
Hammer the metal pins of the CD racks inwards
Point the side that has the metal pins towards table surface (keeps wood fragments away from you)
Tie the rack to the table using the cable ties

Note: We assembled the upper surface but worked on the lower one later and kept the upper free for future motherboards.
Cut 11 lines (in a group) of the rubber mat
Attach the rubber mat to a side and tie it down. Do the same for the other side

We have cut out the extra edges shown in the first picture to reduce heat contraption.
If you want to have a separate power supply unit (PSU) for fans, attach it to the bottom of the lower surface

Note: I'll tell you later how to turn on the PSU without a motherboard (jump-starting).If you're going to run anything at the bottom, now is the time to attach them. You won't be able to do it later on!
Put the power supplies in place and tie them (make sure it's touching the metal table)
Cut plastic mat to fit rest of table area and tie it to the table. Use the nail to punch holes
Punch extra holes for the motherboards and don't tie the motherboards too hard!
Tie fans below the disk rack(s) and connect them in serial to a PSU

Note: We made a mistake above and tied the fans in the opposite direction of the motherboard and were too lazy to reposition them.
Slap in the hard disk drives (HDDs) and hook them to the motherboard(s)
Powering on
To power on a motherboard, you could either use a power switch (or make one), or enable Wake-up On LAN (WOL) from the BIOS (assuming your motherboard supports it. You'd need to know the MAC address of your LAN port.

We enabled WOL but it didn't work for some reason. Crappy MSI board.
If your motherboard doesn't have a built-in video/graphics card, you'll need to bend the tip of the graphics card you're about to attach
Almost done. Attach the upper surface and make sure that the rack is facing the opposite side to the one on the lower surface, so that the fans don't hit the cables

Make sure to double check on the HDD cables after attaching the upper surface.
Jump-starting a PSU
I stripped a cable wrapper that had a metal piece inside it and stuck it in the proper pins. This way, I control the fan PSU using the ON/OFF switch at the back; no need for a separate power switch. Unfortunately we didn't take pictures of that, but here are some references:

To Cluster Or Not To Cluster

Now that you've had your motherboards all hooked up most likely to a gigabit switch, there are different ways to use all this storage capacity:

Make them appear as a single storage unit
This can be done through the use of iSCSI. It allows you to expose either each single hard disk or a whole RAID array as a single storage device to another machine over Ethernet. This way you can combine all the disks/arrays under one machine and create a LV (Logical Volume not Louis Vuitton) then the filesystem on top of the LV.
I don't know how to do this on Windows. I can help you do this on Linux though. If you did this on Windows, drop me an email and I'll link your page.
Use them separately
Well this is a no-brainer: Just assign a different IP for each machine and expose each storage through Samba (on Linux) or share the directories on Windows.
A mix between the above two
Using iSCSI puts a lot of risk on the data because if one motherboard, or multiple disks fail (in case of using RAID5) then you lose all your data. For good. And since we have such a good history of increasing and managing the storage smoothly (NOT!), we decided to not use iSCSI. Maybe if we had a better history, we'd gone with it.

What we're going to do is keep the existing Samba share and move the Anime directory (2.4TB) to another machine. We then mount the other machine using NFS over the existing Anime directory. Mounting a directory over another is called shadowing.
For this to work properly, you need to create the usernames on all systems with the same IDs, otherwise you'll have a heck of a time with permissions.

Now, the users still access the same old single IP and can still access all data, though distributed cross systems. If a machine's disks died, at least we won't lose all the data.

We don't yet have the 2nd motherboard, so I'll write about this in detail when we get it and do the setup.

Post Assembly

After assembling and running the machine, it's been put under heavy load and these are the temperature readings:
/dev/sda: WDC WD10EACS-00D6B1: 27°C
/dev/sdb: WDC WD10EACS-00D6B1: 29°C
/dev/sdc: WDC WD10EACS-00ZJB0: 29°C
/dev/sdd: WDC WD10EADS-65L5B1: 28°C
/dev/sde: WDC WD10EADS-65L5B1: 29°C
/dev/sdf: MAXTOR STM31000340AS: 29°C
/dev/sdg: WDC WD10EACS-00ZJB0: 27°C
/dev/sdh: WDC WD10EADS-00L5B1: 26°C

This is way much better than before! They used to be in the 40's range!!

Last Words

Our baby is running fine now, and for the first time we haven't faced problems, thank God!

If you have any questions or comments, let us know. I suggest you subscribe via email when commenting, or leave a blank comment to just subscribe to stay posted on updates when adding the 2nd motherboard.

Good luck and don't blame us if you get electrocuted ^_^'

Saturday, August 22, 2009

HOWTO: Asterisk and FreePBX using PBX in a Flash

Introduction
0) Requirements
1) Download the ISO file
2) Preparing the PC
3) Installation
  3.1) Phase 1: From CD
  3.2) Phase 2: From the Internet
4) Networking
5) Install latest patches and scripts
6) Passwords and System Administration
7) Prevent Being Blocked
8) Configuring Your PCI Card(s)
  8.1) Configure PCI cards (Asterisk 1.4 + Zapata)
  8.2) Configure PCI cards (Asterisk 1.6 + Dahdi)
9) Caller ID and Busy Detect Configuration
10) Updating Asterisk and FreePBX Modules
11) Tips and Tricks
12) Resources and References

Introduction

This is a guide (HOWTO) on how to install PBX in a Flash (PiaF) and do the initial hardware and software configurations required so that you could start doing your dialplans through the web-interface later on.

This HOWTO will not include steps on configuring extensions, trunks and the like. That will be in another post.

PiaF is a Linux distribution which makes installing and configuring Asterisk and FreePBX an easy task. It's similar to trixbox, only it has no history of security risks and trojans!

It's to be noted that PiaF downloads and compiles from source code. If you have no idea what that means, you'll find out later.

This is a dirty version of the guide. I couldn't get the styling to work properly.
Better versions: HTML Page or Google Document (PDF).

0) Requirements

PC: 800 MHz, 256MB RAM, 10GB disk space, CD-Drive, 1x free PCI slot, 1x network port.
Digium's TDM PCI card if you wish to make/receive calls from a landline.
Internet connection (needed during installation!!!).
DHCP Server (Preferred): If you have an Access Point that connects you to the Internet, it has the option already.
1x Blank CD.
1x Network cable.
1x Switch.
Patience.

1) Download the ISO file

Get the ISO from: http://pbxinaflash.net/downloads/
If you have 4GB of RAM or more, grab the 64-bit ISO, otherwise stick to the 32-bit one. If you wish to venture into the 64-bit world, your processor has to be 64-bit capable.

Burn the ISO image using your favorite CD writing software. Make sure you choose the option to write an image, not put the ISO file in the CD as data!!!

2) Preparing the PC

Boot up your PC and enter the BIOS.

Make sure the time is correct! If you don't know the time in Kuwait, call 113. If the time is not correct in your computer, the installation will fail!
Somewhere in the menus there should be an option defining boot-sequence: Make sure that the CD-Drive is before the hard disk.

3) Installation

As mentioned before, you will need an Internet connection to proceed with the installation after installing the base packages from the CD. Also, you need a running DHCP server, so that your PBX box can obtain an IP during the installation process. If you have a router at home, chances are it has DHCP running already. It doesn't hurt to double check.

3.1) Phase 1: From CD

Boot the CD and follow the instructions:

To install Asterisk 1.4, press Enter. To install Asterisk 1.4 without Logical Volume Management, type ksalt. I chose ksalt.
Keyboard type: us
Time Zone Selection:

System clock uses UTC: No (Deselect)
Asia/Kuwait

Password: This is your system's password. Make sure it's alpha-numeric and at least 10 digits. The system's administrator username is: root. (Typical on *nix Operating Systems)

The installation process now continues for a while to install required packages from the CD. This would take about 10 minutes.

When the installation is over and the PC reboots, remove the CD. The hard disk will boot.

3.2) Phase 2: From the Internet

[if you know how to run a web-server on a normal PC, jump to the tricks section]

When the PC boots from the hard disk, a screen will come up asking you what to do. You should press A, to install the latest stable release of Asterisk 1.4.

Press A; download commences (30MB): http://pbxinaflash.net/ast14/pbx_load.tar.gz
(Asterisk 1.6: http://pbxinaflash.net/ast16/pbx_load16.tar.gz)
MD5 checksum: ecaa6b1bc5c9bf6bca3b086d22e7e4a5 pbx_load.tar.gz
MD5 checksum: 46d7d6bb60fd2973fe37bd50b10ae7e1 pbx_load16.tar.gz
Installation of the package proceeds automatically. Just sit back and read the messages on the screen. Or not.
DO NOT PRESS THE ENTER BUTTON! If the screen turns off when it idles, press an arrow key (right, left, up, down)

Do not remove the network cable. The installation may require some extra packages during the installation.

[i] It takes a long time because the file that was downloaded is the source code, and after downloading it, the installer script unpacks it then starts compiling the source to produce the binaries. Binaries are executable programs. The compilation process usually takes about 20 minutes on a Dual core CPU @ 3GHz.
The PC will reboot after the compilation is done.
After the PC has booted, you get to a login prompt, login with user: root, and the password you chose during installation.

4) Networking

You could leave your machine on DHCP, but you have to configure your DHCP server or Access Point to always assign the same IP to this machine.

I recommend that you could configure a static IP address in the machine itself by following these steps:

Type: netconfig
Would you like to set up networking? Yes
Deselect “Use dynamic IP” and fill in your network information.
Default gateway and primary name server are usually your Access Point's IP address.
Click OK and exit the menu.
Type: service network restart.
echo "nameserver xxx.xxx.xxx.xxx" > /etc/resolv.conf (replace the Xs with your IP)

In my case, even when setting the name server in the netconfig menu, it didn't get written until I do it manually or run netconfig again.

That's it. Now the IP is static to whatever you chose up there.

Now you should be able to reach the web interface on: http://<ip> -- but don't. We need to patch the files then configure some security options and passwords!

5) Install latest patches and scripts

You must not skip this step. It's essential to get things working!

In the shell, type: update-scripts
This will bring up some boring text about the code and what's being installed.
Press Enter and you see a list of more commands to use.
Press Enter again and then press Y. This was just an agreement page and you're back to the shell.
Now run update-scripts again and press Y, to download the required stuff.
This will download the files:
http://www.pbxinaflash.org/ast14/files/securescripts.tar.gz
http://www.pbxinaflash-scripts.com/ast14/files/pbx-scripts.tar.gz
http://www.pbxinaflash-scripts.com/ast14/files/apache-auth.conf.gz

97e8ccfbfd27f611e98ced4e8f791f95 securescripts.tar.gz
52c9435c93eaa0dcadb9d83889145b2f pbx-scripts.tar.gz
430207ba3db9eae8c7fde495a1815731 apache-auth.conf.gz
After that, it will tell you to type: update-fixes. Do it.
Press Y to continue with the script. This will apply patches and security enhancements to your setup.

It asks you later to press Enter, then set a password for the ARI interface. Use the same password used for the root user.
It may take some time (even when it says thank you), so wait for it.

If you wish to see the configurations, press Y, otherwise q.

In the future, if you want to update asterisk and related packages, type: update-source.

[i] For Asterisk 1.6: As of this writing, update-source16 doesn't work yet and the script asks to check the forums for a workaround for now.

[i] To see a list of all helpful commands, type: help-pbx.

6) Passwords and System Administration

Before venturing into the web interface, you have to change the default passwords!
Also, the main password script will ask if you would like to install Webmin, an application that allows you to administrate your Linux box through the web interface. I will not go with this option, to reduce the security risks.

Type: passwd-master

7) Prevent Being Blocked

There's a "security" application running on PBXiaF which if you enter the wrong password 3 times, will block your IP and lock you out of the PBX!!

I would suggest you disable it till you get the hang of things. Type: service fail2ban stop

[i] It will be re-enabled the next time you boot.
[i] The command disable-fail2ban may say that you don't have fail2ban installed, but when typing "status" it will show as running. This may happen on an Asterisk 1.6 setup.

If you wish to disable it for good, type: chkconfig fail2ban off

8) Configuring Your PCI Card(s)

Depending on which version you installed, follow the proper section.

8.1) Configure PCI cards (Asterisk 1.4 + Zapata)

In the shell, type:
amportal stop
genzaptelconf -vvvv

This will generate the file /etc/asterisk/zap-channels.conf which is automatically included by /etc/asterisk/zapata.conf

Now type:
amportal start

8.2) Configure PCI cards (Asterisk 1.6 + Dahdi)

root@pbx:/var/log/asterisk $ dahdi_genconf -vvv
Default parameters from /etc/dahdi/genconf_parameters
Generating /etc/dahdi/system.conf
Generating /etc/asterisk/dahdi-channels.conf
root@pbx:/var/log/asterisk $ dahdi_cfg -vsfd5
DAHDI Tools Version - 2.2.0
Line 9: fxsks=1
Line 10: echocanceller=mg2,1
Line 17: loadzone = us
Line 18: defaultzone = us

DAHDI Version: 2.2.0.2
Echo Canceller(s):
Configuration
======================
1 channels to configure.

About to open Master device

9) Caller ID and Busy Detect Configuration

We'll need to add these lines to the file: /etc/asterisk/zapata.conf

cidsignalling=v23 ;caller-id option
cidstart=polarity ;caller-id option
busydetect=yes ;busy tone detction
busycount=5 ;disconnect after 5 busy tones

nano /etc/asterisk/zapata.conf

Add the lines above under "hidecallerid" then to save the file: CTRL+X, Y, Enter.
Now type: amportal restart

10) Updating Asterisk and FreePBX Modules

Login to the web interface by directing your browser to http://<ip>
Slide the white bar at the bottom-left to the right and click on the admin link, then on the picture: "FreePBX Administration"
Username: maint
Password: What you chose with passwd-master

On the left panel, click Module Admin. Then on the right page, click Update. You should update your installed modules to get the latest stable versions.
You could after that install new modules.
The modules I installed were:

Announcements 2.5.1.8
Asterisk CLI 2.5.0.3
Asterisk Info 2.5.0.2
Backup & Restore 2.5.1.7
Blacklist 2.5.0.5
Callback 2.5.0.3
Call Forward 2.5.0.2
Call Waiting 2.5.0.1
Conferences 2.5.1.7
ConfigEdit 1.0
Day Night Mode 2.5.0.12
Do-Not-Disturb (DND) 2.5.0.6
Follow Me 2.5.1.8
FreePBX ARI Framework 2.5.2.3
FreePBX FOP Framework 2.5.0.1
Asterisk Logfiles 2.5.0.1
Misc Applications 2.5.0.4
Misc Destinations 2.5.0.3
Paging and Intercom 2.5.0.8
Print Extensions 2.5.0.5
Queues 2.5.4.8
Ring Groups 2.5.1.9
Sys Info phpsysinfo 2.5.3
Time Conditions 2.5.0.9

11) Tips and Tricks

If you know how to run a web-server on a local machine, you can minimize the time it takes to download the files (especially that their website can be slow at times).

You need to do two things:

Download the files mentioned above and put them in the proper location
Override DNS records for the domains the scripts use

1.In your default web-directory (/var/www) create these directories: mkdir -p /var/www/ast14/files
Look at the links above and put them in the proper directory.

1.On your Asterisk box, type: nano /etc/hosts
Add these entries:

xxx.xxx.xxx.xxx     pbxinaflash.net
xxx.xxx.xxx.xxx     pbxinaflash.org
xxx.xxx.xxx.xxx     pbxinaflash-scripts.com

Replace the Xs with your web-server's IP.
At least now if you mess-up the installation and can save yourself the time it takes to redownload the files!

12) Resources and References

Security
http://nerdvittles.com/?p=580
http://secunia.com/advisories/34772/3/

PBX in a Flash
http://pbxinaflash.org/docs/InstallPBXInAFlash.pdf
http://pbxinaflash.net/docs/
http://knol.google.com/k/ward-mundy/pbx-in-a-flash/3uqc77rg9tgar/2#Getting_Started_with_PBX_in_a_Flash_1%282E%294
http://bestof.nerdvittles.com/scripts/
http://pbxinaflash.net/
http://pbxinaflash.net/downloads/
http://pbxinaflash.org/piafscripts/Asterisk+1.4

Passwords
http://www.freepbx.org/support/documentation/faq/changing-the-asterisk-manager-password
http://pbxinaflash.org/docs/InstallPBXInAFlash.pdf

Munin Reports
http://pbxinaflash.com/forum/showthread.php?t=978
http://munin.projects.linpro.no/wiki/Documentation
http://nerdvittles.kicks-ass.net/munin/

Hardware Config
http://www.trixbox.org/forums/trixbox-forums/open-discussion/2-3-0-10-problem-when-installing-not-automatically-detecting-m

Asterisk Config
http://www.voip-info.org/wiki/view/Asterisk+config+zapata.conf

FreePBX
http://www.freepbx.org/support/documentation/administration-guide

SIP Phone Recommendations
http://nerdvittles.com/index.php?p=207

Dahdi/Asterisk 1.6
http://bbs.openvox.cn/redirect.php?tid=788&goto=lastpost
http://forums.digium.com/viewtopic.php?p=122503&sid=dbc5a4bc9bec829d2e8d40f5d7dcea6e
http://www.mail-archive.com/asterisk-users@lists.digium.com/msg163600.html
http://forums.whirlpool.net.au/forum-replies-archive.cfm/529755.html
http://manpages.ubuntu.com/manpages/karmic/man8/fxotune.8.html