Benchmarking Untangle Gateway

a work in progress


* This benchmark will be re-done to include more details (gateway's memory & CPU utilization)
* There have been a typo which mistakenly puts the filtering device where it won't be. Will be corrected later

We will NOT deploy a filtering appliance afterall. We'll be using Kaspersky Antivirus to enforce Web Filtering policies (due February). We'll deploy a proxy only, however I'll continue with this benchmark.

Untangle is a gateway software package that installs to x86 hardware. It functions like any gateway appliance from any company like Cisco, BlueCoat, ...etc. Only this one is built on top of Open Source projects running on Linux, giving it the power of guaranteed stability and the flexibility to mesh everything into a single platform that does wonders.

I have deployed this on a subsidiary company that has 120 employees. The machine has been running for about 7 months so far without any problems whatsoever. The total cost was 110 KD ($330 roughly) for the hardware: 1.8GHz dual core CPU, 2GB RAM, 80GB HDD, built-in 10/100Mbps NIC + video card + audio, 2x 1Gbit Linksys NICs.

Untangle offers its basic package for free. This contains the following components:

• Web-Filter


• Spam Blocker


• Spyware Blocker


• Phish Blocker


• Virus Blocker


• Layer-7 Filtering / Protocol Control


• Intrusion Prevention


• Firewall


• Attack Blocker


• VPN


• Reporting


• Routing and Quality of Service

The main reason why Untangle was chosen because it has Layer-7 filtering. This allows you to block certain programs from accessing the network, like Yahoo Messenger, MSN Messenger, ICQ, IRC, Jabber, Peer-to-Peer software, online games, ...etc.

Only the components in bold above were activated, because the rest were useless to our company since there existed other appliances that are doing their job.

Now, we're considering deploying this on the whole company for around 400 users. The stake got raised and we're comparing it against BlueCoat. So we'll need some commercial packages from Untangle to integrate it with our Active Directory Domain Controller using their AD Connector component, and a way to apply different rules to different people using Untangle's Policy Manager component.

Prices apart, since Untangle allows us to demo the commercial components for 14 days for free, we have done the integration with the domain controller and setup test policies for various domain users and everything worked perfectly in less than 10 minutes!

Now it's time to benchmark it and see if it can handle the stress.

Benchmark Details

Host Machine

AMD Dual-core 2.4GHz, 4GB RAM, 2x Gbit NICs, 80GB HDD.

Network Setup

EXT/WAN: 1x Gbit NIC is the external link which is connected to our LAN at work. The Internet link is 1Mbps.
INT/LAN: The 2nd NIC is a bridge to the internal network. I'm using a cross cable and hooking it directly to my laptop. (I've also tested it using a D-Link 5-port Gbit switch and got same results).

Caching: The Untangle box was NOT the DNS server (although it caches DNS requests).
As of this writing, Untangle does not offer a web-caching solution, though it's mentioned in their forums that one will emerge with following updates. Our tests didn't involve a web-caching scheme.

 

Deployment Options

Any appliance that we'll eventually go with will be either sitting between the firewall and the Internet router (bridged setup), or connected to the external switch like the firewalls are.

1) Bridged setup: Requires 2 Ethernet ports/interfaces; an internal and an external interfaces.
No need to change the users' settings nor the firewall's gateway. The external port gets a public IP while the internal one is bridged and holds no IP. Whatever traffic is flowing towards the router (the firewalls' gateway) must flow through the appliance.


2) Switch setup: Requires 1 Ethernet port. It doesn't require the users to change any settings, however, it does require the network admin to change the firewall's default gateway to point at the proxy server. Obviously, the proxy's gateway would be the router like in (1).

Software

Siege is an open-source and free HTTP stress, benchmark and load balance testing tool.
Siege was not used directly. The URLs fed to it were harvested using another nifty tool by the same author called sproxy.

Simulation Criteria

From our subsidiary company we got a max of 250 web transactions per minute. This translates to 4 requests per second.
Our test a bit more stressful than these numbers: 100 browser requests per 5 seconds (random). This translates to 20 requests per second.

The links that were used for testing are:
http://www.eicar.org/download/eicar_com.zip
http://www.eicar.org/download/eicar_com.zip
http://www.eicar.org/download/eicarcom2.zip
http://www.eicar.org/download/eicarcom2.zip
http://www.eicar.org/download/eicarcom2.zip
http://www.eicar.org/download/eicarcom2.zip
http://www.eicar.org/download/eicarcom2.zip
http://www.eicar.org/download/eicarcom2.zip
http://www.eicar.org/download/eicarcom2.zip
http://www.eicar.org/download/eicarcom2.zip
http://www.eicar.org/download/eicarcom2.zip
http://www.eicar.org/download/eicarcom2.zip
http://wwww.cnn.com
http://www.gooogle.com
http://www.kooora.com
http://www.sectorx.org
http://www.fox.com
http://www.joecartoon.com
http://www.killfrog.com
http://www.yahoo.com
http://www.nbk.com
http://www.cakephp.org
http://www.tucows.com
http://www.download.com
http://www.youtube.com
http://www.islamway.com
http://www.jazeeraairways.com

The links are a mix to test the virus blocker when enabled, small file downloads, sites with many images, and sites that should be blocked by the web-filter. I repeated the same virus file to increase the probability of siege using it.
Note: the file eicarcom2.zip is a zip file that contains another zip file. The 2nd zip file contains a virus. This is to test if the Virus Blocker detects multiple packing. (It does).

Siege was run with the following parameters:

for i in `seq 1 5`; do siege -i -b -c 100 -t1m; done

-i, --internet: INTERNET, generates user simulation by randomly hitting the URLs read from the urls.txt file. This option is viable only with the urls.txt file.

-b, --benchmark: BENCHMARK, runs the test with NO DELAY for throughput benchmarking. By default each simulated user is invoked with at least a one second delay. This option removes that delay.

-c NUM, --concurrent=NUM: CONCURRENT, allows you to set the concurrent number of simulated users to num.

-t NUMm, --time=NUMm: TIME, allows you to run the test for a selected period of time. The format is "NUMm", where NUM is a time unit and the "m" modifier is either S, M, or H for seconds, minutes and hours.

This means that connections are established immediately without delay and we're simulating 100 browser requests on all URLs over the period of 1 minute, five times.

The test will be taken on 3 scenarios: Direct Connection, Without Antivirus, With Antivirus.

Results

Direct Connection: Connected directly without passing any filtering appliance.

Resp Time	Trans Rate	Concurrency	Succ. Trans	Failed Trans	Failure Rate
5.53	15	82	948	68	7.17%

Without Antivirus: Connected through Untangle appliance without antivirus blocker.

Resp Time	Trans Rate	Concurrency	Succ. Trans	Failed Trans	Failure Rate
4.4	20	86	1165	83	7.12%

With Antivirus: Connected through Untangle appliance using antivirus blocker.

Resp Time	Trans Rate	Concurrency	Succ. Trans	Failed Trans	Failure Rate
4.49	19.37	86.55	1170	99	8.46%

To be continued

Compiling SimpleScalar Simulator on Linux

SimpleScalar.com provides "fast, flexible and extensible infrastructure for hardware modeling and software analysis."

Their code was last updated in late 2003, but based on 1997 code-base. I got a lot of warnings and errors that prevented the compilation. With some help from twkm from #c on DALnet (IRC), I was able to complete the compilation process (though with warnings still).

I will mention what I have modified in the files, then I'll provide you with a script and .diff files to do the whole patching and compilation process automatically. This also includes the compilation of the SPEC2000int package (compiled against simplesim-ALPHA only).

The files simplesim-3v0d.tgz and SPEC2000int.tgz are hosted on this site (which also provides a guide).

Changes

Open simplesim-3.0/target-alpha/alpha.h: Shift the line 224 to line 239.
-> "extern enum md_opcode md_mask2op[];"
Move it such that it's above this line: "/* enum md_opcode -> description string */"

After extracting SPEC2000int package, the Makefile inside needs one line to point at the directory containing the simplesim-3.0 directory.
-> "/home/students//"
Here's a sample directory hierarchy to understand:
-/
--home
---username
----tmp
-----simplesim-3.0
Your path will be: /home/username/tmp

Scripts & Automagic!

1) Create a directory in your home directory and call it: simtmp
2) Create a file called machine.diff inside simtmp
3) Put the following inside it:

--- simplesim-3.0/target-alpha/alpha.h 2003-10-09 05:14:23.000000000 +0300
+++ simplesim-3.0-fixed/machine.h 2008-12-24 14:02:22.000000000 +0300
@@ -221,7 +221,6 @@
 #define MD_MAX_MASK  2048
 
 /* internal decoder state */
-extern enum md_opcode md_mask2op[];
 extern unsigned int md_opoffset[];
 extern unsigned int md_opmask[];
 extern unsigned int md_opshift[];
@@ -236,6 +235,8 @@
   OP_MAX /* number of opcodes + NA */
 };
 
+extern enum md_opcode md_mask2op[];
+
 /* enum md_opcode -> description string */
 #define MD_OP_NAME(OP)  (md_op2name[OP])
 extern char *md_op2name[];

4) Create a file called setup.sh inside simtmp and put this in it:

#!/bin/bash
echo Extracting simplesim
/bin/tar -xf simplesim-3v0d.tgz

echo Patching simplesim
cd `pwd`
patch -p0 < `pwd`/machine.diff

echo Extracting SPEC
/bin/tar -xf SPEC2000int.tgz

echo Creating results directory
mkdir Results

echo Compile simplesim ALPHA
cd simplesim-3.0
make config-alpha
make

echo Update SPEC Makefile
cd ../
THISDIR=`pwd`
cd SPEC2000int
sed '19s:/home/students/<YOUR USERNAME>/<PATH TO YOUR SIMPLESCALAR DIRECTORY>:'$THISDIR':' ./Makefile > specmake
rm Makefile
mv specmake Makefile

echo Compile SPEC
make

5) cd /home/username/simtmp
Make sure the two packages (simplesim & SPEC2000int) are downloaded here. Don't extract them!
6) chmod +x setup.sh
7) ./setup.sh

That's it. The setup file will extract, patch & compile the programs for you. The result of the spec2000int benchmark will be in a directory called Results in the etderr files.

Spreadsheets Maximum number of Rows

I had a spreadsheet (Excel sheet for you MS Office users) with over 253,000 entries. Since I run on Linux, I use OpenOffice.org's SpreadSheet program, which unfortunately supports a maximum of 65336 rows only.

A quick search got me to a comparison between multiple products that support more than that limit. OK, I lied. Two products only.

This Wikipedia page shows a nice table but people tend to overlook the tiny little IMPORTANT numbers used for special notes!!!

The products are: Microsoft Office 2007 and Gnumeric.

Unfortunately (yes, another unfortunate event), Gnumeric requires that you recompile it with max number of rows you desire. This can be a good thing if you look at it from the point that you can surpass the limit that Microsoft is boasting.

In all cases, opening such large sheets requires large memory. I'm guessing at least 1GB (for the data itself, not counting the program!). I'll test this claim on MS Office 2007.

Winter and Computer Hardware

As the cold sneaks under our blouses and trousers, we keep stocking ourselves with more clothes, and the more we put on, the more charged we get. Static Electricity is NOT your friend!

In case you didn't know, handling computer hardware while being charged with electricity will discharge the electricity into the piece you hold and most likely rendering it a paperweight. Be it RAM, CPU, Graphics Card, or Hard disk drive, unless you hold a piece of metal before touching the piece, you're dooming it!

You should also do the same before touching the fuel hose at the gas station. As much as it's fun to see an awkwardly dancing flaming show, the after-show smell is just obnoxious...

IT Consultant

As tempting as it was, I didn't open the email.

Zain e-Go 180

The new device reported by this blog, is Huawei's E180.

The older model was being sold for 60KD, while the new one is being sold for 45KD.

Here's my advice: If you already own the old model, stick to it as there's no need to get a newer one.

The so called "speed" of 14.4Mbps, is the network speed, not the Internet speed. The network here is the communication between your mobile device (the E180) and Zain's communication tower.

People with the older devices, that have a speed of 7.2 Mbps, would never get 7.2 Mbps of Internet speed, because again, that's the network speed only.

Also, the E180's link above doesn't mention anything about supporting 14.4 Mbps, as the blog claims (or Zain claims).

Plug & Play option is great. It also exists in the older model.

Support for Windows & Mac is great. The older has the same support, and it also supports Linux. I'm using mine on Kubuntu.