Saturday, October 29, 2011

Drizzle, NGINX, PHP and PHP-APC

Introduction

This is a guide/howto on setting up Drizzle database, Nginx webserver, PHP (CGI) and PHP-apc (bytecode cache) on Debian Linux 6 (codename Squeeze).

NOTE: If the guide is too messy to read here, head over to this Google Document for better formatting.

Why?

Apache and MySQL are quite heavy and MySQL has been becoming worse thanks to the idiots in Oracle. NGINX has been getting a lot of recommendations and it’s proving itself to be quite stable and faster than any existing webserver in the arena.

Drizzle DB is the child of the co-founder of MySQL after he left MySQL when Sun bought it. Its main focus is web applications, and easy replication of databases.

From preliminary tests on a non-optimized virtual machine, I was able to reach 800 requests/sec for read/write operations and 2400 requests for read-only operations, hitting my URL shortening web application on NGINX and Drizzle. My VM had 1 core and 512MB RAM, but only 69MB RAM was used during the tests!

Stress Test

Hardware: Lenovo laptop with an Intel(R) Core(TM)2 Duo CPU P8600 @ 2.40GHz, 6GB RAM, running Debian6 32-bit.
Software: VMware Workstation v7. The VM has 512MB RAM and 1 core allocated, running Debian6 32-bit (business card installation). VMware tools were not installed during the stress tests.

I had apc statistics enabled and access logging enabled in NGINX at first, but turning them off reduced load times from 126ms to 16ms for read/write operations and from 40ms to 4ms for read-only requests.

Read/write test: ab -c13 -n 10000 http://192.168.59.135/shorten.php?longurl=http://bit.ly
Requests per second:    789.07 [#/sec] (mean)
Time per request:       16.475 [ms] (mean)
Time per request:       1.267 [ms] (mean, across all concurrent requests)
CPU utilization: 25% Drizzle, 19% NGINX, 2.3% * 15 PHP CGI processes.

Read-only test: ab -c13 -n 10000 http://192.168.59.135/a
Requests per second:    2633.09 [#/sec] (mean)
Time per request:       4.937 [ms] (mean)
Time per request:       0.380 [ms] (mean, across all concurrent requests)
CPU utilization: 49% NGINX, 3.3% * 15 PHP CGI. Drizzle wasn’t showing in “top.”

The read-only test involved engaging a REWRITE rule from NGINX, which puts a tad bit more processing on its shoulders.

The numbers above are very specific to my application, but the numbers can be much better if I install VMware tools to provide VM optimizations, so don’t let the 49% CPU usage put you off.

0) Installation
0.0) Drizzle
Note: Drizzle relies on upstart, which means sysvinit will be removed.

First, add the PPA to /etc/apt/sources.lst using your favorite editor

      deb http://ppa.launchpad.net/drizzle-developers/ppa/ubuntu maverick main
      deb-src http://ppa.launchpad.net/drizzle-developers/ppa/ubuntu maverick main

Run:
      sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 06899068
      sudo apt-get update
      sudo apt-get install drizzle
(to continue, you’ll have to type ‘Yes, do as I say!’ excluding the single quotes)


0.1) NGINX
      sudo apt-get install nginx


0.2) PHP

      sudo apt-get install php5-cgi php5-mysql php-apc

The PHP extension for drizzle is available as a PECL but it’s not kept up to date & since we’re gonna compile either way, we’ll have to install some dev packages & a compiler. Those can (should) be removed after the compilation of the extension, if you’re doing this on a production box.
      apt-get install php5-dev libdrizzle-dev make

This is to fix a bug in the configure script looking in the wrong place
      ln -s /usr/include/libdrizzle-1.0/libdrizzle /usr/include/libdrizzle


Grab the latest stable Drizzle PHP extension from here: https://launchpad.net/drizzle-php-ext
      wget http://launchpad.net/drizzle-php-ext/trunk/0.5/+download/drizzle-php-ext-0.5.tar.gz
      tar -zxf drizzle-php-ext-0.5.tar.gz
      cd drizzle-php-ext-0.5
      ./configure
      make -j2
      make install


Let’s clean up after the mess
      make clean
      apt-get remove php5-dev libdrizzle-dev make
      apt-get autoremove
      rm -I /var/cache/apt/archives/*

In my case, the modules were copied to: /usr/lib/php5/20090626+lfs/  and the module drizzle.so is ready to be included in php.ini later.

Resources:

http://www.phptutorial.info/?apc.configuration
http://devzone.zend.com/article/4793
http://php.net/manual/en/install.pecl.phpize.php
http://chrisschuld.com/2007/07/missing-phpize/

1) Configuration
1.0) NGINX

The configuration of nginx is not a standard process. Each website may have its own specific setup & needs that one would have to tweak the settings around to fit one’s needs. Make sure you refer to the referenced links to see all available options and things to avoid doing.

I’ll include my own configuration changes here and not the full configuration file.

Modify the file /etc/nginx/nginx.conf
   worker_processes 3;


   error_log /var/log/nginx/error.log;
   pid /var/run/nginx.pid;


   events {
       worker_connections 1024;
       multi_accept on;
   }


   http {
       include /etc/nginx/mime.types;
       access_log off;
       # if you want to have an access log, comment the line above and uncomment the following one
       # access_log /var/log/nginx/access.log;


       sendfile on;
       tcp_nopush on;
   }

Modify the file /etc/nginx/sites-enabled/default
   index index.php index.html index.htm;


   server {
       # this block redirects all connections to www.domain.com to domain.com 
       # this is handy for cache configurations like Varnish & for statistics
       # if you prefer to view your site as www.domain.com, swap server names
       listen 80;
       server_name www.domain.com;
       rewrite ^ $scheme://domain.com$request_uri redirect;
   }


   server {
       listen 80;
       #listen [::]:80 default ipv6only=on;


       server_name domain.com;
       root /var/www/;
       #access_log /var/log/nginx/localhost.access.log;


       location / {
           try_files $uri $uri/ /index.php;
       }


       location ~ \.(jpg|jpeg|gif|png|css|js|ico|xml)$ {
           #I don’t want it to search for directories so I removed $uri/
           try_files $uri /index.php;
           #enable this if you enabled access logging previously
           #access_log off;
           expires 30d; #useful for caches
       }


       location ~ \.php$ {
           try_files $uri /index.php;
           include fastcgi_params;
           # unix sockets are faster & better than binding to a port
           fastcgi_pass unix:/tmp/php.socket;
       }
   # remember to include other settings from the original config file
   }

Modify the file /etc/nginx/fastcgi_params
   fastcgi_param  SERVER_SOFTWARE    nginx;
   fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
   fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;



Create a new file /etc/init.d/php-cgi
#!/bin/bash
BIND=/tmp/php.socket
USER=www-data
PHP_FCGI_CHILDREN=15
PHP_FCGI_MAX_REQUESTS=1000
 
PHP_CGI=/usr/bin/php-cgi
PHP_CGI_NAME=`basename $PHP_CGI`
PHP_CGI_ARGS="- USER=$USER PATH=/usr/bin PHP_FCGI_CHILDREN=$PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS=$PHP_FCGI_MAX_REQUESTS $PHP_CGI -b $BIND"
RETVAL=0
 
start() {
      echo -n "Starting PHP FastCGI: "
      start-stop-daemon --quiet --start --background --chuid "$USER" --exec /usr/bin/env -- $PHP_CGI_ARGS
      RETVAL=$?
      echo "$PHP_CGI_NAME."
}
stop() {
      echo -n "Stopping PHP FastCGI: "
      killall -q -w -u $USER $PHP_CGI
      RETVAL=$?
      echo "$PHP_CGI_NAME."
}
 
case "$1" in
    start)
      start
  ;;
    stop)
      stop
  ;;
    restart)
      stop
      start
  ;;
    *)
      echo "Usage: php-fastcgi {start|stop|restart}"
      exit 1
  ;;
esac
exit $RETVAL

Run the commands:
      chmod +x /etc/init.d/php-cgi
      update-rc.d php-cgi defaults
You are now able to start the php cgi service: service php-cgi start

References:

1.1) PHP
These are minor modifications and you should take an overall look at the config file to see if you’d like to make any other changes.

Modify the file /etc/php5/cgi/php.ini
      ; this prevents PHP from executing scripts uploaded by users into image directories
      cgi.fix_pathinfo=0
      ; required for security reasons on CGI deployments
      cgi.force_redirect=1
      expose_php = Off
      ; original value is 128M which in my opinion is too much
      memory_limit = 64M
      ; before the File Uploads section, add this
      ; apc settings
      apc.shm_size = 32M
      apc.stat = 0
      ; before the Module Settings section, add this
      extension=drizzle.so

References:
http://devzone.zend.com/article/12618
http://www.phpjabbers.com/measuring-php-page-load-time-php17.html

1.2) Drizzle
Run:
      mkdir -p /home/drizzle/db/.temporary
      cp -Rv /var/lib/drizzle/* /home/drizzle/db/
      chmod -R 750 /home/drizzle
      chown -R drizzle:drizzle /home/drizzle
      mv /etc/init/drizzle.conf /etc/init/drizzle.conf.orig

Make a new file: /etc/init/drizzle.conf
      # Drizzle Service

      description     "Drizzle Server"
      author          "MBH "

      start on runlevel [2345]

      stop on runlevel [016]

      expect fork
      script
          DRIZ_VARS="--user drizzle --datadir /home/drizzle/db --drizzle-protocol.port 3306"
          start-stop-daemon --quiet --start --pidfile /home/drizzle/db/drizzle.pid --chuid drizzle --group drizzle --startas /usr/sbin/drizzled -- $DRIZ_VARS
      end script

Notes:
  • Drizzle doesn’t support unix sockets to make sure applications connect to 127.0.0.1 rather than localhost, if the DB is local.
  • The drizzle username created in Debian doesn’t have a shell nor a home directory, and it should remain this way.
You can now start poking with Drizzle and make your own applications. Here’s a link for a quick usage of Drizzle and PHP: http://devzone.zend.com/article/4793

References:

Monday, October 24, 2011

Kuwait Traffic Jam Survey for 2011

I created a quick survey (Arabic) to measure the sources, destinations and times of traffic in Kuwait. The survey is 14 questions only, so kindly answer them and spread the link around. Thank you!




The results will be posted on http://www.mbhbox.net later on when enough data has been collected.

Thursday, October 13, 2011

Mobile Apps Development Discussion Panel

Ahmad Al-Ibrahim, co-founder of Koutbo6.com, is moderating a discussion panel about Mobile Apps Development which is hosted by KITS (map) on Tuesday Oct 18 7:30PM-9:30PM (calendar event).

The event's guests/speakers are:
  • Abdullah Al-Khulaifi
  • Abdulrahman Al-Zanki
  • Hussain Al-Bustan
  • Fahad Al-Mudhayan
  • Mohammad Al-Meer
This is a public event and everyone is free to come & invite others, too!

Questions for the speakers can be asked from now and the moderator (Ahmad) and the audience will vote on which questions to be asked: http://goo.gl/fcSgG

Thursday, October 6, 2011

Viva's Insecure Online Payment System

Viva's website allows you to pay your bills using credit cards like VISA or MasterCard, or the regionally accepted K-Net. In the case of VISA and MasterCard, Viva pushes the data (your phone number, email, credit card number, expiration date, cvv code) in clear text, without encryption!

In any website that takes payments or has a user authentication portal, should offer a secure channel using SSL/TLS and the user sees the link starting with "https." In addition, current browsers show part of the address bar in green when the browser is able to verify that the website is secure and it is who it claims to be.

A friend was worried as she didn't see "https" in the URL, so I checked the pages' source code to see if it was sending the data in a secure channel via javascript or some other mean, without showing it in the URL, alas, it was all in the clear text.

Here's a screenshot of a sniffed packet session from my machine to viva.com.kw (94.128.1.30), while submitting the form data.


This is some of the text from the packet (I removed my personal data):
prepRechargeCreditForm%3Aholder=Mojo+Jojo&prepRechargeCreditForm%3AcardNumber=4550xxxxxxxxxxxx&prepRechargeCreditForm%3AyearList=XY&prepRechargeCreditForm%3AmonthList=XY&prepRechargeCreditForm%3Acvv2=XYZ&autoScroll=&prepRecharg

I have contacted VIVA Telecom and Mr. Salman Al-Badran (CEO) via Twitter  on Saturday Oct 1st (when I found out about the issue). Mr. Salman replied on the same day and said he'll forward it to his team. I also gave him my email address in case his team wanted to get in touch with me.

Three days later I told Mr. Salman that the problem is still there and that I'll publish my findings on my blog next Sunday (a week from reporting the issue). He replied saying it'll be fixed on Oct 6th.

[this is fixed now] I checked today (Oct 6th) and the form now redirects to a secure website (https). The address bar may not always appear in a green color; In that case, do not use the website, but instead, refresh or try again until the icon looks like this:  not like this . Description of these can be found here. (the two images were produced by Google.)

[this is fixed now] Also, it seems like the changes they made broke the form in the main page, which sends the @ sign of the email address in hex form (%40). Just change the %40 to @ and submit the form again and it'll work.

I'd like to thank Mr. Salman for his prompt response to the matter. I wish other enterprise corporates' CEOs were as attentive and interactive with the consumers as he is. I would also like to point the finger at the technical team and the audit team who let this one slip by! This is a trivial and pivotal requirement of any online payment system!


What kind of complications the insecure site would have?
An attacker in the same network as you are can capture clear text that is being sent from your machine/mobile to the website. That's why the data shows in clear text in the picture above. If the connection was secure, it would have been garbled.

If the address bar showed in red, it is still possible to attack a visitor from the same network, by altering the content that is being transmitted via insecure channels, which could lead to changing the form itself and the user would end up sending the data to a different script/page or a whole different website that the attacker crafted to collect the data.