Drizzle, NGINX, PHP and PHP-APC

Introduction

This is a guide/howto on setting up Drizzle database, Nginx webserver, PHP (CGI) and PHP-apc (bytecode cache) on Debian Linux 6 (codename Squeeze).

NOTE: If the guide is too messy to read here, head over to this Google Document for better formatting.

Why?

Apache and MySQL are quite heavy and MySQL has been becoming worse thanks to the idiots in Oracle. NGINX has been getting a lot of recommendations and it’s proving itself to be quite stable and faster than any existing webserver in the arena.

Drizzle DB is the child of the co-founder of MySQL after he left MySQL when Sun bought it. Its main focus is web applications, and easy replication of databases.

From preliminary tests on a non-optimized virtual machine, I was able to reach 800 requests/sec for read/write operations and 2400 requests for read-only operations, hitting my URL shortening web application on NGINX and Drizzle. My VM had 1 core and 512MB RAM, but only 69MB RAM was used during the tests!

Stress Test

Hardware: Lenovo laptop with an Intel(R) Core(TM)2 Duo CPU P8600 @ 2.40GHz, 6GB RAM, running Debian6 32-bit.
Software: VMware Workstation v7. The VM has 512MB RAM and 1 core allocated, running Debian6 32-bit (business card installation). VMware tools were not installed during the stress tests.

I had apc statistics enabled and access logging enabled in NGINX at first, but turning them off reduced load times from 126ms to 16ms for read/write operations and from 40ms to 4ms for read-only requests.

Read/write test: ab -c13 -n 10000 http://192.168.59.135/shorten.php?longurl=http://bit.ly
Requests per second:    789.07 [#/sec] (mean)
Time per request:       16.475 [ms] (mean)
Time per request:       1.267 [ms] (mean, across all concurrent requests)
CPU utilization: 25% Drizzle, 19% NGINX, 2.3% * 15 PHP CGI processes.

Read-only test: ab -c13 -n 10000 http://192.168.59.135/a
Requests per second:    2633.09 [#/sec] (mean)
Time per request:       4.937 [ms] (mean)
Time per request:       0.380 [ms] (mean, across all concurrent requests)
CPU utilization: 49% NGINX, 3.3% * 15 PHP CGI. Drizzle wasn’t showing in “top.”

The read-only test involved engaging a REWRITE rule from NGINX, which puts a tad bit more processing on its shoulders.

The numbers above are very specific to my application, but the numbers can be much better if I install VMware tools to provide VM optimizations, so don’t let the 49% CPU usage put you off.

0) Installation
0.0) Drizzle
Note: Drizzle relies on upstart, which means sysvinit will be removed.

First, add the PPA to /etc/apt/sources.lst using your favorite editor

      deb http://ppa.launchpad.net/drizzle-developers/ppa/ubuntu maverick main
      deb-src http://ppa.launchpad.net/drizzle-developers/ppa/ubuntu maverick main

Run:
      sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 06899068
      sudo apt-get update
      sudo apt-get install drizzle
(to continue, you’ll have to type ‘Yes, do as I say!’ excluding the single quotes)

Resouces:

http://docs.drizzle.org/installing/from_source.html#debian-install-commands

http://www.howtoforge.com/install-drizzle-on-debian-lenny

http://docs.drizzle.org/configuration.html

0.1) NGINX

      sudo apt-get install nginx

0.2) PHP

      sudo apt-get install php5-cgi php5-mysql php-apc

The PHP extension for drizzle is available as a PECL but it’s not kept up to date & since we’re gonna compile either way, we’ll have to install some dev packages & a compiler. Those can (should) be removed after the compilation of the extension, if you’re doing this on a production box.
      apt-get install php5-dev libdrizzle-dev make

This is to fix a bug in the configure script looking in the wrong place
      ln -s /usr/include/libdrizzle-1.0/libdrizzle /usr/include/libdrizzle


Grab the latest stable Drizzle PHP extension from here: https://launchpad.net/drizzle-php-ext
      wget http://launchpad.net/drizzle-php-ext/trunk/0.5/+download/drizzle-php-ext-0.5.tar.gz
      tar -zxf drizzle-php-ext-0.5.tar.gz
      cd drizzle-php-ext-0.5
      ./configure
      make -j2
      make install


Let’s clean up after the mess
      make clean
      apt-get remove php5-dev libdrizzle-dev make
      apt-get autoremove
      rm -I /var/cache/apt/archives/*

In my case, the modules were copied to: /usr/lib/php5/20090626+lfs/  and the module drizzle.so is ready to be included in php.ini later.

Resources:

http://www.phptutorial.info/?apc.configuration
http://devzone.zend.com/article/4793
http://php.net/manual/en/install.pecl.phpize.php
http://chrisschuld.com/2007/07/missing-phpize/

1) Configuration
1.0) NGINX

The configuration of nginx is not a standard process. Each website may have its own specific setup & needs that one would have to tweak the settings around to fit one’s needs. Make sure you refer to the referenced links to see all available options and things to avoid doing.

I’ll include my own configuration changes here and not the full configuration file.

Modify the file /etc/nginx/nginx.conf
   worker_processes 3;


   error_log /var/log/nginx/error.log;
   pid /var/run/nginx.pid;


   events {
       worker_connections 1024;
       multi_accept on;
   }


   http {
       include /etc/nginx/mime.types;
       access_log off;
       # if you want to have an access log, comment the line above and uncomment the following one
       # access_log /var/log/nginx/access.log;


       sendfile on;
       tcp_nopush on;
   }

Modify the file /etc/nginx/sites-enabled/default
   index index.php index.html index.htm;


   server {
       # this block redirects all connections to www.domain.com to domain.com 
       # this is handy for cache configurations like Varnish & for statistics
       # if you prefer to view your site as www.domain.com, swap server names
       listen 80;
       server_name www.domain.com;
       rewrite ^ $scheme://domain.com$request_uri redirect;
   }


   server {
       listen 80;
       #listen [::]:80 default ipv6only=on;


       server_name domain.com;
       root /var/www/;
       #access_log /var/log/nginx/localhost.access.log;


       location / {
           try_files $uri $uri/ /index.php;
       }


       location ~ \.(jpg|jpeg|gif|png|css|js|ico|xml)$ {
           #I don’t want it to search for directories so I removed $uri/
           try_files $uri /index.php;
           #enable this if you enabled access logging previously
           #access_log off;
           expires 30d; #useful for caches
       }


       location ~ \.php$ {
           try_files $uri /index.php;
           include fastcgi_params;
           # unix sockets are faster & better than binding to a port
           fastcgi_pass unix:/tmp/php.socket;
       }
   # remember to include other settings from the original config file
   }

Modify the file /etc/nginx/fastcgi_params
   fastcgi_param  SERVER_SOFTWARE    nginx;
   fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
   fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;

Create a new file /etc/init.d/php-cgi

#!/bin/bash

BIND=/tmp/php.socket

USER=www-data

PHP_FCGI_CHILDREN=15

PHP_FCGI_MAX_REQUESTS=1000

 

PHP_CGI=/usr/bin/php-cgi

PHP_CGI_NAME=`basename $PHP_CGI`

PHP_CGI_ARGS="- USER=$USER PATH=/usr/bin PHP_FCGI_CHILDREN=$PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS=$PHP_FCGI_MAX_REQUESTS $PHP_CGI -b $BIND"

RETVAL=0

 

start() {

      echo -n "Starting PHP FastCGI: "

      start-stop-daemon --quiet --start --background --chuid "$USER" --exec /usr/bin/env -- $PHP_CGI_ARGS

      RETVAL=$?

      echo "$PHP_CGI_NAME."

}

stop() {

      echo -n "Stopping PHP FastCGI: "

      killall -q -w -u $USER $PHP_CGI

      RETVAL=$?

      echo "$PHP_CGI_NAME."

}

 

case "$1" in

    start)

      start

  ;;

    stop)

      stop

  ;;

    restart)

      stop

      start

  ;;

    *)

      echo "Usage: php-fastcgi {start|stop|restart}"

      exit 1

  ;;

esac

exit $RETVAL

Run the commands:

      chmod +x /etc/init.d/php-cgi

      update-rc.d php-cgi defaults

You are now able to start the php cgi service: service php-cgi start

References:

http://wiki.nginx.org/DirectiveIndex

http://wiki.nginx.org/IfIsEvil

http://wiki.nginx.org/Pitfalls

http://wiki.nginx.org/NginxHttpLogModule#access_log

http://wiki.nginx.org/PHPFcgiExample

http://tomasz.sterna.tv/2009/04/php-fastcgi-with-nginx-on-ubuntu/

http://www.howtoforge.com/installing-php-5.3-nginx-and-php-fpm-on-ubuntu-debian

http://interfacelab.com/nginx-php-fpm-apc-awesome/

1.1) PHP

These are minor modifications and you should take an overall look at the config file to see if you’d like to make any other changes.

Modify the file /etc/php5/cgi/php.ini

      ; this prevents PHP from executing scripts uploaded by users into image directories

      cgi.fix_pathinfo=0

      ; required for security reasons on CGI deployments

      cgi.force_redirect=1

      expose_php = Off

      ; original value is 128M which in my opinion is too much

      memory_limit = 64M

      ; before the File Uploads section, add this

      ; apc settings

      apc.shm_size = 32M

      apc.stat = 0

      ; before the Module Settings section, add this

      extension=drizzle.so

References:
http://devzone.zend.com/article/12618

http://www.phpjabbers.com/measuring-php-page-load-time-php17.html

1.2) Drizzle

Run:

      mkdir -p /home/drizzle/db/.temporary

      cp -Rv /var/lib/drizzle/* /home/drizzle/db/

      chmod -R 750 /home/drizzle

      chown -R drizzle:drizzle /home/drizzle

      mv /etc/init/drizzle.conf /etc/init/drizzle.conf.orig

Make a new file: /etc/init/drizzle.conf

      # Drizzle Service

      description     "Drizzle Server"

      author          "MBH "

      start on runlevel [2345]

      stop on runlevel [016]

      expect fork

      script

          DRIZ_VARS="--user drizzle --datadir /home/drizzle/db --drizzle-protocol.port 3306"

          start-stop-daemon --quiet --start --pidfile /home/drizzle/db/drizzle.pid --chuid drizzle --group drizzle --startas /usr/sbin/drizzled -- $DRIZ_VARS

      end script

Notes:

• Drizzle doesn’t support unix sockets to make sure applications connect to 127.0.0.1 rather than localhost, if the DB is local.
• The drizzle username created in Debian doesn’t have a shell nor a home directory, and it should remain this way.

You can now start poking with Drizzle and make your own applications. Here’s a link for a quick usage of Drizzle and PHP: http://devzone.zend.com/article/4793

References:

http://www.linuxjedi.co.uk/?p=35

http://upstart.ubuntu.com/wiki/

http://upstart.ubuntu.com/wiki/UnprivilegedUsers

http://devzone.zend.com/article/4793

http://docs.drizzle.org/options.html

http://docs.drizzle.org/

http://www.howtoforge.com/install-drizzle-on-debian-lenny

http://azhari.harahap.us/2011/04/connecting-drizzle-from-php-and-phpmyadmin/

Word Repetition Count and Highlight

Intro
Ever wanted something to count all the repeated words and highlight each word with a different color? No? Well I did!

I looked around but didn't find what I wanted so I wrote my own. A quick & dirty PHP script. It was a good waste of those extra 3 hours I had.

Caveats
Script limitations/bugs:

• All punctuation is removed
• Some long words are not counted for some reason
• The colors are random so some may not be pleasant
• Sometimes it matches partial words sometimes it doesn't.

It's always good practice to sacrifice a Microsoftie before running/using the script, to make sure things run smooth. Depending on the kind of text, you may want to add Lawyers, Salesmen, Marketers and Apple Fanboys into the mix.

Script Types
I wrote 2 scripts: One for use for the command line (read from a text file and outputs to shell, which you could redirect to another file) and another for the web. You can modify the web one to make it check URLs directly, but I'm warning you: It'll massacre all kinds of styling and you'll need to do some extra work to make work properly.

Links
You can [ab]use the script here: http://sectorx.org/word-highlight-web.php

The scripts are available freely for download & redistribution as long as the credits are retained. If you make enhancements to the scripts, I'd appreciate it if you send them back to me to add them to the others (with your credits retained of course).

Demo
What better demo is than to test it on cats and geeks? :D

Yes, I cheated by manually placing the text in the middle for a better screenshot. Spank me for being bad.

CakePHP: Initial Problems

I just started with CakePHP, a PHP framework, and have been having some problems going through the "15 minutes" blog tutorial. It took me about 2 days to dig my way out through openSUSE, Apache, Linux and CakePHP itself.

First issue I had was enabling mod_rewrite on openSUSE. To do that, edit the file: "/etc/sysconfig/apache2" (Assuming Apache2.2)

• Go to the line that starts with "APACHE_MODULES" and add "rewrite" at the end just before the closing quote.
• Save the file then run "sudo SuSEconfig" to update Apache's configuration files.

I'm not using virtual hosts. I have downloaded the latest stable CakePHP and extracted it in my "/srv/www/htdocs" directory and named it "blog"

mj@linux-mlvm:~$ ls -l /srv/www/htdocs
drwxr-xr-x  5 mj   users 4096 2010-01-23 02:28 blog

So I access the blog through: http://localhost/blog/

After configuring the database and the connection settings for CakePHP as instructed in the tutorial, I had to set permissions for the tmp directory: "/srv/www/htdocs/blog/app/tmp"

sudo chown wwwrun /srv/www/htdocs/blog/app/tmp

This will set ownership of the directory to the user running Apache. In my case, it's wwwrun. On a shared web host, it's often nobody. You can find out which username Apache uses by making a small PHP script:

<?php echo `whoami`; ?>

Visit the script in a web browser and voila!

Second issue: when reaching the step to create the posts view, I created the directory:

mkdir app/views/posts

But I got an error message in the browser when visiting the link: http://localhost/blog/posts:

Missing View
Error: The view for PostsController::index() was not found.
Error: Confirm you have created the file: /srv/www/htdocs/blog/app/views/posts/index.ctp
Notice: If you want to customize this error message, create app/views/errors/missing_view.ctp

After verifying that everything is properly spelled and whatnot, turns out it was a dumb mistake of not checking directory permissions:

mj@linux-mlvm:/srv/www/htdocs/blog/app$ ls -la
drwxr-xr-x 12 mj     users 4096 2010-01-23 02:03 .
drwxr-xr-x  5 mj     users 4096 2010-01-23 02:28 ..
drwxr-xr-x  3 mj     users 4096 2010-01-12 05:58 config
drwxr-xr-x  3 mj     users 4096 2010-01-21 22:32 controllers
-rw-------  1 mj     users   67 2010-01-23 02:03 .directory
-rw-r--r--  1 mj     users  169 2010-01-23 02:08 .htaccess
-rw-r--r--  1 mj     users  859 2010-01-12 01:47 index.php
drwxr-xr-x  3 mj     users 4096 2010-01-12 01:47 locale
drwxr-xr-x  4 mj     users 4096 2010-01-19 06:54 models
drwxr-xr-x  2 mj     users 4096 2010-01-12 01:47 plugins
drwxr-xr-x  5 mj     users 4096 2010-01-12 01:47 tests
drwxr-xr-x  6 wwwrun www   4096 2010-01-12 01:47 tmp
drwxr-xr-x  3 mj     users 4096 2010-01-12 01:47 vendors
drwxr--r--  8 mj     users 4096 2010-01-19 07:00 views
drwxr-xr-x  5 mj     users 4096 2010-01-19 07:36 webroot

Never mind the fact that I'm using my own username for the directories. It's just for quick editing.
Focus on the bold line which has "views" -- since I didn't assign the user wwwrun for that directory, the permissions prevented Apache from accessing the directory resulting in the above Missing error message.

You can either change directory ownership as done for tmp, or change permissions:

chmod 755 views

Note: If a directory is not executable, you can't change to it.

Backup Storage Space Estimation

 

a work in progress


Some special characters will be used to give certain meanings:
[ ]: Whatever is between the square brackets, means it's optional.
< >: Whatever is between the angle brackets, means it's mandatory.

Table of Contents:
1. Possible Methods of Backup
  1.1. Scenarios of Foiling Backups
  1.2. Why The Method Matters
2. Estimation Approach
3. Weapons of Mass Dysfunction
  3.1. Guns Loaded
  3.2. Marching
  3.3. Kung-Fu Style

1. Possible Methods of Backup

I was asked to look for FileServer options, to hold backups of employees' data, or hold the data itself.

There are 3 possible schemes, that I could think of:

1. Allocate a shared folder/directory on the fileserver, that only the employee has access to, and force all data to be saved there, not on the local disk.


2. Employees use the local disk for storage, but a copy is kept on a [shared] directory on the fileserver.


3. Employees use the local disk for storage, but a copy is kept on a [shared] directory on the fileserver. File versioning is enabled here, where the latest copy of the file, and previous copies are kept.

1.1. Scenarios of Foiling Backups

The most reliable method is the 3^rd, because if an employee decided to take revenge on the company, s/he would delete the files on the shared directory, or would delete the information inside the existing files, leading to blank files, that will be automatically backed up at a later time. In these 2 scenarios, methods 1 & 2 are defeated, respectively.
Method 3 is foiled when an employee saves the same file, without content (as an example), more times than the maximum versions kept on the fileserver. The result: A set of versions of the file, that have no content.

1.2. Why The Methods Matters

The method should be chosen, depending on the importance of the data. A combination of methods can be used, depending on the employee's rank, and the priority of the data being held. The chosen method will affect the estimation process of the amount of required storage for the backups.

2. Estimation Approach

My idea is to collect some data from workstations, then calculate averages and perform sums, to get the most correct estimation.
The data to be collected is a list of files' path, size, date of creation, last modification & last access. Only the files residing in the user's directory, under "Documents and Settings." will be checked.
As you can see, I already assumed that the machines are running on Windows, and I'm going to further assume it's Windows XP, on an NTFS filesystem. The latter assumption is important, because the required tools to get the data you need, could be specific to a filesystem type.

After data is collected, it's separated into departments; the average file size is calculated for each file extension, and correlations between file creation and modification dates are created, to be able to estimate the amount & average size of new files created & modified per period, hence, estimate the amount of data growth per period. File last access times can be used to estimate the hit-rate on the server. One might need multiple network cards, to serve a big crowd. It's a must, if the fileserver is to have space for other servers as well (i.e., not just for backups).
In case of using file-versioning, keep in mind that file modification could result in copying the whole file as new; this depends on how the versioning utility works: Some copy the difference, others copy the whole file.
The period is a variable, and can be changed to produce different sets of data. Daily check could be beneficial to large corporations, while weekly or monthly, seem more suitable for small to medium companies.

The files' path can be used to calculate the number of sub-directories, and number of files per directory. This can be handy when choosing the hardware for the fileserver, because a directory that holds many sub-directories, with a lot of files inside, requires a lot of RAM & CPU horsepower, however, this applies only when someone wants to restore his/her data. The frequency & times of restores are the factor.

3. Weapons of Mass Dysfunction

Now that the idea is clear, it's time to get some tools & start hacking away. What I need is something to get me all the information I need, quickly, without clutter, without damage, and preferably in command line!
Quickly, in the sense that all information is gathered in one shot, as opposed to traversing the target's directories over & over for each piece of info I'm after. Without clutter means getting the info in a tabular way, to minimize or reduce any chance of filtering and organizing fields of information. It won't be funny, if I caused a faulty filesystem when running a tool, would it? Yeah, I guess not. As for the command line, I just love command line interface (CLI) tools! They give the user so much control over the way the application is used; they can be scheduled, run from scripts, batch files, or other applications, and direct their output to text files, all customized to your set of parameters & options.

3.1. Guns Loaded

Here's a list of what I'm using for this project:

• PHP: To write scripts to manipulate text, and a few more tricks.


• FileList: Traverses the directories & gather the required info.


• Bambalam PHP Compiler: Compile PHP scripts into standalone EXE files.


• Curl: To send files to an HTTP/FTP server.


• Apache HTTP Server: To put a PHP script to receive the files.


• 7-zip: Compression tool.

PHP helps me filter unwanted text and use what I want as an input to other applications, as I can call them from within my PHP script. FileList, is a free tool, that traverses directories and gathers the information I need; it supports patterns and most importantly, the only tool I found that can get the creation time of a file! And it's amazingly fast. Since I'll be executing applications from within PHP, and sending parameters and passwords, I decided to compile my script to an EXE, so that no one looks at the source; a great advantage of doing this, is that I no longer need to put PHP on employees' machines! Apache HTTP is a famous web server, and since I already have a machine at work setup with HTTPD running, I'll use it to receive my files. There is no need to download Curl, as it's part of PHP and is used as an extension.

* Note: All the programs above are free, and all except FileList, are OpenSource.

3.2. Marching

The plan is to execute as follows:

1. Run "run.exe" with proper parameters
2. "run.exe" fetches the machine's globally unique MAC Address
3. "run.exe" feeds the MAC to "FileList.exe" to use it as an output file name
4. "FileList.exe" traverses the target user directory & its sub-dirs, and gathers info
5. "FileList.exe" exits and "run.exe" calls "7z.exe" to compress the output with a password
6. "run.exe" deletes "FileList.exe"'s output file
7. "run.exe" submits the compressed file to the web server
8. PHP script on HTTPD receives the file and saves it on the server
9. "run.exe" deletes all directory files, when server receives the file, then it exits
10. The Puppet Master (me) gathers all files from the web server
11. Files are extracted & sorted by departments
12. A PHP script or more do the proper analysis on the files

3.3. Kung-Fu Style

The time has come, my child, to start the brain whipping and code something useful!

code goes here


To be continued