Techy Title Here: February 2016

Monday, February 29, 2016

Freeing Disk from VMware Virtual Flash Read Cache (vFRC)

I was toying with vFRC in my lab and when I was done, I deleted the volume from the vSphere web client, but the local flash disk had retained its GPT partition format and was still claimed as a VMFS volume. I was unable to use that disk for other applications.

Try deleting using the web client:

Select the host then go to Manage tab then select Storage option and from there choose the Storage Devices entry. Select the disk, then click on the gear icon and choose Erase Partitions. Make sure you selected the right disk because this will wipe everything.

Via CLI: To delete the disk partition, first enable SSH on the host, then login and list all disks:
ls -l /vmfs/devices/disks/

Sample output:
ls -l /vmfs/devices/disks/
total 495867432
-rw-------    1 root     root     8004304896 Feb 29 08:45 mpx.vmhba32:C0:T0:L0
-rw-------    1 root     root       4161536 Feb 29 08:45 mpx.vmhba32:C0:T0:L0:1
-rw-------    1 root     root     262127616 Feb 29 08:45 mpx.vmhba32:C0:T0:L0:5
-rw-------    1 root     root     262127616 Feb 29 08:45 mpx.vmhba32:C0:T0:L0:6
-rw-------    1 root     root     115326976 Feb 29 08:45 mpx.vmhba32:C0:T0:L0:7
-rw-------    1 root     root     299876352 Feb 29 08:45 mpx.vmhba32:C0:T0:L0:8
-rw-------    1 root     root     2684354560 Feb 29 08:45 mpx.vmhba32:C0:T0:L0:9
-rw-------    1 root     root     128035676160 Feb 29 08:45 t10.ATA_____ADATA_SP600_____________________________7F1820011415________
-rw-------    1 root     root     128033579008 Feb 29 08:45 t10.ATA_____ADATA_SP600_____________________________7F1820011415________:1
-rw-------    1 root     root     120034123776 Feb 29 08:45 t10.ATA_____KINGSTON_SV300S37A120G__________________50026B7255068D61____
-rw-------    1 root     root     120032591872 Feb 29 08:45 t10.ATA_____KINGSTON_SV300S37A120G__________________50026B7255068D61____:1
lrwxrwxrwx    1 root     root            20 Feb 29 08:45 vml.0000000000766d68626133323a303a30 -> mpx.vmhba32:C0:T0:L0
lrwxrwxrwx    1 root     root            22 Feb 29 08:45 vml.0000000000766d68626133323a303a30:1 -> mpx.vmhba32:C0:T0:L0:1
lrwxrwxrwx    1 root     root            22 Feb 29 08:45 vml.0000000000766d68626133323a303a30:5 -> mpx.vmhba32:C0:T0:L0:5
lrwxrwxrwx    1 root     root            22 Feb 29 08:45 vml.0000000000766d68626133323a303a30:6 -> mpx.vmhba32:C0:T0:L0:6
lrwxrwxrwx    1 root     root            22 Feb 29 08:45 vml.0000000000766d68626133323a303a30:7 -> mpx.vmhba32:C0:T0:L0:7
lrwxrwxrwx    1 root     root            22 Feb 29 08:45 vml.0000000000766d68626133323a303a30:8 -> mpx.vmhba32:C0:T0:L0:8
lrwxrwxrwx    1 root     root            22 Feb 29 08:45 vml.0000000000766d68626133323a303a30:9 -> mpx.vmhba32:C0:T0:L0:9
lrwxrwxrwx    1 root     root            72 Feb 29 08:45 vml.010000000035303032364237323535303638443631202020204b494e475354 -> t10.ATA_____KINGSTON_SV300S37A120G______________                                   ____50026B7255068D61____
lrwxrwxrwx    1 root     root            74 Feb 29 08:45 vml.010000000035303032364237323535303638443631202020204b494e475354:1 -> t10.ATA_____KINGSTON_SV300S37A120G____________                                   ______50026B7255068D61____:1
lrwxrwxrwx    1 root     root            72 Feb 29 08:45 vml.01000000003746313832303031313431352020202020202020414441544120 -> t10.ATA_____ADATA_SP600_________________________                                   ____7F1820011415________
lrwxrwxrwx    1 root     root            74 Feb 29 08:45 vml.01000000003746313832303031313431352020202020202020414441544120:1 -> t10.ATA_____ADATA_SP600_______________________                                   ______7F1820011415________:1

Find your disk there, and then list its partitions:
partedUtil getptbl /vmfs/devices/disks/

Sample output:
partedUtil getptbl /vmfs/devices/disks/vml.010000000035303032364237323535303638443631202020204b494e475354
gpt
14593 255 63 234441648
1 2048 234440703 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

You can see above that there's one partition labeled as "vmfs" which we need to get rid of. The leading number (in blue) is the partition number.

To delete the partition:
partedUtil delete /vmfs/devices/disks/

Sample output:
partedUtil delete /vmfs/devices/disks/vml.010000000035303032364237323535303638443631202020204b494e475354 1

Done. Look in vSphere web client and it should now report 0 primary partitions on that disk and you're free to use it for something else.

Check the partition table:
partedUtil getptbl /vmfs/devices/disks/vml.010000000035303032364237323535303638443631202020204b494e475354
gpt
14593 255 63 234441648

Saturday, February 6, 2016

NGINX with High Security Ciphers and LetsEncrypt

I want to move away from the bloated Apache web server and NGINX meets my requirements, but this time I want to use SSL/TLS with signed certificates with the highest security ciphers that support Perfect Forward Secrecy, because why not?

Sadly, the information was scattered and not everything is there in the manuals, so this is a documentation of what I've found and done in my setup.

The Let's Encrypt project provides authenticated and validated domain certificates for free! The catch? They expire every 90 days and their official client requires root access & dependencies, but you can (auto)renew and avoid these. Read on to know more.

Article Updates

Mar 3rd

Corrected root's crontab entry.
Corrected headers' content and location
Added more info about security and privacy headers

Environment

My setup consists of the stuff below. This post will presume Debian & NGINX are already installed. In the steps below, a line starting with "#" means it's a command you should type. Type the command without the "#" character (not necessarily as root).

Debian Jessie (8)

#cat /etc/issue

NGINX version 1.6.2, installed from nginx-full package.

#nginx -v

OpenSSL 1.0.1k

#openssl version

Python 2.7.9

python --version

acme-tiny Dec 29, 2015

If you have an older version of openssl or nginx, you're likely to face problems and failures since new ciphers have been introduced in recent versions of OpenSSL only (1.0.1h) and the same for NGINX's settings. Make sure your distro supports the latest versions, otherwise you'll be leaving yourself and your visitors vulnerable.

Why acme-tiny?

The official letsencrypt client requires installing some dependencies such as gcc (GNU C Compiler) and some other things, in addition to requiring it being run as root, not only once, but as a daemon or in a cronjob as it requires to renew the certificate every 90 days!

As much as I appreciate the Let's Encrypt initiative, I'm not granting their software root access to my machines, nor installing gcc on a production machine. That's where acme-tiny comes in: a small (200 lines) client that is using Let's Encrypt API calls and you can (and should) audit the client's code before using it, since it's only 200 lines of human-readable Python code.

Configuring NGINX for TLS/PFS

SSL is dead. You should be using TLS only, and if you don't have to service old devices (Android 4.x, old IE browsers, Windows XP), then you should be using TLS v1.2 only with a strict set of ciphers.

Perfect Forward Secrecy (PFS) is an old standard but hasn't been widely adopted until after Snowden revealed the amount of encrypted data being stored for later decryption. PFS cycles the encryption key during the session, so even when a session is captured, decryption will be possible only for a small portion as the key changes.

TLS Config

If you're going to configure a wildcard certificate, place the config in /etc/nginx/nginx.conf. Otherwise if the certificate is unique to a specific domain/subdomain, you'll need to place the config in a virtual host config file.

In my case, I started with a wildcard but it self a self-signed certificate and was rejected by browsers, which is normal. Later when I made a Let's Encrypt certificate, I moved it to the specific subdomain.

Note: Let's Encrypt doesn't support wildcard certs as of this writing, however, they allow you up to 100 domains/subdomains.

Edit /etc/nginx/nginx.conf

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
        worker_connections 256;
        multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
 # Change the cache name. Read the manual for more info.
        ssl_session_cache shared:YourSSLCacheNameHere:10m;
        ssl_session_timeout 10m;
        ssl_session_tickets off;
        ssl_stapling on;
        ssl_stapling_verify on;
 # contains CBC AES algs which I do not like
        #ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
 # AES256 GCM is not yet supported on most browsers
        #ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384';
        # TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 & TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 ssl_ciphers "EECDH+AESGCM";
        ssl_ecdh_curve secp384r1;
 # self-generated 4096 DH key range
        ssl_dhparam /etc/nginx/ssl/dhparam.pem;

 # Put IPs of your hosting provider here, or a trusted DNS provider. These are Google's.
 resolver 8.8.8.8 8.8.4.4 [2001:4860:4860::8888] valid=300s;
        resolver_timeout 5s;

        # wildcard cert config should go here, if any
        #ssl_certificate /etc/nginx/ssl/;
        #ssl_certificate_key /etc/nginx/ssl/;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs  
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

Make sure you correct the line breaks if you paste. Due to styling on my blog, you may have one line spilling to multiple lines in the config, and this will break your config.

Don't worry about non-existing directories. We'll come to those later as we finish the setup.

Default Virtual Host Config

I don't use a default domain (www, for example) as mine are hidden from public. If you're like me, then this config fits you, otherwise move to the step below.

Edit /etc/nginx/sites-enabled/default

# Default server configuration
server {
 # change IP to match yours
        listen 127.0.0.1:80 default_server;
 # uncomment to enable IPv6
        #listen [::1]:80 default_server;
 # uncomment to enable ssl on IPv4
        listen 127.0.0.1:443 ssl default_server;
 # uncomment to enable ssl on IPv6
        #listen [::1]:443 ssl default_server;
        server_name _; #default server

        ssl_certificate /etc/nginx/ssl/default_wild.crt;
        ssl_certificate_key /etc/nginx/ssl/default_wild.key;

        root /var/www/html;

        # Add index.php to the list if you are using PHP
        #index index.html index.htm index.nginx-debian.html;
        index index.html;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
                autoindex off;
        }
}

This config will load when someone visits the IP(s) NGINX is configured at.

Virtual Host Config

This is where your subdomain config goes. In my case, the certificate belongs to this specific subdomain, so the certificate lines are added here. If you were using a wildcard cert, you should move them to nginx.conf above.

Create a file for your subdomain /etc/nginx/sites-available/mysubdom

server {
        listen 127.0.0.1:80;
 # uncomment if you want IPv6
        #listen [::1]:80;
        #listen 127.0.0.1:443 ssl;
        #listen [::1]:443 ssl;
        server_name subdomain.domain.com;
        keepalive_timeout 70;

        # The certificate is for subdomain.domain.com only
        #ssl_certificate /var/www/challenge/subdomain_chained.crt;
        #ssl_certificate_key /etc/nginx/ssl/subdomain.key;

        root /var/www/subdomain;

        # Add index.php to the list if you are using PHP
        #index index.html index.htm index.nginx-debian.html;
        index index.html;

        # letsencrypt challenge directory to verify domain
        location /.well-known/acme-challenge/ {
                alias /var/www/challenge/;
                try_files $uri =404;
 }

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
                autoindex off; #enable if you want file listing
        }

}

Notice that listening for SSL/TLS is not yet enabled and the ssl_certificate line and the one below have a hash to comment it. This is required for the initial setup since we'll need to reload nginx and it'll fail since the files are not there yet. We'll enable these lines once everything is done.

To make this config file active by NGINX, you need to link it to sites-enabled:

ln -s /etc/nginx/sites-available/mysubdom /etc/nginx/sites-enabled/mysubdom

Create the directory for your subdomain to serve files:

mkdir /var/www/subdomain
mkdir -p /var/www/challenge/.well-known/acme-challenge
chown -R www-data:www-data /var/www/subdomain
chmod 775 /var/www/subdomain
chmod 771 /var/www/challenge

www-data is the user that NGINX runs as, as shown in the first NGINX config above. Don't worry about the challenge directory owner for now. It'll be taken care of later.

Private Keys and Certificates

The overall config is done. What's left is generating private keys, deriving a certificate for the subdomain, then finally working with Let's Encrypt client.

Create the directory /etc/nginx/ssl to place the subdomain private keys and other things in there:

mkdir /etc/nginx/ssl

Modify its permissions to be restricted to root and only those who know exactly which file to use:

chmod 751 /etc/nginx/ssl

Now inside the ssl directory, generate a 4096 bit Diffie-Hellman parameters file (prime numbers) to act as seeds for the PFS/TLS sessions (this will take a VERY LONG time):

openssl dhparam -out dhparam.pem 4096

Generate a self-signed certificate to be used for the default virtual host (i.e., not the one you care about). This will be served to anyone accessing the IP or any subdomain other than the one you specifically define in the virtual host:

openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -sha512 -keyout /etc/nginx/ssl/default_wild.key -out /etc/nginx/ssl/default_wild.crt

If you don't configure this, users will be served your legitimate certificate and they'll be able to find your "hidden" subdomain. Only do the above if you want your domain/subdomain hidden.

Generate a subdomain private key and a certificate request:

openssl genrsa 4096 > subdomain.key
openssl req -new -sha512 -key subdomain.key -subj "/CN=subdomain.domain.com" > subdomain.csr

This one is the domain/subdomain that will be valid to the world. it can also be "domain.com" if you like.

Let's Encrypt and ACME-Tiny

For security purposes, it's best to have the client run as a separate user. Should anything go wrong in the future, its access would be quite isolated.

Environment Setup

Create a user for it:

useradd -m letsencrypt

Copy the subdomain csr file and set home directory permissions:

chmod 751 /home/letsencrypt
cp /etc/nginx/ssl/subdomain.csr /home/letsencrypt/
chown -R letsencrypt:letsencrypt /home/letsencrypt
chown -R letsencrypt:letsencrypt /var/www/challenge

Now switch user to become the letsencrypt user for the rest of the commands:

su - letsencrypt
openssl genrsa 4096 > account.key
wget https://raw.githubusercontent.com/diafygi/acme-tiny/master/acme_tiny.py
chmod 400 account.key
chmod 400 acme_tiny.py
chmod 400 subdomain.csr

The account.key is your private key to identify you to Let's Encrypt. Keep it safe!:

ls -l
-r-------- 1 letsencrypt letsencrypt 9150 Feb  6 12:13 acme_tiny.py
-r-------- 1 letsencrypt letsencrypt 3247 Feb  6 12:44 private.key
-r-------- 1 letsencrypt letsencrypt 1594 Feb  6 12:38 subdomain.csr

Now exit to be root (or you can use sudo) and restart nginx:

service nginx restart

If there are no errors here, it's all good, otherwise look into /var/log/nginx/error.log for hints.

Script Execution

Now that NGINX is functioning on port 80, it will be used to verify the subdomain ownership. acme-tiny writes to LetsEncrypt.org via APIs and they reply with a random hash that is written to the challenge directory, which is accessible via NGINX on port 80, and then LetsEncrypt.org checks that this hash actually exists at the subdomain you supplied and then verifies you.

su - letsencrypt
python acme_tiny.py --account-key account.key --csr subdomain.csr --acme-dir /var/www/challenge/ > /var/www/challenge/subdomain.crt

All should go OK without errors. If any, verify directory paths and file and directory permissions. Make sure the username "letsencrypt" has access to the files private.key, subdomain.csr and the challenge directory.

NGINX requires concatenating the intermediate certificate to the freshly signed certificate from Let's Encrypt:

wget -O /var/www/challenge/lets-encrypt-x1-cross-signed.pem https://letsencrypt.org/certs/lets-encrypt-x1-cross-signed.pem
cat /var/www/challenge/subdomain.crt /var/www/challenge/lets-encrypt-x1-cross-signed.pem > /var/www/challenge/subdomain_chained.crt

That's it! It should now work after enabling the SSL/TLS settings in NGINX.

Enable TLS in NGINX

Modify the file /etc/nginx/sites-enabled/mysubdom to make it look like this:

server {
        listen 127.0.0.1:80;
 # uncomment if you want IPv6
        #listen [::1]:80;
        server_name subdomain.domain.com;

 # force all traffic to go to HTTPS instead of HTTP
        return 301 https://subdomain.domain.com$request_uri;
}

server {
        listen 127.0.0.1:443 ssl;
        #listen [::1]:443 ssl;
        server_name subdomain.domain.com;
        keepalive_timeout 70;

        # The certificate is for subdomain.domain.com only
        ssl_certificate /var/www/challenge/subdomain_chained.crt;
        ssl_certificate_key /etc/nginx/ssl/subdomain.key;

        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
        add_header X-Frame-Options DENY; #or "SAMEORIGIN" always;
        add_header X-Content-Type-Options nosniff;
        add_header Content-Security-Policy 'default-src https://subdomain.domain.com:443';
        add_header X-Xss-Protection '1; mode=block';


        root /var/www/subdomain;

        # Add index.php to the list if you are using PHP
        #index index.html index.htm index.nginx-debian.html;
        index index.html;

        # letsencrypt challenge directory to verify domain
        location /.well-known/acme-challenge/ {
                alias /var/www/challenge/;
                try_files $uri =404;
 }

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
                autoindex off; #enable if you want file listing
        }

}

Notice how listening on port 80 (HTTP) has been shifted to its own segment while the rest uses HTTPS exclusively. Future certificate renewals can also go over HTTPS as long as the certificate is still valid. If not, revert the config to be as it was at the beginning.

Reload NGINX to read the certificates and make the settings active:

service nginx reload

Note: Reload reads the settings again without dropping connections. It's advised for live websites.

About Headers

Previously, I had the security headers in the main nginx.conf file, but that will apply the same headers to all websites, and that's not scalable nor correct. According to Igor Sysoev (NGINX's creator), he created the config in NGINX to not inherit so that troubleshooting becomes simpler. Duplicating code is good because it makes life easy in finding the problem when things go wrong. See the link for his talk below in the references.

This means that headers (and other configs) should be repeated for every virtual host you configure. If you configure a header in the main block in nginx.conf then define another (or modified) header in the subdomain block, the latter will take over and the first one will be ignored.

About Security Headers

The SecurityHeaders service recommends using HTTP Public-Key Pinning (Stapling) or HPKP for short, but there are privacy and performance concerns with that: Pinning means the public key of your own certificate is sent in the header and is sent to your certificate issuer to validate it. This prevents a Man-in-the-Middle attack, but exposes your visit(s) to the certificate issuer! Additionally, it puts a huge burden on the certificate issuer to scale their own performance to reply to every single site visit. If they don't (and why should they?), your site visiting experience will suffer great delays.

The headers also tell the browser to cache your public keys for a very long period (3+ months) to protect you against forged certificates that could come during that period, but since we're using Let's Encrypt certificates which expire every 3 months, it'll become hectic to manage the headers, aging and other aspects.

With all these concerns, I decided against adding Public-Key Pinning headers in my config. It is up to you to evaluate your case. See the references below for more details about the available options for HPKP in addition to the Content-Protection policies and the XSS protection policies, as they may affect your site when you want to load media/material external to your website.

Auto-Renewing The Certificate

LetsEncrypt issues certificates valid for 90 days only to combat spam and fraudulent uses of domains that have been neglected. That means the certificate needs to be renewed before 90 days expire.

As the user "letsencrypt" put the following in a shell script letsencrypt_renew.sh:

#!/bin/bash
python acme_tiny.py --account-key /home/letsencrypt/account.key --csr /home/letsencrypt/subdomain.csr --acme-dir /var/www/challenge/ > /var/www/challenge/subdomain.crt || exit

wget -O /var/www/challenge/lets-encrypt-x1-cross-signed.pem https://letsencrypt.org/certs/lets-encrypt-x1-cross-signed.pem

cat /var/www/challenge/subdomain.crt /var/www/challenge/lets-encrypt-x1-cross-signed.pem > /var/www/challenge/subdomain_chained.crt

Make sure every command is complete and on its own line. The styling here could break them into multiple lines.

This should be run as a cron job so set the permissions:

chmod 744 letsencrypt_renew.sh

Now run # crontab -e and add this line:

# LetsEncrypt cert renewal -- nginx will be reloaded by root in another cron job
1 1 27 * * test $(($(date +\%m)\%2)) -eq 0 && /home/letsencrypt/letsencrypt_renew.sh

This will run the job every 2 months on the 27th day at 01:01 AM (even months of the year). Basically, every 60 days.

Now exit the user "letsencrypt" and as root, run # crontab -e and add this line:

# m h  dom mon dow   command
2 1 27 * * test $(($(date +\%m)\%2)) -eq 0 && `/usr/sbin/service nginx reload`

This will reload nginx at 01:02 AM, a minute after the certificate has been refreshed by the previous job.

Test Site Security and Settings

Now go to SSL Labs and test your website (https://subdomain.domain.com)

Then go to Security Headers and test your website (https://subdomain.domain.com)

References

I highly recommend visiting the sites below from bottom to top. I added them last to first in the order of pages I had on my tabs.