Techy Title Here: ipv4

Showing posts with label ipv4. Show all posts

Saturday, February 6, 2016

NGINX with High Security Ciphers and LetsEncrypt

I want to move away from the bloated Apache web server and NGINX meets my requirements, but this time I want to use SSL/TLS with signed certificates with the highest security ciphers that support Perfect Forward Secrecy, because why not?

Sadly, the information was scattered and not everything is there in the manuals, so this is a documentation of what I've found and done in my setup.

The Let's Encrypt project provides authenticated and validated domain certificates for free! The catch? They expire every 90 days and their official client requires root access & dependencies, but you can (auto)renew and avoid these. Read on to know more.

Article Updates

Mar 3rd

Corrected root's crontab entry.
Corrected headers' content and location
Added more info about security and privacy headers

Environment

My setup consists of the stuff below. This post will presume Debian & NGINX are already installed. In the steps below, a line starting with "#" means it's a command you should type. Type the command without the "#" character (not necessarily as root).

Debian Jessie (8)

#cat /etc/issue

NGINX version 1.6.2, installed from nginx-full package.

#nginx -v

OpenSSL 1.0.1k

#openssl version

Python 2.7.9

python --version

acme-tiny Dec 29, 2015

If you have an older version of openssl or nginx, you're likely to face problems and failures since new ciphers have been introduced in recent versions of OpenSSL only (1.0.1h) and the same for NGINX's settings. Make sure your distro supports the latest versions, otherwise you'll be leaving yourself and your visitors vulnerable.

Why acme-tiny?

The official letsencrypt client requires installing some dependencies such as gcc (GNU C Compiler) and some other things, in addition to requiring it being run as root, not only once, but as a daemon or in a cronjob as it requires to renew the certificate every 90 days!

As much as I appreciate the Let's Encrypt initiative, I'm not granting their software root access to my machines, nor installing gcc on a production machine. That's where acme-tiny comes in: a small (200 lines) client that is using Let's Encrypt API calls and you can (and should) audit the client's code before using it, since it's only 200 lines of human-readable Python code.

Configuring NGINX for TLS/PFS

SSL is dead. You should be using TLS only, and if you don't have to service old devices (Android 4.x, old IE browsers, Windows XP), then you should be using TLS v1.2 only with a strict set of ciphers.

Perfect Forward Secrecy (PFS) is an old standard but hasn't been widely adopted until after Snowden revealed the amount of encrypted data being stored for later decryption. PFS cycles the encryption key during the session, so even when a session is captured, decryption will be possible only for a small portion as the key changes.

TLS Config

If you're going to configure a wildcard certificate, place the config in /etc/nginx/nginx.conf. Otherwise if the certificate is unique to a specific domain/subdomain, you'll need to place the config in a virtual host config file.

In my case, I started with a wildcard but it self a self-signed certificate and was rejected by browsers, which is normal. Later when I made a Let's Encrypt certificate, I moved it to the specific subdomain.

Note: Let's Encrypt doesn't support wildcard certs as of this writing, however, they allow you up to 100 domains/subdomains.

Edit /etc/nginx/nginx.conf

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
        worker_connections 256;
        multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
 # Change the cache name. Read the manual for more info.
        ssl_session_cache shared:YourSSLCacheNameHere:10m;
        ssl_session_timeout 10m;
        ssl_session_tickets off;
        ssl_stapling on;
        ssl_stapling_verify on;
 # contains CBC AES algs which I do not like
        #ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
 # AES256 GCM is not yet supported on most browsers
        #ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384';
        # TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 & TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 ssl_ciphers "EECDH+AESGCM";
        ssl_ecdh_curve secp384r1;
 # self-generated 4096 DH key range
        ssl_dhparam /etc/nginx/ssl/dhparam.pem;

 # Put IPs of your hosting provider here, or a trusted DNS provider. These are Google's.
 resolver 8.8.8.8 8.8.4.4 [2001:4860:4860::8888] valid=300s;
        resolver_timeout 5s;

        # wildcard cert config should go here, if any
        #ssl_certificate /etc/nginx/ssl/;
        #ssl_certificate_key /etc/nginx/ssl/;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs  
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

Make sure you correct the line breaks if you paste. Due to styling on my blog, you may have one line spilling to multiple lines in the config, and this will break your config.

Don't worry about non-existing directories. We'll come to those later as we finish the setup.

Default Virtual Host Config

I don't use a default domain (www, for example) as mine are hidden from public. If you're like me, then this config fits you, otherwise move to the step below.

Edit /etc/nginx/sites-enabled/default

# Default server configuration
server {
 # change IP to match yours
        listen 127.0.0.1:80 default_server;
 # uncomment to enable IPv6
        #listen [::1]:80 default_server;
 # uncomment to enable ssl on IPv4
        listen 127.0.0.1:443 ssl default_server;
 # uncomment to enable ssl on IPv6
        #listen [::1]:443 ssl default_server;
        server_name _; #default server

        ssl_certificate /etc/nginx/ssl/default_wild.crt;
        ssl_certificate_key /etc/nginx/ssl/default_wild.key;

        root /var/www/html;

        # Add index.php to the list if you are using PHP
        #index index.html index.htm index.nginx-debian.html;
        index index.html;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
                autoindex off;
        }
}

This config will load when someone visits the IP(s) NGINX is configured at.

Virtual Host Config

This is where your subdomain config goes. In my case, the certificate belongs to this specific subdomain, so the certificate lines are added here. If you were using a wildcard cert, you should move them to nginx.conf above.

Create a file for your subdomain /etc/nginx/sites-available/mysubdom

server {
        listen 127.0.0.1:80;
 # uncomment if you want IPv6
        #listen [::1]:80;
        #listen 127.0.0.1:443 ssl;
        #listen [::1]:443 ssl;
        server_name subdomain.domain.com;
        keepalive_timeout 70;

        # The certificate is for subdomain.domain.com only
        #ssl_certificate /var/www/challenge/subdomain_chained.crt;
        #ssl_certificate_key /etc/nginx/ssl/subdomain.key;

        root /var/www/subdomain;

        # Add index.php to the list if you are using PHP
        #index index.html index.htm index.nginx-debian.html;
        index index.html;

        # letsencrypt challenge directory to verify domain
        location /.well-known/acme-challenge/ {
                alias /var/www/challenge/;
                try_files $uri =404;
 }

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
                autoindex off; #enable if you want file listing
        }

}

Notice that listening for SSL/TLS is not yet enabled and the ssl_certificate line and the one below have a hash to comment it. This is required for the initial setup since we'll need to reload nginx and it'll fail since the files are not there yet. We'll enable these lines once everything is done.

To make this config file active by NGINX, you need to link it to sites-enabled:

ln -s /etc/nginx/sites-available/mysubdom /etc/nginx/sites-enabled/mysubdom

Create the directory for your subdomain to serve files:

mkdir /var/www/subdomain
mkdir -p /var/www/challenge/.well-known/acme-challenge
chown -R www-data:www-data /var/www/subdomain
chmod 775 /var/www/subdomain
chmod 771 /var/www/challenge

www-data is the user that NGINX runs as, as shown in the first NGINX config above. Don't worry about the challenge directory owner for now. It'll be taken care of later.

Private Keys and Certificates

The overall config is done. What's left is generating private keys, deriving a certificate for the subdomain, then finally working with Let's Encrypt client.

Create the directory /etc/nginx/ssl to place the subdomain private keys and other things in there:

mkdir /etc/nginx/ssl

Modify its permissions to be restricted to root and only those who know exactly which file to use:

chmod 751 /etc/nginx/ssl

Now inside the ssl directory, generate a 4096 bit Diffie-Hellman parameters file (prime numbers) to act as seeds for the PFS/TLS sessions (this will take a VERY LONG time):

openssl dhparam -out dhparam.pem 4096

Generate a self-signed certificate to be used for the default virtual host (i.e., not the one you care about). This will be served to anyone accessing the IP or any subdomain other than the one you specifically define in the virtual host:

openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -sha512 -keyout /etc/nginx/ssl/default_wild.key -out /etc/nginx/ssl/default_wild.crt

If you don't configure this, users will be served your legitimate certificate and they'll be able to find your "hidden" subdomain. Only do the above if you want your domain/subdomain hidden.

Generate a subdomain private key and a certificate request:

openssl genrsa 4096 > subdomain.key
openssl req -new -sha512 -key subdomain.key -subj "/CN=subdomain.domain.com" > subdomain.csr

This one is the domain/subdomain that will be valid to the world. it can also be "domain.com" if you like.

Let's Encrypt and ACME-Tiny

For security purposes, it's best to have the client run as a separate user. Should anything go wrong in the future, its access would be quite isolated.

Environment Setup

Create a user for it:

useradd -m letsencrypt

Copy the subdomain csr file and set home directory permissions:

chmod 751 /home/letsencrypt
cp /etc/nginx/ssl/subdomain.csr /home/letsencrypt/
chown -R letsencrypt:letsencrypt /home/letsencrypt
chown -R letsencrypt:letsencrypt /var/www/challenge

Now switch user to become the letsencrypt user for the rest of the commands:

su - letsencrypt
openssl genrsa 4096 > account.key
wget https://raw.githubusercontent.com/diafygi/acme-tiny/master/acme_tiny.py
chmod 400 account.key
chmod 400 acme_tiny.py
chmod 400 subdomain.csr

The account.key is your private key to identify you to Let's Encrypt. Keep it safe!:

ls -l
-r-------- 1 letsencrypt letsencrypt 9150 Feb  6 12:13 acme_tiny.py
-r-------- 1 letsencrypt letsencrypt 3247 Feb  6 12:44 private.key
-r-------- 1 letsencrypt letsencrypt 1594 Feb  6 12:38 subdomain.csr

Now exit to be root (or you can use sudo) and restart nginx:

service nginx restart

If there are no errors here, it's all good, otherwise look into /var/log/nginx/error.log for hints.

Script Execution

Now that NGINX is functioning on port 80, it will be used to verify the subdomain ownership. acme-tiny writes to LetsEncrypt.org via APIs and they reply with a random hash that is written to the challenge directory, which is accessible via NGINX on port 80, and then LetsEncrypt.org checks that this hash actually exists at the subdomain you supplied and then verifies you.

su - letsencrypt
python acme_tiny.py --account-key account.key --csr subdomain.csr --acme-dir /var/www/challenge/ > /var/www/challenge/subdomain.crt

All should go OK without errors. If any, verify directory paths and file and directory permissions. Make sure the username "letsencrypt" has access to the files private.key, subdomain.csr and the challenge directory.

NGINX requires concatenating the intermediate certificate to the freshly signed certificate from Let's Encrypt:

wget -O /var/www/challenge/lets-encrypt-x1-cross-signed.pem https://letsencrypt.org/certs/lets-encrypt-x1-cross-signed.pem
cat /var/www/challenge/subdomain.crt /var/www/challenge/lets-encrypt-x1-cross-signed.pem > /var/www/challenge/subdomain_chained.crt

That's it! It should now work after enabling the SSL/TLS settings in NGINX.

Enable TLS in NGINX

Modify the file /etc/nginx/sites-enabled/mysubdom to make it look like this:

server {
        listen 127.0.0.1:80;
 # uncomment if you want IPv6
        #listen [::1]:80;
        server_name subdomain.domain.com;

 # force all traffic to go to HTTPS instead of HTTP
        return 301 https://subdomain.domain.com$request_uri;
}

server {
        listen 127.0.0.1:443 ssl;
        #listen [::1]:443 ssl;
        server_name subdomain.domain.com;
        keepalive_timeout 70;

        # The certificate is for subdomain.domain.com only
        ssl_certificate /var/www/challenge/subdomain_chained.crt;
        ssl_certificate_key /etc/nginx/ssl/subdomain.key;

        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
        add_header X-Frame-Options DENY; #or "SAMEORIGIN" always;
        add_header X-Content-Type-Options nosniff;
        add_header Content-Security-Policy 'default-src https://subdomain.domain.com:443';
        add_header X-Xss-Protection '1; mode=block';


        root /var/www/subdomain;

        # Add index.php to the list if you are using PHP
        #index index.html index.htm index.nginx-debian.html;
        index index.html;

        # letsencrypt challenge directory to verify domain
        location /.well-known/acme-challenge/ {
                alias /var/www/challenge/;
                try_files $uri =404;
 }

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
                autoindex off; #enable if you want file listing
        }

}

Notice how listening on port 80 (HTTP) has been shifted to its own segment while the rest uses HTTPS exclusively. Future certificate renewals can also go over HTTPS as long as the certificate is still valid. If not, revert the config to be as it was at the beginning.

Reload NGINX to read the certificates and make the settings active:

service nginx reload

Note: Reload reads the settings again without dropping connections. It's advised for live websites.

About Headers

Previously, I had the security headers in the main nginx.conf file, but that will apply the same headers to all websites, and that's not scalable nor correct. According to Igor Sysoev (NGINX's creator), he created the config in NGINX to not inherit so that troubleshooting becomes simpler. Duplicating code is good because it makes life easy in finding the problem when things go wrong. See the link for his talk below in the references.

This means that headers (and other configs) should be repeated for every virtual host you configure. If you configure a header in the main block in nginx.conf then define another (or modified) header in the subdomain block, the latter will take over and the first one will be ignored.

About Security Headers

The SecurityHeaders service recommends using HTTP Public-Key Pinning (Stapling) or HPKP for short, but there are privacy and performance concerns with that: Pinning means the public key of your own certificate is sent in the header and is sent to your certificate issuer to validate it. This prevents a Man-in-the-Middle attack, but exposes your visit(s) to the certificate issuer! Additionally, it puts a huge burden on the certificate issuer to scale their own performance to reply to every single site visit. If they don't (and why should they?), your site visiting experience will suffer great delays.

The headers also tell the browser to cache your public keys for a very long period (3+ months) to protect you against forged certificates that could come during that period, but since we're using Let's Encrypt certificates which expire every 3 months, it'll become hectic to manage the headers, aging and other aspects.

With all these concerns, I decided against adding Public-Key Pinning headers in my config. It is up to you to evaluate your case. See the references below for more details about the available options for HPKP in addition to the Content-Protection policies and the XSS protection policies, as they may affect your site when you want to load media/material external to your website.

Auto-Renewing The Certificate

LetsEncrypt issues certificates valid for 90 days only to combat spam and fraudulent uses of domains that have been neglected. That means the certificate needs to be renewed before 90 days expire.

As the user "letsencrypt" put the following in a shell script letsencrypt_renew.sh:

#!/bin/bash
python acme_tiny.py --account-key /home/letsencrypt/account.key --csr /home/letsencrypt/subdomain.csr --acme-dir /var/www/challenge/ > /var/www/challenge/subdomain.crt || exit

wget -O /var/www/challenge/lets-encrypt-x1-cross-signed.pem https://letsencrypt.org/certs/lets-encrypt-x1-cross-signed.pem

cat /var/www/challenge/subdomain.crt /var/www/challenge/lets-encrypt-x1-cross-signed.pem > /var/www/challenge/subdomain_chained.crt

Make sure every command is complete and on its own line. The styling here could break them into multiple lines.

This should be run as a cron job so set the permissions:

chmod 744 letsencrypt_renew.sh

Now run # crontab -e and add this line:

# LetsEncrypt cert renewal -- nginx will be reloaded by root in another cron job
1 1 27 * * test $(($(date +\%m)\%2)) -eq 0 && /home/letsencrypt/letsencrypt_renew.sh

This will run the job every 2 months on the 27th day at 01:01 AM (even months of the year). Basically, every 60 days.

Now exit the user "letsencrypt" and as root, run # crontab -e and add this line:

# m h  dom mon dow   command
2 1 27 * * test $(($(date +\%m)\%2)) -eq 0 && `/usr/sbin/service nginx reload`

This will reload nginx at 01:02 AM, a minute after the certificate has been refreshed by the previous job.

Test Site Security and Settings

Now go to SSL Labs and test your website (https://subdomain.domain.com)

Then go to Security Headers and test your website (https://subdomain.domain.com)

References

I highly recommend visiting the sites below from bottom to top. I added them last to first in the order of pages I had on my tabs.

Saturday, August 17, 2013

Configuring FCoE on IBM Flex nodes and V7000 Storage

Disclaimer

I work for an IBM partner. The opinions depicted in this post are solely mine. Any performance degradation, bug, problem mentioned here is generic to any vendor, unless otherwise strictly specified.

Article Revisions

v1.0 - August 17 (17082013) - Initial release
v1.1 - August 17 (17082013) - Small additions to CN4093 section
v1.2 - October 11 (11102013) - Correction to FCoE frames and Ethernet frames (Thanks Anonymous!)

What is FCoE?

FCoE is short for Fiber Channel over Ethernet. It's an encapsulation of FC packets inside Ethernet packets, allowing a server/node to communicate with a storage system through standard Ethernet, instead of investing in dedicated FC infrastructure.

The idea is to combine, or converge as the industry likes to call it, multiple protocols into a single technology, to reduce the datacenter clutter. With Ethernet, you can now transmit data packets (Ethernet), iSCSI (storage protocol) and FCoE (storage protocol).

Issues with FCoE

No Real Optimization

A payload is the data carried by the protocol from one point to another. The typical Ethernet frame payload is 832 bytes, while FC frames can carry 2kB, and iSCSI fits perfectly into Ethernet's packets.
Standard Ethernet supports an increased payload size with something called Jumbo Frames up to 9kB. IPv6 allows a maximum payload size of 4GB (minus 1 byte) but that requires modification of the Transport Layer to allow TCP/UDP to carry larger payloads, and is no longer done on the Ethernet frame.

FCoE runs on Ethernet frames limited to or a max of 9kB when Jumbo Frames are used. However, iSCSI was designed for Ethernet and runs on the Internet Protocol (IP) on top of Ethernet, which allows it to make use of IPv6 Jumbo Frames' large payload size.

So, if you're going with an overhead of protocols already, you might as well go with iSCSI on IPv6 (assuming the storage supports it) and enable Jumbo Frames (assuming the network backend supports very large Jumbo Frames), instead of going with FC over Ethernet!

Overhead and Replication

iSCSI has been in the business for a long time (6+ years) and works well with the standard Ethernet payload size, but better with jumbo frames. Many storage systems offer iSCSI and have been offering it for a long time. It also doesn't require any special protocols and it "just works" including across data-centers, as long as the link the stable (but there's latency, obviously).

FCoE is relatively new, and requires a heap of protocols to maintain a main "feature": Being Lossless. All this protocol overhead means extra latency, and from what I've been reading, it's not yet possible to push FCoE between datacenters. This means that whenever you need data replication across datacenters, you'll need to attach your storage box into an FC-only infrastructure, which means investing in an FC infrastructure! (OK, maybe just 2 switches, but they still cost money!)

Note: This IBM document mentions that it is possible to replicate between different V7000 storage systems via FCoE only, but the article above puts the limitation on the router/networking end, not on the storage end. Also, that post is 3 years old, so things might have changed now. Approach with caution, anyway, and validate with your vendors.

Another overhead is the encapsulation encoding and decoding process. Disks speak SCSI protocol, and what FC packets do, is put the SCSI commands and their data inside an FC packet, then send it over to the storage, which will strip out the FC packet, then execute the SCSI commands and data.

With FCoE, the server is inserting a SCSI payload inside an FC payload and that is inserted into an Ethernet payload!

Multi-Protocol Failure

Currently, FCoE is enabled on Converged Network Adapters (CNAs) that offer standard Ethernet functions (normal network access) + iSCSI + FCoE. When enabling FCoE, the CNA presents to the Operating System (OS) a bunch of storage adapters of type FC.

What happens when you have a failure on the adapter? You lose both network access and storage access. What happens if the FCoE switch fails? You lose both network access and storage access.

A related scenario would be storage upgrades where one path needs to be offline to move the equipment from old stuff to new stuff, this means affecting both network and storage. One more scenario is your usual network administrator mistake where the spanning tree configuration goes wrong, adds a new VLAN, or plugs a cable into a non-configured switch and the network goes into a loop (think of it as a denial of service attack).
While one would think that if the network fails, then why do you need storage access, is completely valid, the issue here is that the sudden loss of storage could also lead to data corruption.

That's why I personally prefer to keep the two activities separate: Network and Storage. It can still be done if you buy separate switches for FCoE and Ethernet, but then where is the "convergence" of your datacenter and its cost reductions?

Port Reservation

I don't know about other vendors, but on the IBM Flex chassis switches, using FCoE requires reserving 2 external ports from the switch (must be Omni Ports), even if you're using a V7000 Flex plugged into the chassis.

The FCoE protocol requires having an FC Forwarder (FCF) even if the traffic is internal to the chassis. These ports have to be reserved and configured in pairs. 2 ports are needed for every storage system to be connected via FCoE.

You do not need to plug SFPs into these reserved ports.

Limited Communications to Storage Systems

FCoE communicates through VLANs on the Ethernet network. Each NIC must belong to one VLAN when talking to an FCoE target. Because of that, a NIC can only talk to one storage system. If you need a node to talk to multiple storage systems, you'll need to assign each pair of NICs to a separate FCoE VLAN belonging to each FCoE storage system.

This limitation is not there for FC infrastructures, as a node's FC adapter registers itself on the FC SAN fabric, and then the administrator zones (groups) each adapter with a storage system, and an adapter can belong be grouped with multiple storage systems, as long as all storage systems use the same FC adapter settings.

The FCoE connectivity limitation can be avoided by virtualizing various storage systems under one storage system, and expose that one system to the nodes. IBM's Storage Volume Controller and its little brother the V7000 can do that.

Administration Role Separation

In large organization, the roles of a network admin and a storage admin are separated. With Network Convergence, who will be responsible for configuring the network switches? Will the admin take responsibility for both network and storage?

Lab Setup

Alright, enough blabbing. Let's get to business. This is the lab setup for this experiment:

IBM Enterprise Flex Chassis
Two x240 nodes (Intel processors)

Windows Server 2012 was preinstalled by a colleagure so I used it for tests
Installed ESXi 5.1 U1 (IBM Customized image) for Boot from SAN tests

One 4-port CN4054 CNA on each node

Firmware: 4.4.180.3
Feature on Demand (FoD) to enable FCoE

V7000 Flex storage (mounted into the chassis)

Firmware: 6.4.1.3

Two CN4093 converged switches

Firmware: 7.5.3
Base license, allowing use of only 2 ports on the 4-port cards

IBM's Flex chassis allows one to contain nodes, Ethernet switches, FC switches, and storage, all into a 10U chassis, and the communication between the components is internal to the chassis at a minimum of 10Gbps. End of shameless plug.

Note0: The 4-port CNA is made by Emulex, and it has the same chipset found on the 2-port LAN on Motherboard (LoM) built into some x240 nodes.

Note1: The firmware levels above are important and you should meet these as a minimum. As of this writing, the storage has newer firmware, but I kept it at this level as it's the minimum required and for testing purposes.

Configuration Overview

Configure x240 nodes and their CNAs

Understanding the CNA
Possible NIC Configurations
Configure FCoE Feature on the NICs
I won't cover OS configuration nor multipathing driver installation
Configure nodes for SAN Boot via FCoE

Configure V7000 Storage
Configure the CN4093 Converged Switches

Sample Configuration
Configuration Explanation

Profit!

If you need help upgrading the firmware of any component, refer to the device's user manual in the device links posted above. I won't cover these here.

Note: Throughout the guide, screenshots and configuration, I have masked the WWPNs and MACs of the devices used in the lab, because I'm paranoid. Deal with it.

1) Configuring x240 nodes and their CNAs

This is easy, but you could lose yourself within the forest of menus, so I have a few screenshots to make you happy. You can either follow the text description, or spoon-feed yourself with my awesome screenshots.

Understanding the CNA

I'll quickly explain how the CNA is going to function, so that you don't get confused when you configure it.

The 4-port 10Gbps CNA and the 2-port LoM, have 4 physical ports, and 2 physical ports respectively. When enabling Multichannel functionality, the CNA automagically splices each physical port into 4 virtual ports (vNICs).

So physical port 1 will have 4 vNICs: A1 = A1.1 + A1.2 + A1.3 + A1.4. Each vNIC can be allocated bandwidth, not exceeding 10Gb, and the total bandwidth of 10Gb is shared among all 4 vNICs, so you cannot over-commit the bandwidth. So, in an OS, you'll see 8 NICs if you have a 2-port LOM, and 16 NICs if you have a 4-port CNA (4 vNICs per physical port).

You can change the bandwidth allocation dynamically from the switch for any port, live. It's up to you how much bandwidth is allocated to the FCoE port. If you have a license to use all 4 ports, I suggest you use Ethernet on the 1st and 2nd NICs, and FCoE on the 3rd and 4th. This way, you'll be able to allocate full 10Gb to FCoE.

Use the NICs in sequence (1+2, 3+4) to make sure Ethernet passes through both CN4093 switches, and FCoE passes through both CN4093 switches. Ports 1 and 3 communicate with switch0 located in Bay1, while ports 2 and 4 communicate with switch1 located in Bay2.

Possible NIC Configurations

Use physical NICs
Use virtual NICs
Use a mix of pNICs and vNICs

Remember that a 2-port LOM will have each of its physical ports connect to 1 switch. port0 to switch0 and port1 to switch1. So, if you have 2 switches only, you have to use option (2): vNICs.

vNICs are mandatory if you want to share Ethernet and FCoE on the same pipe and you want to guarantee bandwidth for FCoE. If you do not use vNICs, FCoE and Ethernet will compete on the bandwidth. If your servers are busy, it may lead to delayed I/Os and performance degradation.

My favorite configuration is if you have a 4-port adapter, and Upgrade1 switch licenses for your 2 switches, then you can use 2 ports as pNICs for FCoE and 2 ports as pNICs for Ethernet. No need for vNIC configuration.

Alternatively, you can also enable vNICs on the first 2 ports, and leave the 3rd and 4th ports as pNICs. Or the opposite. So you can mix, but they'll have to be in pairs.

If you have a 4-port adapter, with the base license of the switches, your options are the same as the LOM, in the first paragraph.

Configure FCoE Feature on the NICs

Power on the node and press F1 to login to the UEFI setup
UEFI main menu -> System Settings -> Network -> Select 1st NIC (PFA 17:0:0 here) -> Emulex 10G NIC
You're now at the Emulex NIC Selection menu
Notice the link speed. It should report a number.
Switch Configuration: Change it to IBM Virtual Fabric -- default: Switch Independent
Personality: Change it to FCoE -- default: NIC
Multichannel: Enable if you want to enable vNICs
Controller Configuration -> View Configuration
The 2nd NIC should report itself as FCoE. Only 1 NIC will have FCoE functions.
Notice that the numbering of the NICs is all even. These NICs belong to switch0 located in Bay1.
Press Esc until you're back at the Emulex NIC Selection menu.
Feature on Demand -> Install FCoE license
You're now done with the first NIC. The 2nd NIC will have the same settings as the 1st. You will need to repeat the steps above for the 3rd NIC, and that NIC's settings will be applied to the 4th.
Press Esc until you're back at the Network menu and select the 2nd NIC.
Notice that the NICs have odd numbers. These are mapped to switch1 located in Bay2.
Esc to the System Settings menu -> Emulex Configuration Utility
If you do not see this option, Esc to UEFI main menu, save, then exit to reboot the node.
Highlight the 1st NIC (001) but don't click on it. Write down the NIC's Port Name and node name in a text file for later use. Highlight the other NICs and write their PNs.
If you don't have an Upgrade1 license for your CN4093 switches, you won't be able to use the 3rd and 4th NICs, so you can ignore them.
Click on the 1st NIC. You're now at the Emulex Adapter Configuration menu.
Configure DCBX Mode: Change it to CEE -- default: CIN
Later on when you're done configuring the storage and the switch, come back here and run Scan for Fiber Devices and you should see the V7000 listed (ID 2145)
Also, scroll down and click on Display Adapter Info sub-menu and you'll see the FCoE VLAN ID, if your switch was configured properly. This is auto-discovered.
Esc to the Emulex Adapter Configuration menu, and select the 2nd NIC (002) then repeat the same steps.
Esc to UEFI main menu, save and reboot back to the Scan for Fiber Devices for later use.

With those easy steps, you have completed ONE node. Repeat the same for all nodes. If you're fortunate enough to have had ordered the Flex System Manager node, then it's your lucky day! You can create a Configuration Template of the configured node, which would capture its hardware component configurations, and deploy its hardware configuration to other nodes. It's magic.

If you intend to use the Configuration Templates, I recommend that you configure all the components (internal disk RAID setup, time, boot order, ...etc.), then create the template out of the node.

UEFI -> System Settings menu

Network -> Select NIC

Click on that to get the juicy settings

Change settings as listed. Multichan is for vNICs

Showing the available options

Click it!

FCoE vNIC is always the 2nd

FCoE requires a license. Install it.

To the next adapter

The 2nd NIC follows the settings of the 1st

Emulex Configuration Utility for FCoE HBA Settings

Select 1st NIC. Note Port Name for FC zoning

Change settings to CEE

After storage and switch config is done, scan fiber devices

Configure nodes for SAN boot via FCoE

Each canister/controller will have 1 port looking at one switch and the other port looking at the other switch, which means on each switch you'll see both controllers.

This guide is specific to V7000 and V7000 Flex and VMware ESXi 5.1 (IBM Customized Image). For other storage types, I highly recommend you read and follow the steps in the "Storage and Network Convergence Using FCoE and iSCSI" redbook (link in references). It explains booting from SAN with FCoE and iSCSI, and has excellent tips.

Configure the FCoE switches and make sure that the storage and nodes are functioning properly.
Create a volume and assign it to the node that will boot from SAN. It must be the first volume assigned to the node (LUN 0/SCSI ID 0).
Boot the node into UEFI -> System Settings menu -> Emulex Configuration Utility
Select the 1st adapter
Set Boot from SAN: Change it to Enable
Validate storage connectivity and volume/LUN assignment: Navigate to Add Boot Device -> Select 1st Controller
If you don't see the storage or LUN 0000, then you need to finish configuring the switches, assign a LUN to the node, then come back here.
Do not select a boot device here. This is only for validation of connectivity.
Configure HBA and Boot Parameters -> Boot Target Scan Method: Select Boot Path Discovered Targets
Commit Changes.
Esc to Adapter Selection menu and select the 2nd adapter, and repeat the same steps.
Configuring FCoE SAN boot should be sufficient on 2 ports.
Esc to System Settings menu -> Devices and I/O Ports -> Enable/Disable Onboard Devices
SAS Controller: Disable to disable booting from local disks on the node. Do this even if you don't have any local disks.
Esc to Devices and I/O Ports -> Device Boot Priority
Drag the SAS Controller to the bottom of the list. Save/Commit.
Esc to Main Menu -> Boot Manager -> Add Boot Option -> Generic Boot Option
Add Hard Disk 0, 1, 2, 3
If you configure 2 FCoE ports, you'll have 4 possible paths to boot from. By selecting 4 disks, the UEFI will configure each path into a Hard Disk, and boot from the first available one.
Esc to Boot Manager -> Delete Boot Option: Delete anything that you don't need (PXE, Floppy)
Esc to main menu -> Save
Reboot and install OS

Note: During adapter preparation phase in UEFI, it'll probe the FCoE ports and see which one is online, and will nominate and use one of them only.

Steps 7 and 15 allow high flexibility and reduce configuration time for implementations that have many nodes. The typical method of implementation is defining the boot LUN and path for each node. So if you have 10 nodes, and 2 FCoE ports, you'd need to repeat those configurations 40 times! Using Boot Discovery and auto Hard Disk assignment by UEFI, you avoid this headache.

It does add some extra time to the boot process, but it's not really important at the advantage of flexibility.

Emulex Configuration Utility

Adapter Selection

Enable Boot from SAN for both adapters

You should be able to see storage and LUNs here

Do not add the LUNs. Just validate connectivity.

Configure HBA and Boot Parameters

Boot Path Discovered Targets

Add Boot Option -> Generic Boot Option

Add Hard Disk 0, 1, 2 and 3 for a total of 4 paths

Devices and I/O Ports

Enable / Disable Onboard Devices

Disable SAS Controller

2) Configure V7000 Storage

If you have purchased the V7000/V7000 Flex with the FCoE daughter cards, there's no configuration for FCoE. If you bought the daughter cards at a later stage, you'll need to activate them from the canisters. This won't be covered here. Please refer to the online manual.

If you login to the V7000's web interface, you'll see each canister's (controller) Port Numbers, for both FC and Ethernet. You'll see these numbers once the switch is configured.

V7000 Flex canister/controller 1

V7000 Flex canister/controller 2

Note that the port type is FC

3) Configure the CN4093 Switches

As mentioned before, a minimum of 2 external Omni ports must be reserved, even if you're using a V7000 Flex inside the same chassis as the nodes.

This switch configuration will assume default bandwidth allocations. I highly advise you to read the CN4093 redbook (link in references) for optimizations.

I'll first write the entire switch config, then explain each section.

Login to the switch in "iscli" mode, then type "enable" to access the enable mode. Now type "config terminal" to be able to modify.

version "7.5.3"
switch-type "IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch"
!
system port EXT15-EXT16 type fc
!
interface port INTA1
name "Flex System Manager node"
no flowcontrol
exit
!
interface port INTA2
name "Power p260 node"
no flowcontrol
exit
!
interface port INTA3
name "x240 node1"
tagging
no flowcontrol
exit
!
interface port INTA4
name "x240 node2"
tagging
no flowcontrol
exit
!
interface port INTA5
tagging
no flowcontrol
exit
!
interface port INTA6
tagging
no flowcontrol
exit
!
interface port INTA7
name "v7000 flex node1"
tagging
pvid 1002
no flowcontrol
exit
!
interface port INTA8
name "v7000 flex node2"
tagging
pvid 1002
no flowcontrol
exit
!
interface port INTA9
tagging
no flowcontrol
exit
!
interface port INTA10
tagging
no flowcontrol
exit
!
interface port INTA11
tagging
no flowcontrol
exit
!
interface port INTA12
tagging
no flowcontrol
exit
!
interface port INTA13
tagging
no flowcontrol
exit
!
interface port INTA14
tagging
no flowcontrol
exit
!
vlan 1
member INTA1-INTA6,INTA9-INTA14,EXT1-EXT2,EXT11-EXT16
no member INTA7-INTA8
!
vlan 1002
enable
name "fcoe"
member INTA3-INTA4,INTA7-INTA8,EXT15-EXT16
fcf enable
!
!
vnic enable
vnic port INTA3 index 1
bandwidth 25
enable
exit
!
vnic port INTA4 index 1
bandwidth 25
enable
exit
!
vnic vnicgroup 1
vlan 3001
enable
member INTA3.1
member INTA4.1
exit
!
spanning-tree stp 80 vlan 3001
!
spanning-tree stp 113 vlan 1002
!
!
!
!
fcoe fips enable
!
fcoe fips port INTA3 fcf-mode off
fcoe fips port INTA4 fcf-mode off
fcoe fips port INTA7 fcf-mode on
fcoe fips port INTA8 fcf-mode on
fcoe fips port EXT15 fcf-mode on
fcoe fips port EXT16 fcf-mode on
!
!
cee enable
!
!
fcalias v7k_node1_p1 wwn 50:00:00:00:00:04:00:76
fcalias v7k_node2_p1 wwn 50:00:00:00:00:04:00:77
fcalias node3 wwn 10:00:00:00:00:00:00:5d
fcalias node4 wwn 10:00:00:00:00:00:00:6b
!
zone name v7k_node3
member fcalias v7k_node1_p1
member fcalias v7k_node2_p1
member fcalias node3
zone name v7k_cluster
member fcalias v7k_node1_p1
member fcalias v7k_node2_p1
zone name v7k_node4
member fcalias node4
member fcalias v7k_node2_p1
member fcalias v7k_node1_p1
zoneset name ActiveConfig
member v7k_node3
member v7k_cluster
member v7k_node4
zoneset activate name ActiveConfig
!
no ip routing
!
!
end

Configuration Explanation

system port EXT15-EXT16 type fc
This changes the type of the Omni ports from being Ethernet ports to FC ports. This is required to bind the ports to a storage system, whether the storage is internal to the chassis or external. If your storage is external, these are the ports where you have to plug the FC SFPs and cables to your external SAN fabric.

interface port INTA1-INTA14
name "port name"
no flowcontrol
tagging
pvid 1002
interface port : defines which ports you want to work on. You can specify 1 port or a range. If you have Upgrade1 license, you can also define INTA1-INTB14 to modify all 28 ports in one shot.

name "port name" : It's better that you do this on a per port basis, to give each port a unique name, to know which system is using that port.

no flowcontrol : Disables traffic flowcontrol. A requirement for FCoE.

tagging : Enable VLAN tagging on a port, allowing that port to belong to multiple VLANs. Do not enable this on ports that will not use FCoE, nor require VLAN tagging. An example to this is a standalone Windows/Linux node.

pvid 1002 : Set the Private VLAN ID (native VLAN) on the port. The default is 1 in all networks. This has to be changed to the VLAN of the FCoE on the V7000 Flex ports. If you do not have a chassis storage, no internal port needs this PVID set.

vlan 1
member INTA1-INTA6,INTA9-INTA14,EXT1-EXT2,EXT11-EXT16
no member INTA7-INTA8
!
vlan 1002
enable
name "fcoe"
member INTA3-INTA4,INTA7-INTA8,EXT15-EXT16
fcf enable
!
These are VLAN definitions, and which ports belong to the VLAN and which don't.
1002 is the preferred VLAN ID for FCoE. You can change this to whatever you want, but make sure the customer network doesn't have the same ID on the Ethernet network to not cause confusion for your nodes.

fcf enable : Enable Fiber Channel Forwarding on this VLAN. This is a must on the FCoE VLANs if you have a V7000 Flex or an upstream (Top of Rack) switch that understands FCoE. If you're connecting the chassis to a SAN fabric, you need to enable NPV mode. See the CN4093 redbook for details.

vnic enable
vnic port INTA3 index 1
bandwidth 25
enable
exit
vnic enable : This is only needed if you need vNICs and want to enable it.

vnic port index 1 : This is vNIC1 of the internal physical port 3. In other words, it's INTA3.1.
You only need to set this, if you want to use this specific vNIC. If you do not set these settings, it'll appear as disconnected on the OS.

bandwidth 25 : Allocate 25% of the 10Gb bandwidth, which is 2.5 Gbps to this vNIC.

Note: You do not allocate bandwidth nor define a vNIC index for the FCoE port.

vnic vnicgroup 1
vlan 3001
enable
member INTA3.1
member INTA4.1
exit
vnic vnicgroup : Create a vNIC Group to add members to it. This is a must for vNIC configurations. Not required for non-vNIC setup.
The group members can be vNICs, internal physical ports, and external ports. In the example above, only internal ports were added. No external ports were configured.

vlan 3001 : Each vNIC Group requires its own VLAN, and this must not be an existing VLAN. This is only for internal communication, and will not conflict with the customer side VLANs.

vNICs not added to a vNIC Group, will appear as disconnected.

spanning-tree stp 80 vlan 3001
If spanning tree is enabled, this will place the VLAN 3001 in its own Spanning Tree Group number 80. The firmware will by default assign each VLAN into its own STG without having to do this manually.

fcoe fips enable
!
fcoe fips port INTA3 fcf-mode off
fcoe fips port INTA4 fcf-mode off
fcoe fips port INTA7 fcf-mode on
fcoe fips port INTA8 fcf-mode on
fcoe fips port EXT15 fcf-mode on
fcoe fips port EXT16 fcf-mode on
!
cee enable

Enable fcoe initialization protocol snooping, which will detect which ports support FCoE and which don't.

fcf-mode off/on/auto : It should be OFF for the internal ports of the compute nodes, and on for the storage and FC ports. You can also avoid messing things, and set this to auto on all ports.

cee enable : Enable Converged Enhanced Ethernet to allow FC packet encapsulation over Ethernet.

fcalias

Define an alias to make it easy to identify nodes and storage ports.

no fcalias wwn

To remove an already configured fcalias.

zone name
Create a zone and add aliases to this zone.

zoneset name
zoneset activate name
Create a zoneset, which is a group of zones to enable this set for the entire switch.

no ip routing
Disable Layer3 routing, and make the switch a Layer2 switch only.

show fcoe database
-----------------------------------------------------------------------
VLAN FCID WWN MAC Port
-----------------------------------------------------------------------
1002 011000 50:00:00:00:00:04:00:77 0e:fc:00:01:10:00 INTA8
1002 011100 50:00:00:00:00:04:00:76 0e:fc:00:01:11:00 INTA7
1002 011101 10:00:00:00:00:00:00:5d 0e:fc:00:01:11:01 INTA3

Total number of entries = 3

-----------------------------------------------------------------------
Displays the currently established FCoE connections on the switch. It doesn't show any node-storage associations. It shows the nodes/storage that has been detected to have FCoE. The section in orange is a sample output.

show zone
List the configured zones on the switch.

For details and explanations of each command, or extra details, do read the CN4093 redbook (linked below in the references).

Note: The above configuration should be the same for the 2nd CN4093 switch, except for the FCalias parts as the WWPNs will be different.