16 Gb Brocade SAN Fabric Merge

Introduction

A customer with an existing setup from HP with HP-branded Brocade switches wanted to connect those switches to the newly acquired IBM setup (also using Brocade switches). The HP switches are the 24-port 8 Gb switches, and the IBM ones are 48-port 16 Gb switches. The final goal is to virtualize the HP storage behind the V7000 storage, but this will not be discussed in this post.

The HP SAN switches had existing configurations & were in production. The IBM switches also had configurations for an ongoing implementation.

To merge the SAN fabrics, there are 2 ways:

1. Wipe one of them (clear the config), disable it, then enable it. The config of the other switch will be written to this empty one.
2. Merge 2 different fabrics without wiping any data.

This post will address point (2), because I didn't want to re-do all the zoning from scratch. That's a waste of time. The steps will be done in command line (CLI), because I hate java.

Why Write This Post?

I was reading Brocade's forums and many were talking about using fabric merge tools and that the two fabrics must have different names, and there was a lot of wrong or outdated information that no longer applies to the new Fabric OS 7.x (new switch firmware).

Status

1. HP switches had Fabric OS (FOS) 7.1.
2. IBM switches had FOS 7.4.
3. HP switches had full fabric license.
4. IBM 48-port switches include "Full Fabric" license by default, but doesn't show with "licenseshow" command. It's bundled & enabled by default.
5. HP switches had domain ID: 11 & 12.
6. IBM switches had default domain ID: 1.
7. Switch configuration name on HP was different from the one on IBM.
8. IBM switch 1 connected to HP switch 1 using 1 FC cable. switch 2 connected to switch 2 using 1 FC cable.
9. IBM switches had 16 Gb SFPs. HP had 8 Gb SFPs. Speed of IBM SFP used for SAN connection was fixed to 8 Gb (no auto negotiate).

Requirements

1. Fabric OS has to be 6.x or 7.x on all switches connecting to each other. The minor version ".x" does not have to match, but it's recommended to keep the switches on the same level, if possible.
2. Full Fabric license must be available on 24-port switches. It's available by default on 48-port switches.
3. Change Domain ID from default value to a unique value. The 2 switches connecting to each other must have different Domain IDs.
4. Switch configuration names must be the same for the fabric to merge. If they are different, "Zone Conflict" error will show on the secondary switch.
5. If you have a lot of traffic going from one switch to another switch, it's advised to purchase the "Trunking License" to allow aggregating multiple FC ports/links together.
6. Aliases and zone names must be unique before merging the fabric. If you have similar alias names on the 2 different switches, you have to rename the aliases/zones on the secondary switch (the one that you can disable to merge the fabric).
7. Aliases that have the same WWN on both secondary and primary switches, must have the same name on both fabrics. This is a very unique case, but possible if you're virtualizing the WWNs of your servers.
8. Make sure switch date, timezone & time are all correct before you merge the switches. Changing the timezone requires a switch restart, so plan for the downtime.
9. Default user is 'admin' and default password is 'password'.
10. Do not connect any FC cables between the HP/IBM (different switches) until you're told to do so. Follow the steps exactly as shown below.

Steps

In the steps below, a line starting with "#" means it's a command you should type. Type the command without the "#" character.

Some steps will require rebooting the switch. Some will require disabling the switch more than one time, which makes it offline, and stops all storage access traffic. It's better to change the paths from the servers to the 2nd switch manually, or if you're sure the multipath drivers are working properly, you can disable server ports.

The primary switch is the one that will remain operational. The secondary switch is the one where we are making all these changes & can afford downtime.

Disable Ports

It's better to disable server ports, to prevent multipath driver from using the paths again when they're online, but before you finish your activity. Do this on ONE switch only! After you successfully merge fabrics on this switch, enable ports, then move to the 2nd switch. Do NOT disable ports on both switches at the same time, if you have active servers connected to the SAN switches.

1. List available ports and WWNs: # switchshow
2. # portdisable <port number>
   Example: # portdisable 15
   This will disable the 16th port (port numbering starts from zero)

Repeat this for all ports.

Change the Timezone

1. # date
   This will show current time, date & timezone. Example: Tue Jan 12 09:00:03 AST 2016. AST = Arab Standard Time timezone.
2. # tstimezone --interactive
3. Follow the prompts. Choose the continent, then the country.
4. After finishing, a message will say: "System Time Zone change will take effect at next reboot"
5. If time is not correct, change it before you reboot. See the steps below.
   If the time is correct, you can now reboot the switch: # reboot

Change the Time and Date

1. date [MMDDhhmm[[CC]YY]]
   MM = Month = 01, 02, ..., 12
   DD = Day = 01, 02, ..., 31
   hh = Hour = 00, 01, 02, ..., 23
   mm = Minute = 00, 01, 02, ..., 59
   CC = First two digits of the year = 20 for 2016
   YY = Last two digits of the year = 16 for 2016
2. To change the time & date to Jan 23 2016 21:43:00 (9:43 PM)
   # date 012321432016
3. Time change does not require a reboot. If you changed the timezone, you should reboot now.

Display Current Domain ID

1. # switchshow
2. Top of the output will show a line: switchDomain: 1
   1 is the default value.

Change Domain ID

1. To change the Domain ID of a switch, the switch must be disabled first:
   # switchdisable
   This will take the switch offline and stop all traffic.
2. Start the configuration process to change switch parameters:
   # configure
3. Fabric parameters (yes, y, no, n): [no] yes
   Domain: (1..239) [1] <Unique ID must be different from the switch you will connect to>
4. Press Enter for all other parameters to use default values. No need to change any of them.
5. # switchenable

Rename Zone Configuration

You should rename the zone config to match the primary switch. The primary switch is the one that will remain operational. The secondary switch is the one where we are making all these changes.

1. # cfgshow
2. This will print current aliases, zones and zone config information. At the top, you'll see the config name:
   Defined configuration:
    cfg: HO_SANSW1_Top
3. The config must be disabled before you can rename it: # cfgdisable
4. Now, rename the config to be the same as the primary switch: # zoneobjectrename <current name>, <new name>
   Example: # zoneobjectrename HO_SANSW1_Top, Production_SAN1
5. Remember, both primary (HP switch in my case) and secondary (IBM in my case) must have the same config name to be able to merge the fabrics.
6. Save the new config changes: # cfgsave
7. Run the command again to see the new config name: # cfgshow
8. Now activate the config: # cfgenable <config name>

Change Port Speed

All ports are disabled. We need to change the speed of the port to make it fixed instead of using auto negotiate. This must be done on both primary and secondary switches.

1. # portcfgspeed <port number> <speed>
   Example: # portcfgspeed 35 16
   This will fix the speed of port 35 to 16 Gbps. Auto negotiation will be disabled.
2. Do this on the port that will connect each primary SAN switch to each secondary SAN switch.
3. Keep the port disabled on the secondary switch.
4. Enable the port on the primary switch: # portenable <port number>
5. Connect your Fiber Channel cables into the ports.

Merging The Fabrics

1. First, save the current zone names of the secondary switch in a text file. We will need them after this step: # cfgshow
   Copy the output and save it in a text/word file.
2. On the secondary switch, disable the config: # cfgdisable
3. Now enable the port connecting the secondary & primary switches: # portenable 35
4. Wait 10-30 seconds before proceeding to give enough time for the link to establish and the 2 switches to talk.
5. Disable the secondary switch to make it the slave and to add the config from the primary:
   # switchdisable
6. Enable the secondary switch: # switchenable
7. Wait 10-50 seconds, then check the switch: # switchshow
   You should see in the line of the port connecting the switches something like this:
   35 35 1f2300 id 8G Online FC E-Port 10:00:00:xx:xx:xx:xx:xx "" (upstream)
8. Wait some time and the name of the primary switch will appear between the double quotes.
9. You should also see both switches in the same fabric now: # fabricshowThis should show the names of the primary & secondary switches.
10. If you type # cfgshow it will show all zones and aliases from both switches, but only those from the primary are in the active config.

Enabling Zones of Secondary Switch

The fabrics are now merged, but the zones of the secondary switch are not in the active config yet. We need to add them to the config and enable the config.

1. Open the text file of the zone names (cfgshow output) from the previous step.
2. To add the zones, type the command: # cfgadd "<zone name>", "zone1; zone2; zone3"
   Notice it's a semicolon between the zone names. You can add multiple zones at the same time to the active config.
   If you're lazy and java works for you, you can use the graphical interface to select the zones and add them to the config.
3. When done, type: # cfgsave
   press "y" to save it.
   Then type: # cfgenable <config name>

Congratulations! Now all zones are active from both switches. The ports are still disabled, though, so let's enable them.

Enable Ports

1. List available ports and WWNs: # switchshow
2. # portenable
   Example: # portdisable 0
   This will enable the 1st port (port numbering starts from zero)
3. Repeat this for all ports.
4. You can now check your servers and storage and all links should be operational.

Congratulations! You're now done with the first switch connectivity. Make sure your links are stable, then move on to the remaining switches.

Errors

Zone Conflicts and Segmentation

For some reason, the switch showed "segmented" and "zone conflict" messages and upon a reboot, all ports were disabled. Trying to enable a specific port gave the error: "Port 35: Port enable failed due to unknown system error"

I rebooted the SAN switch again and the ports (and switch) became online again. Looks like it froze at some point and needed another reboot. If this happens often, upgrade the FOS to latest stable version. For me, it only happened once.

If you still get "zone conflict" after finishing all the steps, then you have an alias with the same WWN but different names. To fix it, rename the alias using the "zoneobjectrename" command as shown above.

Unstable Ports

I was unlucky to have the ports being unstable. The link kept going online & offline, flapping many times and sometimes it connects at 16 Gbps and sometimes at 8 Gbps (before I fixed the speed to 8 Gbps). Also, it prevented the switches from creating a fabric connection.

First clear the stats to not carry any old data: # portstatsclear <port number>, then you can check your port statistics by issuing the command: # portshow <port number>

In the output, if you have very large numbers in any of these parameters:

• Unknown
• Parity_err
• 2_parity_err
• Link failure
• Loss_of_sync
• Loss_of_sig
• Invalid_word
• Invalid_crc

In my case, I had to change 2 SFPs, one on the old HP SAN switch and one on the new IBM SAN switch. I also had to change the port slot on the old HP switch because the port slot itself had problems. I'm glad the FC cable was good.

References

• Implementing IBM b-Type SAN with 8 Gbps Directors and Switches: http://www.redbooks.ibm.com/abstracts/sg246116.html?Open
  Fabric merge is section 13.2 - page 636 (pdf) / 608 (redbook)