Update: We resolved the issue permanently and I understand what was going on, but have little time to post everything soon. If you're in a hurry and want help, leave a comment or email me.
One of our customers purchased two BladeCenter H Chassis for deploying VMware on them. Each chassis came with two Nortel switches providing 2 internal ports and 9 external ports. The external network ports are: 3x 10Gbit and 6x 1Gbit.
The customer has a physically isolated DMZ network, so one 1Gbit port from each switch was dedicated to a DMZ switch (VLAN1).
The client had purchased only 2x 10Gbit SFPs, so the third port was empty and won't be used in this setup.
To make use of VMware's Virtual Switch Tagging (VST) network concept, the switches which the blades have to be connected to must be configured as a trunk and allow the required VLANs to pass. Also, the Nortel switch on the BladeCenter must be configured to pass those VLANs, for both external and internal ports.
The following VLANs were created:
One of our customers purchased two BladeCenter H Chassis for deploying VMware on them. Each chassis came with two Nortel switches providing 2 internal ports and 9 external ports. The external network ports are: 3x 10Gbit and 6x 1Gbit.
The customer has a physically isolated DMZ network, so one 1Gbit port from each switch was dedicated to a DMZ switch (VLAN1).
The client had purchased only 2x 10Gbit SFPs, so the third port was empty and won't be used in this setup.
To make use of VMware's Virtual Switch Tagging (VST) network concept, the switches which the blades have to be connected to must be configured as a trunk and allow the required VLANs to pass. Also, the Nortel switch on the BladeCenter must be configured to pass those VLANs, for both external and internal ports.
The following VLANs were created:
- Management
- vMotion
- Fault Tolerance (FT)
- Virtual Machines
After configuring the external and internal ports of the Nortel switch to be part of those VLANs, a strange problem popped up: I couldn't ping or reach any of the ESXi hosts in any way, unless I pinged my workstation from within the ESXi server first!
To make matters clear, here's how things were connected:
My workstation -> Server Farm Switch
BladeCenter -> Server Farm Switch
Ping from workstation to any ESXi host: Fails
After 1 ping from an ESXi host to my workstation: Succeeds, and all pings from my workstation to that specific ESXi host go through.
Also, even after traffic is established and I connect using vSphere Client, it disconnects me after about 15 minutes and I can no longer communicate with that host until I ping my workstation from that host again!
After poking around for hours, the solution was to take one external port (1Gbit) out of all VLANs except 1. That is, that port must not belong to any VLAN except VLAN1 (untagged). Doing so, allowed us to communicate with all servers smoothly.
I still don't understand why that worked and whether traffic is now passing through the tagged external ports or that specific untagged port. I'll be doing more investigations next week and update this post.
