Hyperthreading Mitigation Security Warnings

13 hours ago VMware issued critical security patches for VMware vCenter, ESXi, Fusion and Workstation products as part of advisory VMSA-2018-0020 to fix the new CPU vulnerabilities Intel disclosed as well.

After applying the patches (Aug 14, 2018), a warning message showed on patched ESXi hosts: esx.problem.hyperthreading.unmitigated

According to the release notes, VMware introduced a new Advanced Configuration on the hosts to mitigate the new hyperthreading attacks, however, it states there's a performance hit that cannot be ignored.

After applying the patches, you have to manually enable the Hyperthreading mitigation setting in the advanced functions to enable the security fix, otherwise the exclamation mark on the host and the warning above will persist. It's set to manual modification due to the performance impact.

Change the value of "VMKernel.Boot.hyperthreadingMitigation" to true, then reboot the host for changes to take effect.

Update 1: Aug 15, 2018 - 14:29 UTC+3

After enabling hyperthreading mitigation, some virtual machines that were running HTTPS/443 services weren't accessible anymore. The VM is accessible, but not services on port 443 TCP. After undoing the configuration and rebooting the host, the services functioned again.

Approach this setting and the security vulnerability with caution and do proper testing for every service you have deployed.

PXE Installation of VMware ESXi 4.1

Introduction

Installing ESXi on multiple hosts at the same time over the network is achieved through PXE booting. Unfortunately many of the guides I found online take a long route to setup PXE booting on Linux and install multiple programs, each with its own config file which complicates the matter.

I chose to use DNSmasq because it provides DHCP, DNS, PXE & TFTP services all in one program. In addition, thanks to Simon, he added a feature where you could assign IPs sequentially rather than based on the Mac address. Read here for details.

This mini-guide assumes the use of Linux. If you're a Windows user, I suggest you use 3C Daemon tool from 3Com which offers DHCP, FTP, TFTP & PXE services for Windows.

I have setup a virtual machine dedicated to PXE booting & installation to make it portable & share it with others. Feel free to run your tests on a VM or a physical box.

Requirements

• Linux OS. My choice was Debian.
• VMware ESXi Hypervisor ISO file.
• Internet connection.
• pxelinux.0 file from syslinux version 3.
• Chocolate chip cookies. mmmmm.

Installation & Configuration

0] Install the operating system (Debian) and setup a static IP on the NIC.
1] Edit the file: /etc/network/interfaces -- My editor of choice is nano.

auto lo
iface lo inet loopback

allow-hotplug eth0
iface eth0 inet static
   address 10.172.0.250
   netmask 255.255.255.0
   gateway 10.172.0.254

2] Run the command: service networking restart
Note: In the initial setup, put the IP above to match your network to be able to download then change it once you're done with this guide to the above to avoid conflicts with any network.

3] Install apache and dnsmasq: apt-get install apache2 dnsmasq
4] Edit: /etc/dnsmasq.conf

dhcp-range=10.172.0.1,172.10.0.100,255.255.255.0,infinite
dhcp-option=66,10.172.0.250
dhcp-option=67,"pxelinux.0"
dhcp-boot=/srv/tftp/pxelinux.0
enable-tftp
tftp-root=/srv/tftp

Note 0: The IPs above do not need to match your network.
Note 1: "infinite" is the lease time. The ESXi installer invokes a lease-release token which will cause the IP to be used by another host. I did not want that to happen because I have scripts assigning hosts their IPs sequentially.

5] Create the directory structure: mkdir -p /srv/tftp/pxelinux.cfg

6] Download syslinux v3, extract pxelinux.0 & put it in /srv/tftp: wget <URL>
7] Extract the files: tar -xf <File name>
8] Copy pxelinux.0: cp ./syslinux-3.86/core/pxelinux.0 /srv/tftp/
9] Create PXE boot file: nano /srv/tftp/pxelinux.cfg/default and edit it:

default esxi_scripted
label esxi_scripted
   kernel vmware/esxi411/mboot.c32
   append vmware/esxi411/vmkboot.gz ks=http://10.172.0.250/ks.php --- vmware/esxi411/vmkernel.gz
   --- vmware/esxi411/sys.vgz --- vmware/esxi411/cim.vgz ---
   vmware/esxi411/ienviron.vgz --- vmware/esxi411/install.vgz

prompt 0
timeout 10

Note: Make sure all of the append parameters are on one line. It may pan out here due to little page width.

A] Edit: /var/www/ks.php

accepteula
rootpw password
autopart --firstdisk --overwritevmfs
install url http://10.172.0.250/vmware/esxi411
network --bootproto=dhcp --device=vmnic0
reboot

The above is a kickstart script which the ESXi installer will execute. These are the defaults that are found in the PXE Guide by VMware.
Note: This will install to the first disk detected by the BIOS and will overwrite existing VMFS filesystems.

File Preparation

B] mkdir -p /srv/tftp/vmware/esxi411
C] Copy the contents of the ISO file to the directory above. You can mount an ISO by: mount -o loop /path/to/isofile /mnt. The files will be in /mnt: cp -Rv /mnt/* /srv/tftp/vmware/esxi411/
D] Link to vmware directory: ln -s /srv/tftp/vmware /var/www/vmware
E] service dnsmasq restart

By now, things should be good to go!

Caveats

• If you set this in a VM, keep the NIC disabled by default to avoid wiping systems by mistake and broadcasting DHCP over the LAN.
• If using a VM, the physical adapter must have a static IP not set on DHCP
• To reset the list of leases: echo "" > /var/lib/misc/dnsmasq.leases

This shows a very basic and default setup of installing ESXi over multiple boxes. Hopefully I'll have the time to post my custom scripts that integrate into the kickstart script to auto-assign IPs, VLANs, and a few more tasks to streamline the installation.

DNSmasq Offers Sequential IP Addressing

A few days back Simon, the developer of DNSmasq, added an option for DNSmasq to serve IPs in a sequential manner rather than based on a hash of the MAC address, upon my request.

He was kind enough to implement it, allowing me to use DNSmasq as a DNS, DHCP, PXE & TFTP daemon for my VMware ESXi automated deployments.

This feature is available in version 2.58 test 4. I tested it on a bunch of virtual machines simulating an ESXi installation and everything went smooth.

I have to note: If you're using it for ESXi deployments, you may want to set the lease expiration time to infinite because the ESXi installer sends a lease release after the installation is done, causing subsequent hosts to get the same IP.

Big thanks go to Simon!

VMware ESXi 4.1 on IBM BladeCenter with Nortel Switches

Update: We resolved the issue permanently and I understand what was going on, but have little time to post everything soon. If you're in a hurry and want help, leave a comment or email me.

One of our customers purchased two BladeCenter H Chassis for deploying VMware on them. Each chassis came with two Nortel switches providing 2 internal ports and 9 external ports. The external network ports are: 3x 10Gbit and 6x 1Gbit.

The customer has a physically isolated DMZ network, so one 1Gbit port from each switch was dedicated to a DMZ switch (VLAN1).

The client had purchased only 2x 10Gbit SFPs, so the third port was empty and won't be used in this setup.

To make use of VMware's Virtual Switch Tagging (VST) network concept, the switches which the blades have to be connected to must be configured as a trunk and allow the required VLANs to pass. Also, the Nortel switch on the BladeCenter must be configured to pass those VLANs, for both external and internal ports.

The following VLANs were created:

• Management
• vMotion
• Fault Tolerance (FT)
• Virtual Machines

After configuring the external and internal ports of the Nortel switch to be part of those VLANs, a strange problem popped up: I couldn't ping or reach any of the ESXi hosts in any way, unless I pinged my workstation from within the ESXi server first!

To make matters clear, here's how things were connected:

My workstation -> Server Farm Switch

BladeCenter -> Server Farm Switch

Ping from workstation to any ESXi host: Fails

After 1 ping from an ESXi host to my workstation: Succeeds, and all pings from my workstation to that specific ESXi host go through.

Also, even after traffic is established and I connect using vSphere Client, it disconnects me after about 15 minutes and I can no longer communicate with that host until I ping my workstation from that host again!

After poking around for hours, the solution was to take one external port (1Gbit) out of all VLANs except 1. That is, that port must not belong to any VLAN except VLAN1 (untagged). Doing so, allowed us to communicate with all servers smoothly.

I still don't understand why that worked and whether traffic is now passing through the tagged external ports or that specific untagged port. I'll be doing more investigations next week and update this post.

Converting Windows 2000 with VMware Converter

Converting a Windows 2000 machine has a bunch of caveats and problems when going from physical to virtual (p2v), using VMware vCenter Converter Standalone.

Requirements:

• VMware Standalone Converter version 4.0.1 (See Additional Info at the end)
• Update Rollup 1 for Windows 2000 SP4 (KB891861)
• Windows 2000 Sysprep tools (Q257813)
• A Windows or Linux LiveCD. I recommend Knoppix (6.4+ - Linux) or Hiren (Windows).
  If you need to modify registry keys, use Hiren.

Procedure:

1. Install VMware Standalone Converter version 4.0.1
2. Extract sysprep tools and place them in C:\Documents and Settings\All Users\Application Data\VMware\VMware vCenter Converter Standalone\sysprep\2k
   That should be on the same machine that has VMware Converter, not the Windows 2000 server.
   * On Windows 2008, the location is C:\Users\All Users\VMware\VMware vCenter Converter Standalone\sysprep\2k (Thanks Anonymous for the tip!)
   or C:\ProgramData\VMware\VMware vCenter Converter Standalone\sysprep\2k (thanks Ben!)
3. Either apply the update rollup to the server or extract the update rollup and replace it with the file SCSIPORT.SYS in C:\WINNT\system32\drivers. Applying the update is recommended if the system is stable.
4. If you’re using a static IP on the Windows 2000 server, see this Knowledge Base article.
5. Run the Converter and deploy the agent. If you’re asked to restart, restart then start the VMware Converter service manually before running the Converter again, otherwise it’ll ask you to deploy the agent again.
6. In Step 3: View / Edit Options, Click on the Devices pane and change the disk controller to BusLogic SCSI.
7. Keep the number of processors as is, because if you change it, Windows 2000 won’t auto-detect new CPUs and you’ll need to update the Hardware Abstraction Layer (HAL) on it manually. See KB234558 and KB249694 for more details.
8. In the Networks pane, deselect the option to connect at power on.
9. In the Advanced Options pane, do not select the options to power off the source and select the option to power on the target (VM). Do install VMware tools.
   Do NOT select "configure guest preferences for the virtual machine"

With that, you should be set to convert that machine. After the conversion is complete, the VM will start, install VMware tools, then restart. After it comes up you should apply the proper network settings then shutdown and enable the NIC to connect at power on.

Problems and Solutions:

• "disk read error" when starting the virtual machine.
  This happens because you have selected the Disk Controller as “Preserve Source” or “IDE” -- you must select “SCSI” -- after doing so, you’ll need to reconvert the machine.
• “KMODE_EXCEPTION_NOT_HANDLED” Blue Screen of Death (BSOD) during boot up.
  This happens because Windows 2000 is using the old SCSI driver (SCSIPORT.SYS).
  You must boot into a LiveCD and replace the file in the location mentioned above.
  This happened to me even after I copied the SCSIPORT.SYS to the target machine before converting.
• After installing the Converter agent, you face problems & restart the Windows 2000 server, then when running converter again, it asks you to re-deploy the agent.
  This happens because when the Windows 2000 system comes up again, the Converter agent service isn’t started again.
  Open the services console (services.msc in run) and right-click VMware Converter then choose Start. After the service is started, run VMware Converter and it should connect.
• Unable to communicate to the agent.
  The network traffic is probably blocked by firewalls that are on the Converter machine, the Windows 2000 target machine or in between. Make sure the firewalls are disabled or port 9089 is allowed to pass through.
• "Inaccessible boot device" Blue Screen of Death (BSOD) during boot up.
  This happens due to some misconfiguration of drivers in the registry.
  To fix this, run the Converter program again and do a machine reconfiguration only (don't reconvert). Let it install VMware Tools, select "Reconfigure destination virtual machine" and do NOT select "Configure guest preferences for the virtual machine"
  If that does not solve your problem, read this thread.

Using The Linux LiveCD:

If you’re new to Linux, then here are some steps to help you replace files on virtual machines.

1. Boot the virtual machine from the LiveCD, by either attaching the ISO file from the data store, your machine, or burning the ISO to a CD (as an image!) and booting it from your CD drive.
2. At the boot prompt of Knoppix, just press enter to boot into the graphical interface.
3. Now we need to attach the VM’s disk to the Linux system: open a root shell / terminal.
4. Type: fdisk -l
   This will list all disks in your VM. Identify your operating system hard disk (by capacity if possible). If it’s not possible, then proceed with the next steps until you find your desired partition by looking at its contents.
   You will see things like: /dev/sda, /dev/sda1, /dev/sda2, ...etc. sda is your first hard disk. sdb is your second hard disk. sda1 is the first partition in your first hard disk.
5. Type: mkdir /mnt/os
6. If your operating system (OS) is installed on the first hard disk, first partition, then type: mount /dev/sda1 /mnt/os
7. You can now open a file manager in the graphical interface and go to this directory: /mnt/os -- you’ll see the contents of that partition.
   If that is not your desired partition, skip to step 10 then try mounting another partition.
   
   Note: Make sure you mount a partition & not a disk!
   mount /dev/sda1 is correct. mount /dev/sda is not.
8. To copy a file over the network from a Windows share on another machine, open a file manager and in the address tab type: smb://ip
   Example: smb://192.168.0.1, where the IP is of the machine you want to access over the network to copy a file from.
9. Right click & copy the file, then go to /mnt/os and paste it there.
10. You’re almost done. Now you just need to unmount the partition, so close the file manager window that opens /mnt/os and then in the root shell type: umount /mnt/os
11. Reboot the VM and unattach the CD / ISO.

Additional Info:

• Knoppix is like any *nix system: case-sensitive when it comes to file names. So you may have to delete to the original file manually then copy the new .SYS file due to the difference in letter case.
• The sysprep tools will be used by the VMware Converter to prepare a new copy of Windows. It’s required for the cloning process.
• Support for Windows 2000 has been dropped in VMware Converter version 4.3.
• VMware Converter Standalone is free. VMware requires that you register to be able to download, but their servers are slow (at least in my experience). I got my copy from 4shared, so just search for it & verify the md5 checksum.
  Windows: VMware-converter-4.0.1-161434.exe - 35f22a3b40b114d70cdbda2d5056c10f
  Linux: VMware-converter-4.0.1-161434.tar.gz - 90ce68a9f75af91aed9119d419a98b3c
• LiveCD Selection: You can choose anything that works for you as long as it has SCSI disk drivers, otherwise you won’t be able to see the VM’s disks (which is why getting Damn Small Linux was a waste of time...) and can read & write to the NTFS filesystem.