Wednesday, August 15, 2018

Hyperthreading Mitigation Security Warnings

13 hours ago VMware issued critical security patches for VMware vCenter, ESXi, Fusion and Workstation products as part of advisory VMSA-2018-0020 to fix the new CPU vulnerabilities Intel disclosed as well.

After applying the patches (Aug 14, 2018), a warning message showed on patched ESXi hosts: esx.problem.hyperthreading.unmitigated

host summary showing hyperthreading unmitigated error

According to the release notes, VMware introduced a new Advanced Configuration on the hosts to mitigate the new hyperthreading attacks, however, it states there's a performance hit that cannot be ignored.

After applying the patches, you have to manually enable the Hyperthreading mitigation setting in the advanced functions to enable the security fix, otherwise the exclamation mark on the host and the warning above will persist. It's set to manual modification due to the performance impact.

hyperthreading mitigation advanced setting

Change the value of "VMKernel.Boot.hyperthreadingMitigation" to true, then reboot the host for changes to take effect.

Update 1: Aug 15, 2018 - 14:29 UTC+3

After enabling hyperthreading mitigation, some virtual machines that were running HTTPS/443 services weren't accessible anymore. The VM is accessible, but not services on port 443 TCP. After undoing the configuration and rebooting the host, the services functioned again.

Approach this setting and the security vulnerability with caution and do proper testing for every service you have deployed.

3 comments:

Anonymous said...

Same boat here, thank you for the heads up on HTTPS/SSL, we're going to hold off on additional patches on our cluster until there's another update to fix whatever is broken. I heard somewhere else that disabling hyperthreading can lead to an 80% decrease in performance? If that's the case Intel's really screwed the pooch.

Anonymous said...

Performance hit is 30% for enabling the new VMware scheduler.

MBH said...

Re: SSL
Yeah I found it to be very fishy. If I'm to guess, it could be related to the patch blocking Intel's AES encryption extension(s).

As for performance hit, if you disable HT altogether, you lose half your "logical" cores. So 80% sounds correct, but this patch doesn't disable HT. Instead, it forces the VMs to use different cores to not share the data. I think I saw more technical details in the links I posted, so poke around there if you need details on the patch/mitigation implementation.

Re: Performance Hit
Any idea if VMware officially published that? The articles they posted said 'not trivial' without specifying numbers.