After applying the latest set of Windows Updates to some of our EC2 instances, we saw some problems, including very high CPU utilisation by lsass.exe. The strange thing being, it only happened when that instance was listed in an EC2 Load Balancer's instance list - even if there was no traffic hitting that load balancer.
I spent some time this evening tracking it down, and it looks like there's a serious interaction between the patch for the Microsoft SChannel vulnerability and EC2's load balancer.
Here's the reproduction steps:
With that setup, run Wireshark on the instance. As soon as the instance is in service on the load balancer, you will see several Mbps of HTTPS handshake traffic between the two. Remove the instance from the load balancer, and after a short period, the traffic stops.
Now, uninstall KB2992611, reboot, add the instance back into the load balancer, and observe everything behaves as expected - only the occasional health checks and one or two handshakes.
By way of comparison, 10s of Wirecast capture with KB2992611 installed resulted in a 20MB capture file; 90s without it installed was only 53kB.
If you then manually download and reinstall KB2992611, the problem recurs, with very high traffic levels.
If the instance is not behind an EC2 Load Balancer, direct HTTPS access from Chrome appears to work fine, with no abnormal traffic observed.
My guess is it's a cipher negotiation issue, but at this point, my knowledge of TLS is inadequate to say any more...
Update: disabling TLS 1.2 works around the problem.
If you're affected, set the following registry key to disable TLS 1.2. This causes TLS 1.1 to be negotiated, which is successful:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Server]
"Enabled"=dword:00000000
It's probably better just to disable the affected ciphers though, as [now - 2014-11-17] recommended by Microsoft.
Update 2, 2014-11-12: Amazon have acknowledged the issue, and are on the case: http://aws.amazon.com/security/security-bulletins/ms14-066-advisory/
Update 3, 2014-11-17: Microsoft have acknowledged the issue, and updated https://support2.microsoft.com/kb/2992611 with a workaround. Thanks @superkully for the spot.
comments powered by Disqus