IIS causing TLS connections to reset when no SNI binding is matched

25 June 2015 | Comments

We just hit a rather confusing issue while building up some new infrastructure for the new Redgate website. We have a couple of IIS servers, sitting behind nginx, which proxies requests through.

Pointing nginx at the old IIS boxes worked fine, but changing it to the new ones resulted in a gateway unavailable error from nginx. More curiously, running curl from the load balancer's console gave an SSL unknown protocol error, yet if we put a hosts file entry on the IIS box itself, everything worked.

Cue tcpdump, Wireshark, and so on, and we could see that after curl sent an SSL Client Hello packet, IIS instantly terminated the connection with [RST, ACK], never sending the expected Server Hello.

Eventually, it turned out we had no default (no-SNI) binding set, and nginx and curl weren't passing a hostname that matched an SNI-enabled binding. So IIS was terminating the connection - very aburptly!

The fix was easy: use the --resolve switch in curl, and the proxy_ssl_server_name and proxy_ssl_name settings in nginx to pass the correct SNI header to IIS.

comments powered by Disqus