CQ / Hello World…

A quick introduction: by day, I'm a DevOps Engineer at Red Gate, a software company in Cambridge, UK. Outside of work, I enjoy both amateur radio (hence the callsign, M0VFC) and community broadcast radio at Cambridge 105. This blog aims to span all those interests - so feel free to ignore the posts that aren't relevant!

Feel free to get in touch on Twitter (@rmc47).

73 / Best wishes,
Rob


Engineering Homegrown music festival with Cambridge 105

31 August 2015 | Comments

This bank holiday weekend, Cambridge 105 broadcast live from the Homegrown music festival in Barrow, Suffolk. We bought sets live from the two stages, as well as interviews and live acoustics sets from a backstage green room area.

Jon Watts has published some fantastic photos of the weekend if you want to see what it's like from the point of view of an audience member, but this is a look at the behind-the-scenes engineering.

At the highest level, it's a simple problem: you need some set of sources (microphones, music sources, feeds from stages), a way of mixing them together, and some way of getting the output back to the studio in the centre of Cambridge. However, the reality looks more like this:

By using a digital mixer like the X32, a lot of routing that would previously been complex is made easy. For example:

Talkback was particularly important for this setup: we engineered the show from our outside broadcast van, but the presenters and acoustic performers were in a green room area, out of sight. We needed to be able to cue presenters when the main stages (also out of sight) were about to start, stop, go into new tracks and so on, all while the presenters were live on air.

As well as getting the engineering of this right, it's a real skill for the presenters to be able to keep going while having a voice in their ear talking at them - all credit to Neil and Julian for doing this so well!

Along with the digital mixer comes digital snakes or multi-cores. The S16 allows us to have 16 inputs and 8 outputs remote from the mixer using only a single Cat5E cable - much lighter and easier to handle than a conventional snake.

Finally, we have to get the signal back to the studio. Barrow is much further away from the studio than most of our outside broadcasts, so this demanded a new approach. We used a combination of an IP stream from a laptop in the OB van over 3G and a band 1 (~53MHz FM) link back to a remote receive site where it then hit a separate IP stream. The 3G stream was our primary feed, but automatically fell back to the band 1-fed stream in case it dropped. In the event, 3G held up remarkably well, but it's dangerous to rely on it as the only option, particularly at a festival where a much larger than normal collection of people suddenly turn up in a rural area with their mobile devices.

IIS causing TLS connections to reset when no SNI binding is matched

25 June 2015 | Comments

We just hit a rather confusing issue while building up some new infrastructure for the new Redgate website. We have a couple of IIS servers, sitting behind nginx, which proxies requests through.

Pointing nginx at the old IIS boxes worked fine, but changing it to the new ones resulted in a gateway unavailable error from nginx. More curiously, running curl from the load balancer's console gave an SSL unknown protocol error, yet if we put a hosts file entry on the IIS box itself, everything worked.

Cue tcpdump, Wireshark, and so on, and we could see that after curl sent an SSL Client Hello packet, IIS instantly terminated the connection with [RST, ACK], never sending the expected Server Hello.

Eventually, it turned out we had no default (no-SNI) binding set, and nginx and curl weren't passing a hostname that matched an SNI-enabled binding. So IIS was terminating the connection - very aburptly!

The fix was easy: use the --resolve switch in curl, and the proxy_ssl_server_name and proxy_ssl_name settings in nginx to pass the correct SNI header to IIS.

Windows Server 2012 R2 VM blue-screens in VMWare after export from EC2

20 February 2015 | Comments

Amazon's EC2 provides a very useful virtual machine import and export service that lets you move VMs from your own virtual environment into EC2 and vice versa.

I've used this with good success in the past - at Redgate, we have a standard VM we use for demonstrating our products, which needs to be run offline on laptops at various events, but hosting it in the cloud gives us useful flexibility in situations where we do have a good internet connection.

For our previous Windows Server 2012-based image, we've round-tripped it several times with no problems, but a recent export of a newer 2012 R2 image gave this error on its second boot:

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (xen.sys)

Subsequent boot attempts had the same problem, though Safe Mode worked OK.

Skip forward several frustrating hours trying to work out how to fix this, and with some invaluable help from the wonderful Clive, here's the somewhat dirty fix:

This removes the Xen filter driver (Amazon EC2 uses the Xen hypervisor), which for some reason seems to confuse Server 2012 R2 running under VMWare Workstation, at least.

Disclaimer: I've not tried re-importing one of these VMs to EC2 after removing the Xen filter driver. My assumption (hope?) is that on import, the Amazon PV driver will be re-installed and put it back, but you may want to check this.