My Heart Bleeds over OpenVPN security hole
Unless you’ve been under a rock you will have heard something about the Heartbleed security flaw in OpenSSL.
Heartbleed is arguably the most serious publicised security flaw in the history of the internet, because so many servers have been affected. Even if you have nothing to do with computer admin you could be affected if any of the sites you visit have been exploited. All internet users need to seriously consider changing all their passwords thanks to some negligent coding practices in an open source project…
The Heartbleed Loophole
There’s extensive information on the Heartbleed flaw here at heartbleed.com, but here’s a very brief summary:
The Heartbleed security hole was introduced when an extension was added to the OpenSSL project way back in December 2011. It’s conceivable that astute hackers may have been exploiting the problem for years! At the time the developers committing an extenstion to the project didn’t detect a potential bounds-checking problem. The name “Heartbleed” is derived from the fact that this new extension handled TLS “heart beats”. Essentially versions 1.0.1 through 1.0.1f of OpenSSL are affected by the bug and allow hackers to potentially extract chunks of private data by sending malformed packets and getting back the contents of raw memory buffers. Unfortunately for us, OpenSSL is the most widely used open-source implementation of secure sockets layer and is included with many operating systems (dozens of modern linux distros).
Hardening my VPN
This particular blog post covers my experience patching a potentially vulnerable VPN that I manage when I’m not coding. The VPN is based on OpenVPN community edition and it turns out OpenVPN may have been a lot more “open” than I expected!
If like me, you also use OpenVPN but have done nothing yet, you are almost guaranteed to be vulnerable to the exploit. Importantly Heartbleed can be exploited on both clients and servers, so you need to harden things up, fast. It’s even possible for a hacker to masquerade as your server if they’ve acquired your secret keys. It’s also nearly impossible to know if your system has been compromised already. Unfortunately if your VPN is large, securing things can entail a bit of work…
The Checklist
Here is the checklist of things that need to be done:
- Update the version of OpenSSL on your server and ALL clients
- Revoke your existing private keys
- Generate new private keys
- Create new certificates
- Assume everything’s compromised already and change all passwords in use on every system
When you have more than 30 remote clients (Nix and Windows) that little list can take a lot of fixing…
Here’s a breakdown of my approach to correct my VPN with a CentOS Server, about 7 Windows Clients and about 25 Linux Mint Clients:
Windows OpenVPN Client Machine Fix
On Windows you can check if your openssl version is vulnerable by opening a command prompt and typing:
openssl version -a
However, this will only give you the version of the default .exe on your system path. It’s important to dig out every vulnerable version resident on every system for optimal security.
If any version is 1.0 or below it’s OK, if it’s 1.0.1g or above then it’s also fine. Anything else is potentially vulnerable.
Sometimes the version command will only list the major version ie. 1.0.1 (without the letter for the minor version). You can double-check the minor version by looking at the “built on” date: if it’s more recent than 7 April 2014 your version is OK.
On my Windows laptop there were three different copies of openssl: the default one being used by OpenVPN, an unused copy that may have been bundled with some other software, and a copy that came with some Git tools. The Git version was 0.98 so didn’t feature the TLS Heartbeat extension, and therefore wasn’t affected by the vulnerability. The other version that had been bundled with some software wasn’t something I used so I simply deleted it. The version that was being used by OpenVPN needed to be removed too.
The next step was to upgrade to the patched openssl version. Since I was also patching the version of OpenVPN on the system, the best way to kill two birds with one stone was to download the latest OpenVPN installer which is bundled with a secure openssl.exe. You can download the installer for the latest version of OpenVPN with the patched version of openssl from the OpenVPN community site here:
http://openvpn.net/index.php/download/community-downloads.html
After uninstalling the old version, and removing existing openssl.exe binaries, I ran the installer making sure that openssl dll installation was checked under dependencies (it was by default). On Windows the 64 bit installer installs openssl.exe into by default:
\Program Files\OpenVPN\bin\
The version installed on my Windows systems are now up to:
OpenSSL 1.0.1h 5 Jun 2014
built on: Thu Jun 5 14:59:25 EEST 2014
..but by the time you read this there may be an even newer version out there.
CentOS OpenVPN Server Fix
The process for correcting the openssl version on my CentOS linux server was a little different. RedHat back-port fixes into existing releases for CentOS rather than allowing potentially breaking upgrades to untested software to be installed. Because of this, your version may still appear to be 1.0.1e (a vulnerable version) even after the upgrade. As with windows, you can fetch the openssl version from the terminal:
openssl version -a
If your built-on date is earlier than April 7 2104 you must update. To do so, as root run:
yum update openssl
which will grab the non-Heartbleed backported fix of openssl. You should see something like this in the terminal:
--> Running transaction check
---> Package openssl.x86_64 0:1.0.1e-16.el6_5.4 will be updated
---> Package openssl.x86_64 0:1.0.1e-16.el6_5.14 will be an update
--> Finished Dependency Resolution
When prompted enter y to confirm the installation of the update. To double check that the update has been effective you can query the changelog:
rpm -q --changelog openssl | grep CVE-2014-0160
Which should yield a result along the lines of:
- fix CVE-2014-0160 - information disclosure in TLS heartbeat extension
After that, retrieving the openssl version on the CentOS server produces:
OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Thu Jun 5 12:49:27 UTC 2014
To ensure that this new version takes effect and no out-of-date version remains active in memory, the safest, cleanest and easiest thing to do is reboot the server.
Linux Mint Client Machine Fix
The last (and largest) batch of computers I needed to update were my numerous Linux Mint clients hanging off the VPN. The Mint update process was a little different to CentOS. Mint, being Debian based uses apt-get for package management. I took this as an opportunity to thoroughly update all my machines by running the following commands as root:
apt-get update -- updates the internal database
apt-get upgrade -- upgrades all out of date packages
Both steps may take a while to update systems fully and you may get confirmation prompts for some packages during the upgrade process.
Once the systems are upgraded, all services using libssl must be restarted. As with the CentOS server, the safest, cleanest and easiest way to guarantee that this occurs is to reboot your client system.
Replacing Compromised Certificates and Keys
After openssl has been updated it’s time to create new keys and certificates for every OpenVPN server and client. TThis is no different from the normal process of key/cert generation, but must be done, because all existing keys and certificates have to be treated as compromised. Producing replacement keys and certs and distributing them to the client machines can take a while.
Changing Compromised Passwords
After every system has had its keys and certs replaced, you need to go through and change all your passwords.
If you don’t already have one, you should seriously consider having a password change policy in place. It has to be assumed that there will be other vulnerable openssl systems out there and any passwords used with these systems could be compromised. Frequent password changing is probably the safest thing to do until you can be sure that no clients will be connecting to insecure openssl connections.
On Linux you can set a password change policy in the file /etc/login.defs by editing the lines:
PASS_MAX_DAYS XXX
PASS_MIN_DAYS 0
PASS_WARN_AGE 7
Final Thoughts
I think it’s great that this flaw has been picked up and is largely thanks to the fact that the OpenSSL project is open-source. But it requires a stack of work to re-secure a compromised system, and it’s impossible to know if the system had been compromised in the first place.
It is also a pity that OpenSSL ever got released with such a fault. One of the wonderful aspects of open-source projects is that lots of pairs of eyes get to review the code, hopefully increasing the chance that any security flaws will be picked up prior to release. But on the other side of this coin, open-source software is often contributed to by numerous developers, and sometimes the bigger picture can be overlooked when an increment patch is introduced. Furthermore, hackers are able to scrutinise the source code themselves, potentially picking up on any inconsistencies that can be exploited. We have to stay vigilant, and deal with these problems as quickly as possible when they arise.
It’s a brave new world.