Jérôme Petazzoni

Docker + Joyent + OpenVPN = Bliss

TL,DR: in my quest to CONTAINERIZE ALL THE THINGS!, I replaced my cheap VPS with a Linux VM at Joyent, installed Docker on it, then authored an OpenVPN image for Docker. The Dockerfile and scripts used are on jpetazzo/dockvpn on Github.

Let me sing you the song of my people

Do you remember that revised Maslow pyramid, the one with WiFi at the base of everything? Well, somewhere in my pyramid, there is a Linux box, with root access, a fast link, and low latency. I used to run an hosting company in France and I also worked for a very disruptive dark fiber provider in the Paris area; so when I was in the land of baguette, foie gras, and Tour Eiffel, those commodities were kind of granted.

But when I set foot in California, I was more than 150ms away (round-trip) from my home plate, so I wanted to secure a cheap, low-profile VPS. It’s actually one of the first things I did joining the dotCloud team. I could have spun an EC2 instance (since that’s what the dotCloud platform was using back then), but since one of the roles of the machine was to give me access to our servers when everything else was on fire, I wanted something in a completely different location.

Recently, I started to run more things on that server (including Gunsub, my tool to keep my Github notifications under control), and I saw it coming: I would pile up software experiments on that box until it would become a sysadmin nightmare. (My “software experiments” sometimes involve installing a dozen new packages, half of them from PPAs or even other distros.)

So I decided that I needed a new box, and that everything on that box would run within its individual Docker container.

Docker on Joyent

My requirements were the following:

  • full root access;
  • ability to switch kernels easily (Docker needs 3.8, or even better, 3.10);
  • Ubuntu 12.04 images;
  • provisioning API (not that I really needed it, but APIs are fun);
  • good I/O performance (that’s a good way to rule out EC2);
  • low latency with San Francisco;
  • cheap ($10/mo or less).

Since I had been checking Joyent for a while, I decided that it was time to give them a try. Interestingly, their pricing page says that the cheapest instance is the equivalent of the infamous t1.micro of EC2, but once you hit the dashboard, they have instances with 512 MB and 256 MB of RAM, for 1.6 and 0.8 cents per hour respectively. The latter maps to less than $6/month, and 256 MB is more than enough for Docker.

Spinning up a VM

This is covered by Joyent material, so I will keep it super brief. To setup an account with them, you will need a credit card. They have a free trial, and if you run just this tiny instance, it means that you won’t get charged the first two months (more than enough to decide if you want to keep it). You also need to provide a SSH public key which will be used when privisioning your servers.

When starting your VM, you will need to specify the base image to use, the VM size, and datacenter location.

I strongly recommend to use Ubuntu 12.04, because 12.04 is a “LTS” release, i.e. Long Term Support. It means that it will get security updates for at least 5 years. It might not be as bleeding edge as the latest Ubuntu out there, but if you happen to still run it in 3 years, you won’t have to worry about updates. I like that peace of mind.

The smallest VM size has 256 MB of RAM, and this is enough to run Docker and a bunch of containers. That’s what I’m using. By the way, it says that you have 16 GB of disk, but that’s actually the size of the data disk (mounted on /data): the VM also comes with a system disk which is almost 10 GB, a separate /tmp partition (3 GB), and a separate swap partition (4 GB). Nice.

Last but not least, you probably want to pick the datacenter closest to your location.

Once that your VM is running, login with your SSH key as root.

Upgrading the kernel

Joyent Ubuntu image comes with an “optimized kernel”. It might be optimized, but it doesn’t have AUFS support, so you want to install an official Ubuntu kernel instead:

apt-get install linux-image-generic-lts-raring

Also, since the Ubuntu kernel is 3.8.0 and the Joyent one is 3.8.6 (at least when I installed my machine), it means that by default, the Joyent kernel will be considered more recent, and be booted by default. To boot our shiny new kernel:

  • check /etc/default/grub and make sure that GRUB_DEFAULT=saved;
  • if GRUB_DEFAULT had a different value, change it to saved and run update-grub;
  • finally, run grub-set-default 2.

0 is the Joyent kernel, and 1 is the Joyent kernel in failsafe mode.

It’s now time to reboot, login again, and check with uname -r that we have our new AUFS-capable kernel.

Installing Docker

That’s the easy part! Since we’re a lazy bunch, we’ll use a convenience script to automatically add the Docker package repository to our list of APT sources, add the repository key to the list of trusted keys, and install Docker from the repository:

curl https://get.docker.io/ubuntu/info | sh -x

Moments later, Docker will be running on the machine. It will also be started automatically if you reboot. Actually, it would be a good idea to reboot once more to make sure that everything starts properly.

Pre-flight check

Now that Docker is installed (on Joyent or elsewhere), let’s do a quick check to make sure that everything is fine:

root@32450c46-...-a53829c059fe:~# docker run busybox echo "Ahoy Matey"
Ahoy Matey

Note: the first time, it will take a few seconds to pull the busybox image, and the output will show a bit more than “Ahoy Matey”. That’s OK

Worked? Good, we can get to the VPN setup.

OpenVPN on Docker

The key elements to run OpenVPN within Docker are:

  • the -privileged mode flag, which unlocks the capabilities required by OpenVPN to manipulate tun/tap interfaces, and perform required network funkiness;
  • a proper mapping of UDP and TCP ports;
  • obviously, a working OpenVPN configuration.

Since writing an OpenVPN configuration is boring, we will make sure that it gets generated automatically; and we will add an easy way to transfer the configuration to the client.

See it in action

Before examining how to put everything together, let’s have a look at the result. To start your own OpenVPN server, just do this:

CID=$(docker run -d -privileged -p 1194:1194/udp -p 443:443/tcp jpetazzo/openvpn)

This has generated configuration files for the server and client, and started the server processes, exposing them on ports 1194 (for UDP access) and 443 (for TCP access). Now, to download the client configuration file, just do this:

docker run -p 8080:8080 -volumes-from $CID jpetazzo/openvpn serveconfig

An URL will be shown on the output: it is a link to a downloadable OpenVPN profile (ovpn file), embedding the key and certificate needed to connect to the server.

After downloading the profile, stop that container (otherwise anyone can download your configuration and certificate, and that’s certainly not good).

Under the hood

Now let’s see how everything works!

The Dockerfile

The Dockerfile is not very long. Let’s review it in depth.

FROM ubuntu:precise

We make sure to use the precise version of the Ubuntu image, because on the next line, we will add the universe repository. If we specify FROM ubuntu, we could end up with something else, and mixing the packages could lead to random results.

RUN echo deb http://archive.ubuntu.com/ubuntu/ precise main universe > /etc/apt/sources.list.d/precise.list

We add the universe repository because later, we will install socat; and socat is not in the main repository.

RUN apt-get update -q
RUN apt-get install -qy openvpn iptables socat curl

Now we update and install the required packages: openvpn (for obvious reasons!), iptables (because we need it to add a NAT rule so that VPN clients can use our network connectivity), socat (used to run a pseudo HTTP server), and curl (used to figure out the public IP address of the server, using the http://myip.enix.org/ service).

ADD ./bin /usr/local/sbin

The bin directory contains the run and serveconfig scripts which are used later.

VOLUME /etc/openvpn

By making /etc/openvpn a volume, we allow its sharing across multiple containers. The goal is simple: the main service will execute the run script, which will generate the private key, certificate, and configuration files. Then another container can be started, executing the serveconfig script, but using the same volumes (thanks to the -volumes-from flag). This other container will be able to access the configuration files generated by the first one, and serve them to the clients.

EXPOSE 443/tcp 1194/udp 8080/tcp

Those are the network ports that we want to use. Note that this line was not strictly necessary, since in our examples, we explicitly mapped the ports; in other words, our examples would work even without the EXPOSE line. But if you want to do something special, like running multiple OpenVPN containers in parallel, this line will make it easier (because you won’t have to specify -p flags every time).

CMD run

This last line tells Docker to execute the run script by default. In other words, docker run jpetazzo/openvpn is equivalent to docker run jpetazzo/openvpn run.

The run script

This is a shell script. Let’s see what it does!

[ -d /dev/net ] ||
    mkdir -p /dev/net
[ -c /dev/net/tun ] ||
    mknod /dev/net/tun c 10 200

This will make sure that /dev/net/tun exists. It is needed by OpenVPN to interface with the tun/tap device. See how we carefully check if things already exist before trying to create them: this is to avoid File exists errors if we are restarting the container (or if the device was already created for any reason).

cd /etc/openvpn
[ -f dh.pem ] ||
    openssl dhparam -out dh.pem 512
[ -f key.pem ] ||
    openssl genrsa -out key.pem 2048
chmod 600 key.pem
[ -f csr.pem ] ||
    openssl req -new -key key.pem -out csr.pem -subj /CN=OpenVPN/
[ -f cert.pem ] ||
    openssl x509 -req -in csr.pem -out cert.pem -signkey key.pem -days 36525

Now we go to /etc/openvpn (which is the volume where we want to store all the configuration files), and we generate the Diffie-Hellman parameters, the RSA private key, the Certificate Signing Request, and the signed certificate required by OpenVPN and its client.

We use a 2048-bits RSA key; this is supposed to be NSA-proof, but if you are serious about security, there are other steps that you will want to take, as indicated in the Security Discussion.

Again, before creating each file, we check if it’s already there. That way, we don’t end up overwriting keys if we restart the container.

[ -f tcp443.conf ] || cat >tcp443.conf <<EOF
server 192.168.255.0 255.255.255.128
...
EOF

[ -f udp1194.conf ] || cat >udp1194.conf <<EOF
server 192.168.255.128 255.255.255.128
...
EOF

I haven’t included the server configuration files in full because they are fairly standard. The server directives will take care of all the heavy lifting (they are actually macros expanding to longer configuration sections). As you can see, we actually create two configuration files, for two different OpenVPN servers: the first one will accept TCP connections, the second one UDP packets. They use different network ranges.

Let’s have a look at the generation of the client configuration file:

[ -f client.ovpn ] || cat >client.ovpn <<EOF
client
nobind
dev tun
redirect-gateway def1

<key>
`cat key.pem`
</key>
<cert>
`cat cert.pem`
</cert>
<ca>
`cat cert.pem`
</ca>
<dh>
`cat dh.pem`
</dh>

<connection>
remote `curl -s http://myip.enix.org/REMOTE_ADDR` 1194 udp
</connection>

<connection>
remote `curl -s http://myip.enix.org/REMOTE_ADDR` 443 tcp-client
</connection>
EOF

If you are an OpenVPN guru, this is probably very basic for you. Otherwise, there are 3 things that are worth mentioning.

  1. We use the redirect-gateway directive, to let all the traffic to through the VPN. After establishing the connection, the client will add a static route letting the VPN traffic go through the existing default gateway, and two /1 routes pointing to the VPN, effectively overriding the default gateway for all the traffic, except for the VPN connection itself. In other words, this will force all your traffic to use the VPN.
  2. The SSL key, certificate, and CA certificate are embedded within the configuration file. That way, instead of downloading multiple files, all you need is within a single file.
  3. We declare two connections profiles (with the <connection> sections): one for UDP, one for TCP. The first one (UDP) is used by default (since it’s more efficient), but if the connection fails (because your local network filters UDP traffic), the client will automatically fall back to the second one, which uses the HTTPS port (which is rarely filtered). Also, to figure out the IP address of the server, we use the http://myip.enix.org/ service.

Then, we generate a helper file for the pseudo HTTP server:

[ -f client.http ] || cat >client.http <<EOF
HTTP/1.0 200 OK
Content-Type: application/x-openvpn-profile

`cat client.ovpn`
EOF

The pseudo HTTP server works like this: it listens on a TCP socket, and when a connection arrives, it doesn’t even read the request, it blasts the response right away. Crude, but it works. The client.http file taht we generate here is the response. It contains a basic HTTP status line, a header sending the appropriate content type (so that Android and Windows devices will automatically open the VPN client), and the configuration file itself.

iptables -t nat -A POSTROUTING -s 192.168.255.0/24 -o eth0 -j MASQUERADE

Now we setup an iptables rule to masquerade all the traffic coming from our VPN clients. And finally …

touch tcp443.log udp1194.log http8080.log
while true ; do openvpn tcp443.conf ; done >> tcp443.log &
while true ; do openvpn udp1194.conf ; done >> udp1194.log &
tail -F *.log

We start both OpenVPN processes (one for TCP, another for UDP connections). They both are started within a while true ... loop; so if they crash or exit for any reason, they will be restarted automatically. They send their output to different log files.

Last but not least, we tail -F those log files. This command has two purposes.

  1. It allows us to aggregate the logs of the OpenVPN processes, and you can check them with docker logs, or if you started the container in the foreground.
  2. We need something to remain in the foreground, otherwise Docker will think that our container has exited, and terminate it. As long as the top-level process (our shell script) is here, we’re fine.

The serveconfig script

The run script generated a client configuration file, but now we need an easy way to transfer that configuration file to our client(s). There are many ways to do this:

  • we could exploit the fact that container file systems are located in /var/lib/docker/containers/<containerid>/rootfs;
  • since we used a volume to store the configuration, we could use docker run -volumes-from ... cat /etc/openvpn/... to start another container sharing this volume, and peek inside;
  • or we could use a variation of the previous idea, but to start an HTTP server exposing the configuration file to our clients. That’s what we decided to do.

Of course, serving security-sensitive OpenVPN configuration information (with embedded SSL keys) without authentication is not very wise. To limit the risk, the HTTP server will run only for a very short period (while you download the configuration), then you will kill it. This means that we start the container with the appropriate options, point our device to the right URL, download the configuration file, and stop the container.

To avoid easvesdropping, the HTTP server serves requests over SSL, using the same private key and certificate than the OpenVPN server.

The serveconfig script, which starts the pseudo HTTP server, looks like this:

cd /etc/openvpn

[ -f client.http ] || {
    echo "Please run the OpenVPN container at least once in normal mode,"
    echo "to generate the client configuration file. Thank you."
    exit 1
}

echo "https://$(curl -s http://myip.enix.org/REMOTE_ADDR):8080/"
socat -d -d 
    OPENSSL-LISTEN:8080,fork,reuseaddr,key=key.pem,certificate=cert.pem,verify=0 
    EXEC:'cat client.http' 
    >> http8080.log

It makes sure that the client.http file exists, and then it starts a socat process. socat will serve TCP connections wrapped with SSL on port 8080, send the content of the client.http file (which contains the HTTP response, headers, and body), and log the request (for mere debugging purposes).

Also, if you want to use that on Android, note that there is a bug in the Download Manager in all versions of Android, preventing the download of files from non-trusted SSL servers. Since we use a self-signed certificate, we are a non-trusted server. To work around the issue, you can download the file with e.g. Firefox, or download it on your computer and then transfer it to the phone or tablet one way or another.

Last words

I hope that this work can be useful in two or three ways:

  1. It’s an easy way to run an OpenVPN server in Docker (and, broadly speaking, almost anywhere, since Docker runs on so many different kinds of servers).
  2. It shows some advanced features of Docker: the -privileged flag, port mapping, and an original use of the -volumes-from flag.
  3. It adds Joyent to the ever-growing list of Docker-friendly infrastructure providers!

I would like to remind you that this system has a few security shortcomings. They are detailed in the security discussion on Github.

About Jérôme Petazzoni

sam

Jérôme is a senior engineer at dotCloud, where he rotates between Ops, Support and Evangelist duties and has earned the nickname of “master Yoda”. In a previous life he built and operated large scale Xen hosting back when EC2 was just the name of a plane, supervized the deployment of fiber interconnects through the French subway, built a specialized GIS to visualize fiber infrastructure, specialized in commando deployments of large-scale computer systems in bandwidth-constrained environments such as conference centers, and various other feats of technical wizardry. He cares for the servers powering dotCloud, helps our users feel at home on the platform, and documents the many ways to use dotCloud in articles, tutorials and sample applications. He’s also an avid dotCloud power user who has deployed just about anything on dotCloud – look for one of his many custom services on our Github repository.

Connect with Jérôme on Twitter! @jpetazzo

, ,

Jérôme Petazzoni

Docker + Joyent + OpenVPN = Bliss


4 Responses to “Docker + Joyent + OpenVPN = Bliss”

  1. nadrimajstor

    Wow… Excellent!
    And thanks for bringing up that nifty socat tool.

    Reply
  2. Arnaud

    Thanks for that docker/openvpn tutorial!

    What’s the best practice to keep your openvpn container up to date, especially regarding security fixes of the base image or new openvpn versions?
    (On a typical server I use the apt-get unattended-upgrades fonctionality to keep up to date, but obviously that wouldn’t work for a container.)

    Thanks!

    Arnaud

    Reply

Leave a Reply to Docker + Joyent + OpenVPN = Bliss | Rocketboom

Click here to cancel reply.

Get the Latest Docker News by Email

Docker Weekly is a newsletter with the latest content on Docker and the agenda for the upcoming weeks.