We are extremely excited to announce the first beta release of Swarm, a native clustering tool for Docker.
For the past two years, Docker has made the lives of millions of developers easier by making building, shipping and running applications simpler through containers. However, things get complicated when dealing with more than one host for Docker containers in a distributed environment.
This is where Swarm comes in. Swarm pools together several Docker Engines and exposes them as a single virtual Docker Engine. It serves the standard Docker API, so any tool that already works with Docker can now transparently scale up to multiple hosts.
Back in December, to address issues with working in a distributed environment, we announced a tech preview of Swarm built on a few core principles:
- Extremely easy to get started
- Same Docker API you already know
- “Batteries included but swappable” implementations for discovery and scheduling
Check out this video for a demo of how it works:
Since Swarm ships as a standard Docker image with no external infrastructure dependencies, getting started is a simple, three-step process:
- Run one command to create a cluster.
- Run another command to start Swarm.
- On each host where the Docker Engine is running, run a command to join said cluster.
Find out more about the setup process in the documentation.
Docker API Compatibility
Most Docker API endpoints are implemented by Swarm, meaning that tools built on top of it will work out of the box, including the Docker CLI.
This guarantees consistency for Docker users: Whether you are developing on your laptop, testing in a staging environment, or actually deploying in production on hundreds of hosts, the experience will be the same since you’ll be using the same tools you are already familiar with.
For instance, by pointing the Docker CLI to Swarm, running an interactive shell is as easy as:
docker run -H swarm_ip:swarm_port -it ubuntu bash
Behind the scenes, Swarm will create a container in one of the hosts and proxy the standard input/output back to the CLI.
Swarm is aware of the resources available in the cluster and will place containers accordingly.
The default placement strategy takes into account the resource requirements of the container and the available resources of the hosts composing the cluster to optimize the placement using a bin packing algorithm.
For instance, the following command will schedule a Redis container with one gigabyte of memory on any machine that has enough resources:
docker run -d -m 1g redis
In order to meet the specific requirements of each container, their placement can be fine-tuned using constraints. For instance, Swarm can be instructed to run MySQL on a host with flash storage:
docker run -d -e constraint:storage==ssd mysql
Constraints operate on Docker daemon labels. To make the previous example work, Docker must be started with the –label storage=ssd option.
More advanced expressions are also supported:
docker run --rm -d -e constraint:node!=fed* docker run --rm -d -e constraint:node==/ubuntu-[0-9]+/
In some cases, the placement of a container must be relative to other containers. Swarm lets you define those relationships through affinities.
The following will run two Redis servers, while guaranteeing they don’t get scheduled on the same machine:
docker run -d --name redis_1 -e ‘affinity:container!=redis_*’ redis docker run -d --name redis_2 -e ‘affinity:container!=redis_*’ redis
Affinities are automatically generated when the relationship between containers is implied. Containers that use options such as –link, –volumes-from and –net=container: get co-scheduled on the same host.
TLS authentication can be enabled to secure the communication to and from Swarm.
At some point, Swarm will be able to reschedule containers upon host failure.
Let’s say you schedule a frontend container with some constraints:
docker run -d -e constraint:storage==ssd nginx
If the host of this container goes down, Swarm will be able to detect the outage and reschedule this container on another host that can respect the constraint storage==ssd
Highly Available Scheduler
Right now, Swarm operates in single master mode. In the future, Swarm will support Master election, meaning that, if a Swarm master goes down, another one will get elected and container scheduling on the cluster can continue without interruption.
Batteries Included But Swappable
Since the inception of the project, the goal has been to provide default built-ins that can be swapped out.
We are excited to announce that the Scheduler Driver API has been specified and is currently being implemented. This API will allow Swarm to integrate with other clustering solutions such as Mesos and Kubernetes.
Swarm integrates with third-party container orchestration products as well as with orchestration services offered by cloud providers. We have collaborated with Mesosphere to develop a scheduling driver for Mesos and Mesosphere DCOS as a reference implementation of the Scheduler Driver API. This integration allows Swarm to orchestrate and schedule containers at scale in production on hundreds and thousands of machines. Integrations are also planned for Amazon EC2 Container Service (ECS), Google Kubernetes, IBM Bluemix Container Service, Joyent Smart Data Center and Microsoft Azure.
In order to discover the hosts that are part of a cluster, Swarm relies on a discovery service.
By default, it will use the Docker Hub to perform discovery since this is the easiest way to get started with no external infrastructure dependencies.
However, that can be swapped out for a different service through the Swarm Discovery API. Currently, the list of other discovery services supported includes Zookeeper, Consul and etcd.
Swarm is part of the Docker orchestration tools, alongside Machine and Compose. The three can be used separately, but have also been designed to work together.
Read more about using them together in the blog post about orchestrating Docker with Machine, Swarm and Compose.
We’d like to thank every community member that made this release possible by submitting contributions:
Chanwit Kaewkasi, Jeff Chien, Pierre Wacrenier, Shijiang Wei, Jessica B. Hamrick, Rob Haswell, zhangbaitong, Ankush Agarwal, Derek Schultz, Gp De Ciantis, Keegan Lowenstein, Luke Marsden, Matt Drollette, Matthew Fisher, Nick Schuch, Omer Katz, Radek Simko, Ranjandas, Thimo, Thomas Meson.
Learn More about Swarm
We would love to get your feedback on this first beta, so please test it out:
- Installation instructions for Swarm
- Report issues and contribute on GitHub
- We will be happy to answer your questions and gather your feedback on IRC, #docker-swarm on Freenode
- Read more about orchestrating Docker with Machine, Swarm and Compose
- Register for this Online Meetup on Machine, Swarm and Compose