Adam Herzog

Extending Docker with Plugins

This is a guest blog post from Luke Marsden, CTO at ClusterHQ, the creators of Flocker and Alexis Richardson, CEO at Weaveworks, the creators of Weave.

Back in December last year, we started getting closely involved in the Docker plugins project with Solomon and other folks on the Docker team, along with a loose group of other collaborators and the now-famous DockerCon Europe iron.

That meeting, which Alexis organised, was the start of an open collaboration involving developers around the world, including Jeff Lindsay from Glider Labs.

Docker plugins available today

Today at DockerCon 2015, we are incredibly proud to announce that this open collaboration from the Docker team has paid off. Docker are indeed delivering on the promise of “batteries included but swappable”.

The Docker plugins mechanism is now available in the new Docker experimental channel. Also available are the first Docker plugins, including Flocker for portable volumes across hosts, and networking plugins available from Weave, Project Calico, Nuage Networks, Cisco, VMware, Microsoft, and Midokura.

What are plugins?

As a user of Docker, you can now can extend the capabilities of the Docker Engine by loading third-party plugins. Plugins are specific to certain Docker subsystems. The two plugin mechanisms (aka “extension points”) we are highlighting today are:

Network plugins, which allow third-party container networking solutions to connect containers to container networks, making it easier for containers to talk to each other even if they are running on different machines.
Volume plugins, which allow third-party container data management solutions to provide data volumes for containers which operate on data, such as databases, queues and key-value stores and other stateful applications that use the filesystem.

In both cases, the plugin mechanism takes a piece of core functionality that Docker already provides, and allows users and tool-makers to load, or write, plugins that extend that functionality in cool, new and interesting ways. After installing the plugins, using them is as easy as this.
For network plugin enabled containers:

$ docker run --publish-service=service.network.weave your_networked_container

For volume plugin enabled containers:

$ docker run -v volume_name:/data --volume-driver=flocker your_stateful_container

These extension points are just the beginning. They are a starting point for making other parts of Docker more extensible, composable and modular. For more details on this and the plugin mechanism, check out the plugin docs in the docker experimental docs tree.

 

Why are plugins good for users?

The Docker ecosystem of tool-makers is growing exponentially. The value of plugins is to integrate this ecosystem seamlessly with the Docker user experience. Every time a new plugin is created your investment in Docker becomes more valuable.

Customization leads to applications that fit your end users’ needs better. And so everyone has been asking for a supported way to extend Docker with third party or custom-built tools. This extensibility must retain Docker’s portability, consistency and ease of use.  That is the idea behind Docker plugins: one set of interchangeable tools via one Docker open platform. A user can swap out a plugin and replace with another without having to modify their application; the model is that seamless.

 

Why are plugins good for tool-makers?

Tool-makers in the Docker ecosystem have long struggled with the problem of wrapping Docker. For example, the first versions of both Weave and Flocker contained scripts that the user had to run in order to start a Flocker- or Weave-enabled container. This is bad for users because they can’t use the tools together, and it’s bad for the tool-makers because they unnecessarily have to compete with each other for the user interface.

Plugins solve this problem elegantly by making it possible to compose plugins together and do so unified under the Docker UI. So for example now you can run Flocker and Weave together under Swarm and Compose. Or you can swap in different volumes, networking, composition or scheduling framework, depending on user preferences and the special requirements of each user’s applications.

 

Example of what plugins can do: Container Migration

To demonstrate the value of plugins we are going to show you an example, that anyone can replicate themselves today. The example two plugins to enable container migration for an application on the Docker platform.  We enhance an application with two extension points.

Weave for portable networking, discovery, monitoring.
Flocker for portable volumes and data.
• A Docker Swarm cluster runs the application described by Docker Compose
• We start a database on node 1, then later reschedule it on node 2, and it keeps its volumes and network identity!
• All of this is combined into one easy Docker experience via plugins.

This migration is stateful – the data moves along with a database container – and preserves network identity – meaning that clients don’t need to be reconfigured to point to the database’s new location when it moves.  Production applications need this kind of migration, for added reliability and availability, and for continuous operation and maintenance.

You can see longer version of this video and try out the code on this plugins demo page.

In summary – plugins add functionality for real world production use cases. More cases can now be delivered using the Docker platform plus the Docker ecosystem. Every customer can benefit, no matter how unusual their application – because the platform is open and extensible in this way. Customers have been telling us they want Docker applications to do this, and they want choice over which technologies they plug in. Now they have it!

 

Try it yourself

Docker plugins and documentation are available in the Docker experimental binary, so you can plug them in yourself. But we’ve also put together a simple demo environment so that you can try out the container migration example.

For information on the individual plugins, and how to install them on your own servers, go to:

Flocker Docker plugin on ClusterHQ.com
Weave Docker plugin on Weaveworks website

There are also blog posts about networking plugins on the Docker and Weave blogs.

 

Get Involved

• Come to the technical session at DockerCon 2015 (Day 1 – 2:50pm – Yerba Buena 8)
View the full demo and try it yourself
• Check out the code: docker plugins, weave plugin, flocker plugin
• Come and hang out in #docker-dev on Freenode
Get started with your own plugin
Use experimental labels on Docker issues to provide feedback

 

Learn More about Docker plugins, Weave and Flocker

Read about Docker experimental features
Learn more about Docker plugins on the Docker experimental docs
See Flocker and Weave working with Docker Swarm and Docker Compose and try it yourself
Come to the technical session on Monday at DockerCon
Come to the Birds of a Feather session on Tuesday at DockerCon
Learn about Flocker
Learn about Weaveworks and Weave
• Links to networking plugins blog posts on Docker and Weave blogs
Flocker Docker plugin on ClusterHQ.com
Weave Docker plugin on Weaveworks website

 

How can I try the experimental binary?

Unlike the regular Docker binary, the experimental channels is built and updated nightly on https://experimental.docker.com. From one day to the next, new features may appear, while existing experimental features may be refined or entirely removed.

Each experimental feature is documented in a dedicated experimental section.  This section explains the feature’s function and design goals. Because your feedback is essential to refine and decide on a feature’s future, you will also find links to GitHub issues where you can leave your comments, look for reported problems, or report on your experience.

 

Geek out: How we build our experimental binary

The experimental binary is built from Docker’s regular master branch using a slightly different compilation command line. A feature is marked as experimental by segregating either parts or all of its code using Golang’s build constraints mechanism and conditional compilation. The question of how much feature code should be conditionally built is an interesting one that typically answered on a case-by-case basis. At the bare minimum, the command-line and API elements users can use to access a feature drive the experimental buildtag.

For example, we used this strategy in the case of volume drivers, a recently introduced experimental feature. In that case, the vast majority of the feature code is actually in mainline. This is  acceptable as long as we have a sufficient level of confidence that the feature won’t negatively impact stability or performance of Docker’s official release in any way.

From an open-source perspective, the very nature of experimental features also makes it important that reverting always remains an option. Reverting could become challenging if subsequent patches are built on top of experimental code.

A key element in our build design was consideration of Golang build constraints. They are stricly file-based, and do not allow for fine-grained conditional compilation like, for example, C-derived languages allow with inline #ifdef directives. In other words, the unit of “compile-time branching” can only either be the function or the interface, not the statement or even the expression. This restriction is both an encouragement for better code separation, and a strong constraint to cope with when attempting to isolate a particular code path.

Finally, we also have continuous integration on our experimental build to ensure that no incoming patch merged to master can break the experimental build.

So – want to have a go living on the bleeding edge? Head over to experimental.docker.com to try out the experimental release. If you have any feedback on any of the new features in the release, open a thread on the mailing list or add comments on the associated GitHub issues.


 

Learn More about the Docker News from DockerCon 2015

Join our next Docker online meetup recapping all of the news from DockerCon including demos of the latest features of Docker 1.7. The meetup is on Monday, June 29 at 10:00 PDT / 19:00 CEST – click here to register!

 

Learn More about Docker

 

, ,

Adam Herzog

Extending Docker with Plugins


One Response to “Extending Docker with Plugins”

  1. Docker Networking 1.9 and Weave

    […] worked with Docker, ClusterHQ and Glider Labs to launch Network Extensions at DockerCon (blog, video).  These are also known as Network Plugins.  Plugins use a Docker library called […]

    Reply

Leave a Reply

Get the Latest Docker News by Email

Docker Weekly is a newsletter with the latest content on Docker and the agenda for the upcoming weeks.