Solomon Hykes

The Dawn of the Orchestration-for-All Era: Docker welcomes the creators of the Aurora project

Behind every tech giant is a treasure of innovative technology kept under wraps to preserve a strategic advantage.

At Docker, we believe our job is to democratize these technologies by integrating them in tools that are easy to use and help people create new things. We did this for Linux containers, to help make applications more portable. We are also doing it with hypervisors and unikernels with the help of the Unikernel Systems team. Today we are proud to take a new step in this direction by acquiring Conductant, Inc.

I am delighted to welcome the Conductant team to the Docker family. Bill Farner, David Chung and John Sirois have made key contributions to operating and scaling production systems at Google, Twitter and Zynga, with Bill having the additional distinction of being the creator of the Aurora Project. As an ops engineer by trade, I am particularly delighted to add such valuable talent and experience to the collective DNA of the Docker team.

Aurora is a popular extension of the Apache Mesos clustering system optimized for extremely large-scale production environments. It is widely recognized as the most scalable and operationally-robust component of the Mesos stack. Aurora is the perfect example of a powerful methodology which we call operations-driven development (ODD). The team at Conductant ran and operated some of the largest cloud environments in the world at Google, Twitter and Zynga. In the process of managing operations for these companies, they had to develop tools that no one else could build for them. This resonated with us because Docker itself is the fruit of our team’s experience operating large-scale cloud platforms. Bill Farner’s team at Twitter built Aurora to meet its own requirement: automate its massive server farms so that they could manage them with just a handful of experienced operations engineers. Very few tools available today have been battle-tested at such operational scale. There are many commercial distributions of Mesos, but none of them incorporate Aurora. We believe that is a wasted opportunity. We plan on incorporating the best ideas from Aurora into Docker Swarm, and are exploring integrating Aurora as an optional component of the official Docker stack.

One very exciting possibility is to integrate Aurora with Docker Swarm to form a powerful large-scale web operations stack. Swarm’s goal is to provide robust, standardized primitives for deploying any distributed application, on any infrastructure, at any scale. This makes it an ideal standard base layer to scale all kinds of applications, from databases to stateless web workers, scientific computing or big data pipelines. In contrast, Aurora is optimized for the specific needs of large-scale consumer apps reaching hundreds of millions of users, and developed with a bleeding edge architecture and methodology. We believe that in many use cases, a stack combining Docker Swarm and Aurora could democratize the battle-tested Twitter operations model. Of course, not every application is a good fit for Aurora, and it will remain completely optional for Docker users – and vice-versa. By making two of the most popular open-source infrastructure projects interoperate better, we believe both communities will benefit.

image00
 

Bill, David and John will continue to practice ops-driven development at Docker as part of our infrastructure group: between Docker Cloud deploying containers to dozens of datacenters, and Docker Hub passing 2 billion pulls, there is plenty of scaling to do! Our operational experience, in turn, is contributed back to the entire community in the form of improvements to the Docker platform and our Container as a Service (CaaS) commercial offering.

If you want to learn more about secure, large-scale container deployment, stay tuned for SwarmWeek: a week-long series of in-depth articles and exclusive data on real-world usage of Docker Swarm. Recommended for both beginners and experts!


 

Learn More about Docker

, , , ,

Solomon Hykes

The Dawn of the Orchestration-for-All Era: Docker welcomes the creators of the Aurora project


Leave a Reply

Get the Latest Docker News by Email

Docker Weekly is a newsletter with the latest content on Docker and the agenda for the upcoming weeks.