Bill Farner

Democratizing Scale: From Google’s Borg to Twitter’s Aurora, to Docker

Today, I’m proud to announce that the Conductant team has joined Docker. You probably haven’t heard of Conductant before because we have been operating in stealth mode for some time. We are Bill Farner, David Chung and John Sirois, and we come from places like Google, Twitter, and Zynga to join Docker on this amazing journey to change how people build, ship, and run code everywhere. In doing so, we expand the expertise at Docker in building and operating large-scale, mission critical systems. Expect great things in the future with our combined teams!
 

Conductant

 

The Quest for an Open Borg

In 2009, I left Google to join Twitter and quickly noticed a dearth of infrastructure and tools for delivering and running microservices. I particularly missed Google’s Borg, the software used to manage massive clusters of computers and make them easily accessible by engineers building systems and services. Recognizing Twitter would greatly benefit from such a system, I spent the next six years serving as the technical lead of a group of excellent engineers building similar infrastructure.

When we started, we learned of a fledgling software project, known as Mesos, from Berkeley that provided an infrastructure framework for building distributed applications, with facilities for task execution on remote hosts and resource accounting. To build the platform we wanted, we needed a layer capable of making complex decisions in scheduling and orchestration, from where to place workloads to supporting complex requirements like jobs (from services to batches and cron), deploys, quotas, preemption, and latency sensitive workloads. Our idea was to focus on ops-driven development by leveraging Mesos for resource accounting and task execution as plumbing and build a high-level orchestration system that would solve for Twitter the problems Borg solved for Google. The result of our efforts is Aurora – a system for running Twitter-scale workloads on large clusters. Aurora is responsible for management and scaling of microservices at Twitter and is used daily by over a thousand engineers across the company. Twitter shared this work as open source software in 2013 under the Apache Software Foundation as Apache Aurora, of which I am the Program Management Committee (PMC) chair, and Apache Mesos. I’m proud to say both projects are still very active and continue to move forward.

In building Aurora, we focused on technical as well as organizational scalability. Technical metrics (e.g. number of nodes, number of containers in a cluster) are often used exclusively to judge the scalability of an orchestration system. On the other hand, organizational scalability refers to the manageability of the system as the number of users and workloads grow. Organizational scalability is more difficult to assess but has a larger impact on productivity. At Twitter, we needed Aurora to excel at both. Today, Aurora manages clusters with tens of thousands of machines used by hundreds of engineers to develop and deploy hundreds of services. It accomplishes this feat with an operations team of just three engineers. With Aurora, Twitter engineers have massive computational resources at their disposal; and they are free to experiment and explore new ideas without worrying about system capacity and risks of impacting the performance and stability of other critical services. Aurora not only improved operational efficiency but became a platform for innovation as well.

In 2015, David Chung and I co-founded Conductant with the mission to make this technology accessible to organizations large and small. David had previously spent a year consulting companies on bridging the infrastructure gap between developers and production, drawing from his prior experience at Google and Zynga. John Sirois completed the founding team: John also came from Google; and was one of the original Aurora engineers at Twitter before embarking on his own journey to create the Pants open source build system. Together we saw an opportunity to help organizations ship code rapidly and improve developer productivity, without requiring an operations team that would balloon in size when the infrastructure and services scale up.

 

All Roads Lead to Docker

We formed Conductant with the mission of empowering teams everywhere to deploy and scale microservices with the kind of ease and efficiency that for a long time have been privy to only a handful of large Internet companies. Instead of building this again for the sole use of a single company, we want to help all application teams overcome the hurdles from explorations all the way through deployment to scaling to millions of users. To achieve this, we couldn’t ignore the impact and adoption of a game-changing technology that is Docker. As a platform, Docker has solved many of the problems we had to overcome when building Aurora on Mesos. As a tool for developers, Docker has made infrastructure approachable. The ubiquity of Docker Engine means anything that runs Docker can be orchestrated – from bare-metal servers to VMs in the cloud, or even a Raspberry Pi!

With this observation, and much excitement, we set out to reimagine an orchestration engine on top of this modern platform. One capable of managing microservices composed of Docker containers at massive scale. Docker technologies provide us with key building blocks for a system that would share the same guiding principles as Google’s Borg in terms of scalability, productivity, and operational efficiency. Yet, the reach can go far beyond that of any single cloud provider.

As we began talking with various partners, we had an opportunity to meet with Ben Golub and Solomon Hykes from Docker to share our vision of a world where developers can deploy and manage countless microservices with ease. It was immediately clear that Docker wanted to do the same. Docker’s vision to make the entire Internet programmable not only fascinates but also challenges us. We envision a world where code can come to life as soon as it is conceived and written, and we see a group of dreamers, just like us, in the Docker team. We are excited to bring the lessons we have learned over the years to the Docker platform so together, we can empower every developer and operations team to easily build, ship, and run code anywhere, at any scale.


 

Learn More about Docker

, , , , ,

Bill Farner

Democratizing Scale: From Google’s Borg to Twitter’s Aurora, to Docker


Leave a Reply

Get the Latest Docker News by Email

Docker Weekly is a newsletter with the latest content on Docker and the agenda for the upcoming weeks.