John Willis

Docker and the Three Ways of DevOps Part 3: The Third Way – Culture of Continuous Experimentation and Learning

written by John Willis, Evangelist at Docker

The Third Way of DevOps completes the full cycle. It has also been referred to as just “Continuous Learning”.

In DevOps speak, we sometimes allude to the word Kaizen as a method of continuous improvement in an organization. This “Kaizen” being achieved through a culture of continuous experimentation and learning throughout all activities in an organization.

If you recall, the First Way post in this series was about a left to right flow and systems thinking. While the Second Way defined right to left feedback loops with quick turnaround.

In this Third Way, we use a symbol of a complete loop, because it ties the first two ways together through a rigorous implementation of a learning process.

image_pdsa
source: The Deming Institute

Organizations that follow the principle of the Third Way employ a form of experiential learning. Edward Deming, a famous American management thought leader; calls this the Plan Do Study Act Cycle (PDSA). PDSA is rooted in principles of scientific method in that every thing you do is a small experiment.

In 1999, Steven Spear and Kent Brown published an article in the Harvard Business Review called “Decoding the DNA of the Toyota Production System (TPS)”. TPS is well know as the birthplace of Lean and it has been well established that DevOps inherits an abundance of Lean principles. In the article, Spear and Brown explained that TPS applied scientific method to everything that happened in the plant. One of my favorite quotes from the article is: “TPS is a community of scientists continuously experimenting”.

Mike Rother, author of Toyota Kata, takes this idea a step further. Rother suggests that TPS leadership used PDSA cycles combined with a vision (i.e., a true-north) as a set of coaching tools to impose a sort of memory muscle through repetition (Kata). The term Kata is a Japanese word used to describe choreographed patterns of movements. Kata is used in the traditional Japanese art of Kabuki as well as in martial arts.

675px-Odori_Keiyō_Edo-e_no_sakae_by_Toyokuni_III

Rother suggests that TPS would align all of their activities as target conditions bound by a PDSA cycle always studying the results of each experiment before proceeding with the next action (the “A” in PDSA). They would ask the question in the “Study” step of the cycle – Did the experiment produce results that moved in the direction of the vision? These concepts of this Third Way become the tools for implementing DevOps in an organization.

Docker and the Third Way

In order for an organization to behave like a community of scientists, it has to have reliable lab equipment. In the classical sense, a scientists has a common set of equipment used while experimenting (test tubes, beakers, a microscope and of course safety goggles). Here again, like in the previous two posts in this series, we would like to suggest that Docker is a great adjunct, the lab equipment if you will; to the Third Way.

third-way-400x224

In the business of delivering software and services speed to market is essential. It’s not just how fast you can deliver services, it’s how fast can you react and reproduce to a customer’s validation of the delivered service.

In the previous two articles, we have show how Docker is a great solution for achieving exponential speed. I recently was talking to a large financial institution that is using Docker as a sort of “Container as a Service” vehicle for their internal customers. They told me that they have over 100 data scientists in their organization that have to constantly analyze large sets of data with disparate context. These data scientists have to match the right analysis tool to the right data. If they misjudge the fit, a lot of time is wasted or worse, sub-optimum results can be produced after a long running job.

Prior to implementing a Docker-based “Container as a Service” solution, it was extremely hard for these scientists to match an analysis tool to the data. Some analysis tool/data combinations perform well with a tool like Hadoop while others data sets are better suited for a tool like Spark. Quite frankly other sets work just fine with something like R. The meta point is that there are a lot of tools out there for data analysis and in fact new ones get added every other day.

What this organization has done is create a sandbox environment of prebuilt container images (i.e., Docker images) that encapsulate all of the ingredients required to run the data analysis tool. The result of this solution is that a data scientists in this organization can instantiate a containerized set of data and a specific analysis tool (i.e., in a container) in minutes and can confirm or reject the experiment results in a two-order of magnitude less time.

Prior to implementing this Docker-based “Container as a Service” offering, the lead time for experimenting the data-to-tool process could take over two days of setup and request time to to deploy the test. Now they can run through a set of experiments testing multiple tools in a few hours.

Imagine the second order effects of this solution. Imagine that in this organization, a data scientist might actually be able to try out three or four different analysis tools before landing on a best-fit solution for the larger data analysis project. While prior to the “Container as a Service” offering, a data scientist might have just settled on the first “just good enough” solution.

This model can be used as a pervasive solution for any process in an organization that requires a similar selection process.

Recently, I was working with my 12 year old teaching him analytics through baseball statistics. As I was beginning to install R and some cool baseball R packages on his laptop, I saw a tweet by my colleague Jessie Frazelle on a blog article she had just posted, “Using an R Container for Analytical Models”. In her blog example, she showed how you could create a container based on the r-base Docker image. My son and I created our own Docker image that included the r-base back as well as a baseball R package that includes all sorts of cool statistical functions and about 30 matrix databases (gliderboy380/lahman). These databases include almost every baseball statistic all the way back to 1871. We literally can download the image from the Docker Hub to any computer running anywhere and have immediate access to all of this incredible statistical data.

We are currently trying to use this tool (lab equipment if you will) to gain a competitive advantage in a fantasy baseball league. The net-net here is that the mean-time to experiment from any computer anywhere has been drastically reduced for anyone wanting to work with baseball statistics in R.

We have shown similar examples in the previous two posts in this series where being able to pivot fast (e.g., baselining and rebasing in the CI flow) can be extremely effective in overall service delivery lead times. We hope that this series on “Docker and the Three Ways of DevOps” has motivated you to “experiment” with Docker in you organization. In the classic PDSA cycle, we would love to hear the results or your “Study” of the effectiveness of Docker in your organization.

 

Read Part 1 and Part 2 in this series


 

Learn More about Docker

 

, ,

John Willis

Docker and the Three Ways of DevOps Part 3: The Third Way – Culture of Continuous Experimentation and Learning


Leave a Reply

Get the Latest Docker News by Email

Docker Weekly is a newsletter with the latest content on Docker and the agenda for the upcoming weeks.