Arnaud Porterie

Docker 1.10 Release Candidate Now Available: Important Tips on Updating Engines and Migrating Images to New IDs

This post describes the upcoming changes to the way Docker Engine stores images and filesystem data in containers. These changes are coming to users starting from version v1.10.

Key points:

  • A more secure foundation for referencing images and layers
  • A new distribution manifest and pull features
  • Upgrading old images will include a migration step
  • A migration tool to minimize migration time

Starting from v1.10 we completely change the way Docker addresses the image data on disk. Previously, every image and layer used a randomly assigned UUID. In 1.10 we implemented a content addressable method using an ID, based on a secure hash of the image and layer data.

The new method gives users more security, provides a built-in way to avoid ID collisions and guarantee data integrity after pull, push, load, or save. It also brings better sharing of layers by allowing many images to freely share their layers even if they didn’t come from the same build.

Addressing images by their content also lets us more easily detect if something has already been downloaded. Because we have separated images and layers, you don’t have to pull the configurations for every image that was part of the original build chain. We also don’t need to create layers for the build instructions that didn’t modify the filesystem.

Content addressability is the foundation for the new distribution features. The image pull and push code has been reworked to use a download/upload manager concept that makes push and pull much more stable and mitigate any parallel request issues. The download manager also brings retries on failed downloads and better prioritization for concurrent downloads.

We are also introducing a new manifest format that is built on top of the content addressable base. It directly references the content addressable image configuration and layer checksums. The new manifest format also makes it possible for a manifest list to be used for targeting multiple architectures/platforms (…more to come on that later). Moving to the new manifest format will be completely transparent.

Preparing for upgrade

To make your current images accessible to the new model we have to migrate them to content addressable storage. This means calculating the secure checksums for your current data.

All your current images, tags and containers are automatically migrated to the new foundation the first time you start updated Docker daemon. Before loading your container, the Docker daemon will calculate all needed checksums for your current data, and after it has completed, all your images and tags will have brand new secure IDs.

While this is simple operation, calculating SHA256 checksums for your files can take time if you have lots of image data. On average you should assume that migrator can process data at a speed of 100MB/s. During this time your Docker daemon won’t be ready to respond to requests.

Minimizing migration time

If you can accept this one time hit, then upgrading Docker Engine and restarting the daemon will transparently migrate your images. However, if you want to minimize the daemon’s downtime, a migration utility can be run while your old daemon is still running.

This program will find all your current images and calculate the checksums for them. After you upgrade and restart your daemon, the checksum data of the migrated images will already exist, freeing the daemon from that computation work. If new images appeared between the migration and the upgrade, those will be processed at time of upgrade to 1.10.

You can get it from https://github.com/docker/v1.10-migrator/releases.

The migration tool can also be run as a Docker image. While running the migrator image you need to expose your Docker data directory to the container. If you use the default path then it would look like docker run --rm -v /var/lib/docker:/var/lib/docker docker/v1.10-migrator. If you use the devicemapper storage driver, you also need to include --privileged to give the tool access to your storage devices.

Docker Engine 1.10 Release Candidate is now available. Try it out and make sure to file any issues and send feedback.


 

Learn More about Docker

 

, ,

Arnaud Porterie

Docker 1.10 Release Candidate Now Available: Important Tips on Updating Engines and Migrating Images to New IDs


3 Responses to “Docker 1.10 Release Candidate Now Available: Important Tips on Updating Engines and Migrating Images to New IDs”

  1. beetree

    Awesome, would be great if you also could explain how to update?

    The naive assumption of experimental.docker.com or test.docker.com does not work, even though it reads in test.docker.com that: `# For test builds (ie. release candidates):
    # ‘curl -fsSL https://test.docker.com/ | sh’`

    Thanks!

    Reply
  2. Maxim Vladimirskiy

    Great job guys, thank you! Could you please also elaborate on how these changes affect the Docker Private Registry? We are running one in house and it would be nice to know if we need to run some sort of image migration before upgrading to 1.10.

    Thanks in advance

    Reply
  3. c7dev

    For access to downloads the release-notes on github:
    https://github.com/docker/docker/releases

    Reply

Leave a Reply

Get the Latest Docker News by Email

Docker Weekly is a newsletter with the latest content on Docker and the agenda for the upcoming weeks.