diff --git a/content/manuals/docker-hub/repos/manage/hub-images/bulk-migrate.md b/content/manuals/docker-hub/repos/manage/hub-images/bulk-migrate.md index df6a9253459f..b2523ea78f78 100644 --- a/content/manuals/docker-hub/repos/manage/hub-images/bulk-migrate.md +++ b/content/manuals/docker-hub/repos/manage/hub-images/bulk-migrate.md @@ -1,74 +1,95 @@ --- -title: Bulk migrate Docker images +title: Bulk migrate images description: Learn how to migrate multiple Docker images and tags between organizations using scripts and automation. -keywords: docker hub, migration, bulk, images, tags, multi-arch, buildx +keywords: docker hub, migration, bulk, images, tags, multi-arch --- This guide shows you how to migrate Docker images in bulk between Docker Hub -organizations or namespaces. Whether you're consolidating repositories, -changing organization structure, or moving images to a new account, these -techniques help you migrate efficiently while preserving image integrity. +organizations or namespaces. Whether you're consolidating repositories, changing +organization structure, or moving images to a new account, these techniques help +you migrate efficiently while preserving image integrity. -## Prerequisites +The topic is structured to build up in scale: -Before you begin, ensure you have: +1. [Migrate a single image tag](#migrate-a-single-image-tag) +2. [Migrate all tags for a repository](#migrate-all-tags-for-a-repository) +3. [Migrate multiple repositories](#migrate-multiple-repositories) -- Docker CLI version 20.10 or later installed -- Docker Buildx (optional but recommended for multi-architecture images) -- Push access to both source and destination organizations -- `jq` installed for JSON parsing in scripts -- `curl` for API calls +The recommended tool for this workflow is `crane`. An equivalent alternative +using `regctl` is also shown. Both tools perform registry-to-registry copies +without pulling images locally and preserve multi-architecture images. -## Authenticate to Docker Hub +`crane` is recommended for its simplicity and focused image-copying workflow. +`regctl` is also a good choice, particularly if you already use it for broader +registry management tasks beyond image copying. -Sign in to Docker Hub to authenticate your session: +> [!NOTE] +> +> The main workflows in this topic operate on tagged images only. Untagged +> manifests or content no longer reachable from tags are not migrated. In +> practice, these are usually unused artifacts, but be aware of this limitation +> before migration. While you can migrate specific untagged manifests using +> [digest references](#migrate-by-digest), there is no API to enumerate untagged +> manifests in a repository. -```console -$ docker login -``` +## Prerequisites + +Before you begin, ensure you have: -Enter your credentials when prompted. This authentication persists for your -session and prevents rate limiting issues. +- One of the following installed and available in your `$PATH`: + - [`crane`](https://github.com/google/go-containerregistry) + - [`regctl`](https://regclient.org/usage/regctl/) +- Push access to both the source and destination organizations +- Registry authentication configured for your chosen tool -## Migrate a single image tag +## Authenticate to registries -The basic workflow for migrating a single image tag involves three steps: pull, -tag, and push. +Both tools authenticate directly against registries: -1. Set your source and destination variables: +- `crane` uses Docker credential helpers and `~/.docker/config.json`. See the + [crane documentation](https://github.com/google/go-containerregistry/tree/main/cmd/crane/doc). +- `regctl` uses its own configuration file and can import Docker credentials. + See the [regctl documentation](https://github.com/regclient/regclient/tree/main/docs). - ```bash - SRC_ORG=oldorg - DEST_ORG=neworg - REPO=myapp - TAG=1.2.3 - ``` +Follow the authentication instructions for your registry and tool of choice. -2. Pull the image from the source organization: +## Migrate a single image tag - ```console - $ docker pull ${SRC_ORG}/${REPO}:${TAG} - ``` +This is the simplest and most common migration scenario. -3. Tag the image for the destination organization: +The following example script copies the image manifest directly between +registries and preserves multi-architecture images when present. Repeat this +process for each tag you want to migrate. Replace the environment variable +values with your source and destination organization names, repository name, and +tag. - ```console - $ docker tag ${SRC_ORG}/${REPO}:${TAG} ${DEST_ORG}/${REPO}:${TAG} - ``` +```bash +#!/usr/bin/env bash +set -euo pipefail -4. Push the image to the destination organization: +SRC_ORG="oldorg" +DEST_ORG="neworg" +REPO="myapp" +TAG="1.2.3" - ```console - $ docker push ${DEST_ORG}/${REPO}:${TAG} - ``` +SRC_IMAGE="${SRC_ORG}/${REPO}:${TAG}" +DEST_IMAGE="${DEST_ORG}/${REPO}:${TAG}" -Repeat these steps for any additional tags you need to migrate, including -`latest` if applicable. +# Using crane (recommended) +crane cp "${SRC_IMAGE}" "${DEST_IMAGE}" -## Migrate all tags for a repository +# Using regctl (alternative) +# regctl image copy "${SRC_IMAGE}" "${DEST_IMAGE}" +``` -To migrate all tags from a single repository, use this script that queries the -Docker Hub API and processes each tag: +### Migrate by digest + +To migrate a specific image by digest instead of tag, use the digest in the +source reference. This is useful when you need to migrate an exact image +version, even if the tag has been updated. Replace the environment variable +values with your source and destination organization names, repository name, +digest, and tag. You can choose between `crane` and `regctl` for the copy +operation. ```bash #!/usr/bin/env bash @@ -77,155 +98,208 @@ set -euo pipefail SRC_ORG="oldorg" DEST_ORG="neworg" REPO="myapp" +DIGEST="sha256:abcd1234..." +TAG="stable" -# Paginate through tags -TAGS_URL="https://hub.docker.com/v2/repositories/${SRC_ORG}/${REPO}/tags?page_size=100" -while [[ -n "${TAGS_URL}" && "${TAGS_URL}" != "null" ]]; do - RESP=$(curl -fsSL "${TAGS_URL}") - echo "${RESP}" | jq -r '.results[].name' | while read -r TAG; do - echo "==> Migrating ${SRC_ORG}/${REPO}:${TAG} → ${DEST_ORG}/${REPO}:${TAG}" - docker pull "${SRC_ORG}/${REPO}:${TAG}" - docker tag "${SRC_ORG}/${REPO}:${TAG}" "${DEST_ORG}/${REPO}:${TAG}" - docker push "${DEST_ORG}/${REPO}:${TAG}" - done - TAGS_URL=$(echo "${RESP}" | jq -r '.next') -done -``` +SRC_IMAGE="${SRC_ORG}/${REPO}@${DIGEST}" +DEST_IMAGE="${DEST_ORG}/${REPO}:${TAG}" -This script automatically handles pagination when a repository has more than -100 tags. +# Using crane +crane cp "${SRC_IMAGE}" "${DEST_IMAGE}" -> [!NOTE] -> -> Docker Hub automatically creates the destination repository on first push if -> your account has the necessary permissions. +# Using regctl +# regctl image copy "${SRC_IMAGE}" "${DEST_IMAGE}" +``` -### Migrate private repository tags +## Migrate all tags for a repository -For private repositories, authenticate your API calls with a Docker Hub access -token: +To migrate every tagged image in a repository, use the Docker Hub API to +enumerate tags and copy each one. The following example script retrieves all +tags for a given repository and migrates them in a loop. This approach scales to +repositories with many tags without overwhelming local resources. Note that +there is a rate limit on Docker Hub requests, so you may need to add delays or +pagination handling for large repositories. -1. Create a personal access token in your - [Docker Hub account settings](https://hub.docker.com/settings/security). +Replace the environment variable values with your source and destination +organization names and repository name. If your source repository is private, +also set `HUB_USER` and `HUB_TOKEN` with credentials that have pull access. You +can also choose between `crane` and `regctl` for the copy operation. -2. Set your credentials as variables: +```bash +#!/usr/bin/env bash +set -euo pipefail - ```bash - HUB_USER="your-username" - HUB_TOKEN="your-access-token" - ``` +# Use environment variables if set, otherwise use defaults +SRC_ORG="${SRC_ORG:-oldorg}" +DEST_ORG="${DEST_ORG:-neworg}" +REPO="${REPO:-myapp}" -3. Modify the `curl` command in the script to include authentication: +# Optional: for private repositories +# HUB_USER="your-username" +# HUB_TOKEN="your-access-token" +# AUTH="-u ${HUB_USER}:${HUB_TOKEN}" +AUTH="" - ```bash - RESP=$(curl -fsSL -u "${HUB_USER}:${HUB_TOKEN}" "${TAGS_URL}") - ``` +TOOL="crane" # or: TOOL="regctl" -> [!IMPORTANT] -> -> If you encounter pull rate or throughput limits, keep `docker login` active -> to avoid anonymous pulls. Consider adding throttling or careful parallelization -> if migrating large numbers of images. +TAGS_URL="https://hub.docker.com/v2/repositories/${SRC_ORG}/${REPO}/tags?page_size=100" -## Migrate multiple repositories +while [[ -n "${TAGS_URL}" && "${TAGS_URL}" != "null" ]]; do + RESP=$(curl -fsSL ${AUTH} "${TAGS_URL}") -To migrate multiple repositories at once, create a list of repository names -and process them in a loop. + echo "${RESP}" | jq -r '.results[].name' | while read -r TAG; do + [[ -z "${TAG}" ]] && continue -1. Create a file named `repos.txt` with one repository name per line: + SRC_IMAGE="${SRC_ORG}/${REPO}:${TAG}" + DEST_IMAGE="${DEST_ORG}/${REPO}:${TAG}" - ```text - api - web - worker - database - ``` + echo "Migrating ${SRC_IMAGE} → ${DEST_IMAGE}" -2. Save the single-repository script from the previous section as - `migrate-single-repo.sh` and make it executable. + case "${TOOL}" in + crane) + crane cp "${SRC_IMAGE}" "${DEST_IMAGE}" + ;; + regctl) + regctl image copy "${SRC_IMAGE}" "${DEST_IMAGE}" + ;; + esac + done -3. Use this wrapper script to process all repositories: + TAGS_URL=$(echo "${RESP}" | jq -r '.next') +done +``` - ```bash - #!/usr/bin/env bash - set -euo pipefail +> [!NOTE] +> +> Docker Hub automatically creates the destination repository on first push if +> your account has permission. - SRC_ORG="oldorg" - DEST_ORG="neworg" +## Migrate multiple repositories - while read -r REPO; do - [[ -z "${REPO}" ]] && continue - echo "==== Migrating repo: ${REPO}" - export REPO - ./migrate-single-repo.sh - done < repos.txt - ``` +To migrate several repositories, create a list and run the single-repository +script for each one. -## Preserve multi-architecture images +For example, create a `repos.txt` file with repository names: -Standard `docker pull` only retrieves the image for your current platform. -For multi-architecture images, this approach loses other platform variants. +```text +api +web +worker +``` -### Use Buildx imagetools (recommended) +Save the script from the previous section as `migrate-single-repo.sh`. Then, run +the following example script that processes each repository in the file. Replace +the environment variable values with your source and destination organization +names. -The recommended approach uses Buildx to copy the complete manifest without -pulling images locally: +```bash +#!/usr/bin/env bash +set -euo pipefail -```console -$ docker buildx imagetools create \ - -t ${DEST_ORG}/${REPO}:${TAG} \ - ${SRC_ORG}/${REPO}:${TAG} -``` +SRC_ORG="oldorg" +DEST_ORG="neworg" -This command copies the source manifest with all platforms directly to the -destination tag. +while read -r REPO; do + [[ -z "${REPO}" ]] && continue + echo "==== Migrating repo: ${REPO}" + SRC_ORG="${SRC_ORG}" DEST_ORG="${DEST_ORG}" REPO="${REPO}" ./migrate-single-repo.sh +done < repos.txt +``` -Verify the migration by inspecting both manifests: +## Verify migration integrity -```console -$ docker buildx imagetools inspect ${SRC_ORG}/${REPO}:${TAG} -$ docker buildx imagetools inspect ${DEST_ORG}/${REPO}:${TAG} -``` +After copying, verify that source and destination match by comparing digests. -Compare the platforms and digests in the output to confirm they match. +### Basic digest verification -### Manual manifest creation +The following example script retrieves the image digest for a specific tag from +both source and destination and compares them. If the digests match, the +migration is successful. Replace the environment variable values with your +source and destination organization names, repository name, and tag. You can +choose between `crane` and `regctl` for retrieving digests. -If you need to use the pull/tag/push workflow for multi-architecture images, -you must pull each platform variant and recreate the manifest using -`docker manifest create` and `docker manifest push`. This approach is slower -and more error-prone than using Buildx imagetools. +```bash +#!/usr/bin/env bash +set -euo pipefail -## Verify migration integrity +SRC_ORG="oldorg" +DEST_ORG="neworg" +REPO="myapp" +TAG="1.2.3" -After migrating images, verify that they transferred correctly. +SRC_IMAGE="${SRC_ORG}/${REPO}:${TAG}" +DEST_IMAGE="${DEST_ORG}/${REPO}:${TAG}" -### Single-architecture images +# Using crane +SRC_DIGEST=$(crane digest "${SRC_IMAGE}") +DEST_DIGEST=$(crane digest "${DEST_IMAGE}") -Compare image digests between source and destination: +# Using regctl (alternative) +# SRC_DIGEST=$(regctl image digest "${SRC_IMAGE}") +# DEST_DIGEST=$(regctl image digest "${DEST_IMAGE}") -```console -$ docker pull ${SRC_ORG}/${REPO}:${TAG} -$ docker inspect --format='{{index .RepoDigests 0}}' ${SRC_ORG}/${REPO}:${TAG} +echo "Source: ${SRC_DIGEST}" +echo "Destination: ${DEST_DIGEST}" -$ docker pull ${DEST_ORG}/${REPO}:${TAG} -$ docker inspect --format='{{index .RepoDigests 0}}' ${DEST_ORG}/${REPO}:${TAG} +if [[ "${SRC_DIGEST}" == "${DEST_DIGEST}" ]]; then + echo "✓ Migration verified: digests match" +else + echo "✗ Migration failed: digests do not match" + exit 1 +fi ``` -The SHA256 digests should match if the migration succeeded. +### Multi-arch verification -### Multi-architecture images +For multi-architecture images, also verify the manifest list to ensure all +platforms were copied correctly. Replace the environment variable values with +your source and destination organization names, repository name, and tag. You +can choose between `crane` and `regctl` for retrieving manifests. -For multi-arch images, compare the output from Buildx imagetools: +```bash +#!/usr/bin/env bash +set -euo pipefail -```console -$ docker buildx imagetools inspect ${SRC_ORG}/${REPO}:${TAG} -$ docker buildx imagetools inspect ${DEST_ORG}/${REPO}:${TAG} +SRC_ORG="oldorg" +DEST_ORG="neworg" +REPO="myapp" +TAG="1.2.3" + +SRC_IMAGE="${SRC_ORG}/${REPO}:${TAG}" +DEST_IMAGE="${DEST_ORG}/${REPO}:${TAG}" + +# Using crane +SRC_MANIFEST=$(crane manifest "${SRC_IMAGE}") +DEST_MANIFEST=$(crane manifest "${DEST_IMAGE}") + +# Using regctl (alternative) +# SRC_MANIFEST=$(regctl image manifest --format raw-body "${SRC_IMAGE}") +# DEST_MANIFEST=$(regctl image manifest --format raw-body "${DEST_IMAGE}") + +# Check if it's a manifest list (multi-arch) +if echo "${SRC_MANIFEST}" | jq -e '.manifests' > /dev/null 2>&1; then + echo "Multi-arch image detected" + + # Compare platform list + SRC_PLATFORMS=$(echo "${SRC_MANIFEST}" | jq -r '.manifests[] | "\(.platform.os)/\(.platform.architecture)"' | sort) + DEST_PLATFORMS=$(echo "${DEST_MANIFEST}" | jq -r '.manifests[] | "\(.platform.os)/\(.platform.architecture)"' | sort) + + if [[ "${SRC_PLATFORMS}" == "${DEST_PLATFORMS}" ]]; then + echo "✓ Platform list matches:" + echo "${SRC_PLATFORMS}" + else + echo "✗ Platform lists do not match" + echo "Source platforms:" + echo "${SRC_PLATFORMS}" + echo "Destination platforms:" + echo "${DEST_PLATFORMS}" + exit 1 + fi +else + echo "Single-arch image" +fi ``` -Verify that the platforms and manifest digest match between source and -destination. - ## Complete the migration After migrating your images, complete these additional steps: @@ -256,4 +330,4 @@ After migrating your images, complete these additional steps: 5. Deprecate the old location: - Update the source repository description to point to the new location - Consider adding a grace period before making the old repository private or - read-only + read-only \ No newline at end of file