cloud-importer

This is a small tool to import and manage private images to cloud providers. It basically automate (and optimize) the commands you would need to run otherwise to import an image as a image on a cloud provider.

In addition to the import command it offers a share command to allow share images accross accounts. Initially the image imported is private to the account which run the tool. In order to allow to use the image from a different account we would need to run the share command.

Prerequisites

Before you begin, ensure you have the following:

• Cloud Account: An active AWS or Azure account
• Cloud Credentials:
  • AWS Credentials: Your AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION must be configured as environment variables.
  • Azure Credentials: Your ARM_CLIENT_ID, ARM_CLIENT_SECRET, ARM_SUBSCRIPTION_ID and ARM_LOCATION_NAME must be configured as environment variables.

Params

There are several params common to all offerings:

• replicate: Indicate the image will be replicated to any possible region on the provider
• org-id: Identifies the top level organization to share the images with

RHEL AI

In order to test RHEL AI on AWS we need to import the image according to RHEL AI installation guide this tool will run those steps for us. Alhough previously the raw image should be donwloaded by an authenticated user to agree with EULA License.

AWS

To run the tool we can use the OCI container:

podman run --rm --name import-rhelai -d \
    -v ${PWD}:/workspace:z \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} \
    ghcr.io/mapt-oss/cloud-importer:latest rhelai aws \
        --project-name "rhelai3-136d47d1" \
        --backed-url s3://bucket/folder \
        --image-name rhelai3-136d47d1 \
        --image-path "/workspace/rhel-ai-nvidia-aws-1.5-1747399384-x86_64.raw" \ 
        --share-orgs-ids arn:aws:organizations::XXXXX:organization/XXXXX,arn:aws:organizations::XXXXX:organization/XXXX1 \
        --replicate \
        --debug \
        --debug-level 9

podman logs -f import-rhelai

Azure

To import a VHD image to Azure, you can use the rhelai azure command. This command will create a storage account, upload the VHD, and create a VM image.

To run the tool we can use the OCI container:

podman run --rm --name import-rhelai-azure -d \
    -v ${PWD}:/workspace:z \
    -e AZURE_TENANT_ID=${AZURE_TENANT_ID} \
    -e AZURE_CLIENT_ID=${AZURE_CLIENT_ID} \
    -e AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET} \
    -e AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID} \
    -e AZURE_LOCATION=${AZURE_LOCATION} \
    ghcr.io/mapt-oss/cloud-importer:latest rhelai az \
        --project-name "rhelai3-136d47d1" \
        --backed-url azblob://blobcontainer/folder \
        --image-name rhelai3-136d47d1 \
        --image-path "/workspace/rhel-ai-nvidia-aws-1.5-1747399384-x86_64.vhd" \ 
        --share-orgs-ids tenanId1,tenantId2 \
        --replicate \
        --debug \
        --debug-level 9

podman logs -f import-rhelai-azure

SNC (Openshift Local)

In order to run Openshift Local we need to transform the bundle generated by snc then the image will be uploaded as cloud provider image and will be available to create ephemeral clusters.

To run the tool we can use the OCI container:

AWS

podman run --rm --name import-openshift-local -d \
    -v ${PWD}:/workspace:z \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} \
    ghcr.io/mapt-oss/cloud-importer:latest snc aws \
          --project-name "snc-4.20.0" \
          --backed-url s3://bucket/folder \
          --bundle-uri ${BUNDLE_URL} \
          --shasum-uri ${SHASUM_URL} \
          --arch ${ARCH} \
          --replicate \
          --share-orgs-ids arn:aws:organizations::XXXXX:organization/XXXXX,arn:aws:organizations::XXXXX:organization/XXXX1 \
          --debug \
          --debug-level 9

Azure

podman run --rm --name import-openshift-local -d \
    -v ${PWD}:/workspace:z \
    -e ARM_CLIENT_ID=${ARM_CLIENT_ID} \
    -e ARM_CLIENT_SECRET=${ARM_CLIENT_SECRET} \
    -e ARM_SUBSCRIPTION_ID=${ARM_SUBSCRIPTION_ID} \
    -e ARM_LOCATION_NAME=${ARM_LOCATION_NAME} \
    ghcr.io/mapt-oss/cloud-importer:latest snc az \
          --project-name "snc-4.20.0" \
          --backed-url azblob://blobcontainer/folder \
          --bundle-uri ${BUNDLE_URL} \
          --shasum-uri ${SHASUM_URL} \
          --arch ${ARCH} \
          --replicate \
          --share-orgs-ids tenanId1,tenantId2 \
          --debug \
          --debug-level 9

Destroy

Imported images are now controlled by a remote tfstate in order to destroy them there is a common destroy function and it depedns on the provider to run it (set the rigt credentials)

AWS

podman run --rm --name import-openshift-local -d \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} \
    ghcr.io/mapt-oss/cloud-importer:latest destroy \
          --project-name "snc-4.20.0" \
          --backed-url s3://bucket/folder 

Azure

podman run --rm --name import-openshift-local -d \
    -e ARM_CLIENT_ID=${ARM_CLIENT_ID} \
    -e ARM_CLIENT_SECRET=${ARM_CLIENT_SECRET} \
    -e ARM_SUBSCRIPTION_ID=${ARM_SUBSCRIPTION_ID} \
    -e ARM_LOCATION_NAME=${ARM_LOCATION_NAME} \
    ghcr.io/mapt-oss/cloud-importer:latest destroy \
          --project-name "snc-4.20.0" \
          --backed-url azblob://blobcontainer/folder 

Troubleshooting

cloud-importer performs the following steps:

1. Bundle Download:

• The tool first downloads the OpenShift Local bundle and its checksum from the provided URLs
  • Linux (libvirt) bundle which has the qcow2 image is easier to convert to RAW or VHD
• It then verifies the integrity of the downloaded bundle using the checksum
• Troubleshooting: If you encounter errors at this stage, double-check the --bundle-url and --shasum-url values

2. Disk Extraction:

• Extract and convert disk image to cloud provider expected format:
  • Decompression: The downloaded bundle (.xz archive with zstd compression) is uncompressed and files are extracted
  • Image Location: The tool locates the qcow2 disk image within the extracted files
  • Image Conversion: AWS requires the disk image to be in .raw format and for Azure it should be in .vhd format
• Troubleshooting:
  • Corrupted Archive: An error during decompression could indicate a corrupted download. Try removing the local bundle and running the tool again
  • Disk Space: Ensure it has sufficient free space to store both the downloaded bundle and the extracted disk image (~ 60GB)

3. Upload to Cloud Provider storage (S3, blob storage):

• The prepared disk image is uploaded to an S3 bucket for AWS or a Storage blob for Azure, cloud-importer creates temporary resources for this purpose
• Troubleshooting:
  • Authentication: Ensure your cloud provider credentials are correct and have the necessary permissions

4. Disk Image Import:

• AWS: The tool initiates a VM import task, pointing to the uploaded disk image in S3. This process converts the disk image into an EBS snapshot
• Azure: The tool creates a Compute Gallery then a Gallery Image Definition, after which an Image Version pointing to the Blob storage containing the disk image

• AMI/Disk Image Creation: Once the snapshot/Galley Image Definition is created, it can be used to register a new AMI for AWS or Image Version for Azure in your account

• Troubleshooting:
  • IAM Role: The VM import process requires a specific IAM role (e.g., vmimport). If this role doesn't exist or lacks the necessary permissions, cloud-importer will attempt to create this role for you
  • Permissions: Your AWS user needs permissions for EC2 VM import (ec2:ImportSnapshot, ec2:DescribeImportSnapshotTasks)