diff --git a/docs/hpc/07_containers/01_intro.md b/docs/hpc/07_containers/01_intro.md index 116fe28fca..310f60612e 100644 --- a/docs/hpc/07_containers/01_intro.md +++ b/docs/hpc/07_containers/01_intro.md @@ -10,94 +10,43 @@ There are multiple reasons to use Apptainer on the HPC clusters: - **Security**: Apptainer provides a layer of security as it does not require any root access on our clusters. This makes it safer against malware and bad scripts that might jeopardize the outer system. Thus we only support Apptainer on our clusters (there are not other options such as Kubernetes or Docker on our clusters right now). - **Containerization**: Apptainer will run all your images (packaged and prebuilt programs) inside of its containers, each container works like a small vm. They contain all the required environment and files of a single Linux kernel and you don't have to worry about any pre-installation issues. - **Inter-connectivity**: Containers are able to talk to each other, as well as the home system, so while each container has its own small space, they are still a part of a big interconnected structure. Thus enabling you to connect your programs. -- **Accessibility**: Probably the most important feature of all, Apptainer allows you to run your program in 2 to 3 simple steps, as shown below. +- **Accessibility**: Probably the most important feature of all, Apptainer allows you to run your program in 2 to 3 simple steps. -## How to Run an Apptainer Container -There are 3 steps to run an Apptainer container on our clusters: - -:::warning -Running containers on login nodes is not encouraged, as processes may be terminated due to resource limits; please use compute nodes instead. -::: - -### 1. Pull an Image from Docker Hub -```sh -$ apptainer pull docker://godlovedc/lolcow -# image name can be for example docker://godlovedc/lolcow -``` - -![apptainer1](./static/apptainer1.png) +## How to Create an Apptainer Container +So what if you want to create an image from your container and save it for a rainy day? -### 2. Build the Image (Optional) -```sh -$ apptainer build .sif -# the image name can be a local image or an image from a hub -``` +Apptainer documentation has [instructions for building containers](https://apptainer.org/docs/user/latest/build_a_container.html) for your convenience. Please read through them to create your own Apptainer container and package it into an image! -Building an image is optional for most use cases. In many cases, users can directly run containers pulled from Docker Hub without building a local image. +Similarly, you can build docker containers using the information from [Docker's documentation](https://docs.docker.com/get-started/docker-concepts/building-images/). You can then upload them onto docker hub and pull them using Apptainer. Apptainer supports all Docker images! -![apptainer2](./static/apptainer2.png) +## Apptainer vs Docker +Why are there so many mentions of Docker? The reason is that Apptainer is essentially compatible with Docker and you don't need to relearn Apptainer if you already have experience with Docker. Now let's get into some pros and cons between the two programs. +- Docker is more accepted commercially than Apptainer. You can download and run Docker on your own computer with any operating system and build containers with ease while Apptainer is used in a more academic setting. Apptainer primarily supports Linux environments and is designed for HPC use cases. +- However, Docker requires root or admin access for the operating system it deploys on, and our clusters do not offer that access to any software that requires this criteria. Thus Docker is not available on the clusters and Apptainer is. +- A silver lining in all of this is that Apptainer fully supports Docker images and you can do everything in Docker and push your image to Docker Hub and pull them on the clusters. Thus making sure that you don't need to relearn workflows and can just use it through the simplest of commands in this documentation. -You can now run your container using the built image. +## Getting Started +### Common Commands +Pull an image from Docker Hub: -### 3. Run Container ```sh -# this is one way of running a container -$ apptainer run .sif -# this is another way to run a container -$ ./.sif +apptainer pull docker:// ``` -:::info -Apptainer images are immutable by default. You can mount an writable overlay file and edit files within the overlay. -::: - -running this would yield a menu for output: - -![apptainer3](./static/apptainer3.png) - -#### Enter Container +Run a container: ```sh -apptainer shell .sif -# after this step, you will be going into the container and start your programming +apptainer run .sif ``` -![apptainer4](./static/apptainer4.png) - -:::tip - -Run commands outside the container - -You can run commands for the container using exec arguments without actually going into the container - +Execute a command inside a container: ```sh -$ apptainer exec .sif -# adding commands to the back will return the display result of these commands in the container without actually going into the container +apptainer exec .sif ``` -::: - -### Using `fakeroot` - -In some cases, you may need elevated permissions inside the container to install software or modify system files. Apptainer provides a `--fakeroot` option that allows you to run commands inside the container with root-like privileges, without requiring actual root access on the system. +Start an interactive shell: ```sh -$ apptainer exec --fakeroot .sif +apptainer shell .sif ``` -That's it! Now you're good to go and can just use these simple steps to run Apptainer images and run your programs. - For full information and documentation please visit [Apptainer](https://apptainer.org/). - -## How to Create an Apptainer Container -So what if you want to create an image from your container and save it for a rainy day? - -Apptainer documentation has [instructions for building containers](https://apptainer.org/docs/user/latest/build_a_container.html) for your convenience. Please read through them to create your own Apptainer container and package it into an image! - -Similarly, you can build docker containers using the information from [Docker's documentation](https://docs.docker.com/get-started/docker-concepts/building-images/). You can then upload them onto docker hub and pull them using Apptainer. Apptainer supports all Docker images! - -## Apptainer vs Docker -Why are there so many mentions of Docker? The reason is that Apptainer is essentially compatible with Docker and you don't need to relearn Apptainer if you already have experience with Docker. Now let's get into some pros and cons between the two programs. -- Docker is more accepted commercially than Apptainer. You can download and run Docker on your own computer with any operating system and build containers with ease while Apptainer is used in a more academic setting. Apptainer primarily supports Linux environments and is designed for HPC use cases. -- However, Docker requires root or admin access for the operating system it deploys on, and our clusters do not offer that access to any software that requires this criteria. Thus Docker is not available on the clusters and Apptainer is. -- A silver lining in all of this is that Apptainer fully supports Docker images and you can do everything in Docker and push your image to Docker Hub and pull them on the clusters. Thus making sure that you don't need to relearn workflows and can just use it through the simplest of commands in this documentation. - Good luck with Apptainer, and have fun! diff --git a/docs/hpc/07_containers/02_apptainer_tutorial.md b/docs/hpc/07_containers/02_apptainer_tutorial.md new file mode 100644 index 0000000000..0840bf9dcb --- /dev/null +++ b/docs/hpc/07_containers/02_apptainer_tutorial.md @@ -0,0 +1,133 @@ +# Getting Started with Apptainer on Torch + +This tutorial walks you through running your first container on the Torch HPC cluster using Apptainer. By the end of this guide, you will be able to pull a container image, run it, and execute commands inside a containerized environment. + +For background on what Apptainer is and why we use it on Torch, see [Custom Applications with Containers](../containers/). + +## Step 1: Pull a Container Image + +Make sure you are on a compute node before proceeding, as running containers on login nodes is not recommended. + +Pulling an image downloads it from Docker Hub and converts it into an Apptainer `.sif` file that can be run on the cluster. + +```sh +apptainer pull docker://python:3.10 +``` + +This may take a few minutes. You should see progress output like: + +``` +INFO: Converting OCI blobs to SIF format +INFO: Creating SIF file... +[=====================================================================] 100 % 0s +``` + +Once complete, you will find a file called `python_3.10.sif` in your current directory: + +```sh +ls *.sif +``` + +``` +python_3.10.sif +``` + +## Step 2: Run the Container +:::info +Apptainer images are immutable by default. You can mount an writable overlay file and edit files within the overlay. +::: + +Every container image has a default command defined by its creator. You can run it with: + +```sh +apptainer run python_3.10.sif +``` + +For the Python image, this drops you into a Python interactive shell: + +``` +Python 3.10.x (main, ...) [GCC ...] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> +``` + +Type `exit()` to leave the Python shell. + +## Step 3: Execute Commands Inside the Container + +Instead of entering the container interactively, you can run a one-off command using `exec`. This is useful for scripting and batch jobs. + +```sh +apptainer exec python_3.10.sif python -c "print('Hello from container')" +``` + +You should see: + +``` +Hello from container +``` + +The command ran inside the containerized environment, then returned control to your shell. + +### Using `fakeroot` + +In some cases, you may need elevated permissions inside the container to install software or modify system files. Apptainer provides a `--fakeroot` option that allows you to run commands inside the container with root-like privileges, without requiring actual root access on the system. + +```sh +$ apptainer exec --fakeroot .sif +``` + +## Step 4: Run a Simple Scientific Example + +Containers are commonly used in research workflows to run code in a reproducible environment. For example, you can run a NumPy computation without installing NumPy locally: + +```sh +apptainer exec python_3.10.sif python -c "print(list(range(5)))" +``` + +Expected output: + +``` +[0, 1, 2, 3, 4] +``` + +This confirms that the container's Python environment is working correctly. + +## Step 5: Explore the Container Environment + +To explore the container interactively, open a shell inside it: + +```sh +apptainer shell python_3.10.sif +``` + +Your prompt will change to indicate you are now inside the container: + +``` +Apptainer> +``` + +From here you can run commands as you normally would. For example: + +```sh +python --version +``` + +```sh +which python +``` + +When you are done, exit the container shell: + +```sh +exit +``` + +Your prompt will return to the normal Torch shell. + +## Understanding `exec` vs `shell` + +**`exec`** runs a single command you specify inside the container, then exits immediately. You never "enter" the container — control returns to your shell as soon as the command finishes. This makes it ideal for scripts and batch jobs. + +**`shell`** gives you an interactive bash shell inside the container, so you can explore the environment, run multiple commands, and inspect files. Think of it as "stepping inside" the container. +