Skip to content

Update IBM Code Engine Backend #1477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JosepSampe merged 5 commits into from
Jun 18, 2026
Merged

Update IBM Code Engine Backend #1477

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6b5aaf4
Update code_engine backend
JosepSampe Jun 16, 2026
aeda12b
Update
JosepSampe Jun 16, 2026
a8a4420
Update
JosepSampe Jun 16, 2026
dd27b1e
Update config
JosepSampe Jun 16, 2026
46ff496
Update
JosepSampe Jun 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
- [AWS EC2] Updated default Ubuntu Image to Ubuntu 24
- [Azure VMS] Updated default Ubuntu Image to Ubuntu 24
- [Aliyun FC] Updated backend to Function Compute 3.0 (FC3 API) and added custom-container deploy mode support
- [Code Engine] Rewrote backend to use the IBM Code Engine SDK v2 (`CodeEngineV2`) instead of the Kubernetes API

### Fixed
- [K8s] Fixed default runtime builds impacted by Debian Buster end-of-life.
Expand Down
32 changes: 15 additions & 17 deletions config/config_template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,9 +259,9 @@
# IBM Cloud – credentials shared by all ibm_* / code_engine sections
# =============================================================================
#ibm:
#iam_api_key: <IAM_API_KEY>
#region: <REGION> # One of: eu-gb, eu-de, us-south, us-east, br-sao, ca-tor, jp-tok, jp-osa, au-syd
#resource_group_id: <RESOURCE_GROUP_ID>
#iam_api_key: <IAM_API_KEY> # Mandatory for code_engine
#region: <REGION> # One of: eu-gb, eu-de, eu-es, us-south, us-east, br-sao, ca-tor, jp-tok, jp-osa, au-syd
#resource_group_id: <RESOURCE_GROUP_ID> # Mandatory for code_engine (auto-create project)

# IBM Cloud Functions – DEPRECATED serverless compute
#ibm_cf:
Expand All @@ -283,20 +283,17 @@
#code_engine:
#region: <REGION> # Falls back to ibm.region
#project_name: <PROJECT_NAME> # Existing Code Engine project (auto-created otherwise)
#namespace: <NAMESPACE> # Alternative to project_name
#kubecfg_path: <PATH_TO_KUBECONFIG>
#docker_server: docker.io
#docker_user: <REGISTRY_USERNAME>
#docker_user: <REGISTRY_USERNAME> # Required for private container registries
#docker_password: <REGISTRY_TOKEN>
#docker_namespace: <REGISTRY_NAMESPACE>
#runtime: <RUNTIME_NAME>
#runtime_cpu: 0.125 # See valid combinations in the IBM docs
#runtime_memory: 256 # MB
#docker_namespace: <REGISTRY_NAMESPACE> # Required for IBM Container Registry (icr.io)
#runtime: <RUNTIME_NAME> # Docker image name. Default: auto-built image
#runtime_cpu: 0.125 # vCPU. See valid combinations in the IBM docs
#runtime_memory: 256 # MB. See valid combinations in the IBM docs
#runtime_timeout: 600 # Seconds
#max_workers: 1000
#worker_processes: 1
#connection_retries: <RETRIES>
#runtime_include_function: False
#connection_retries: <RETRIES> # API retries on HTTP 500 errors

# IBM VPC – standalone compute
#ibm_vpc:
Expand Down Expand Up @@ -331,12 +328,13 @@

# IBM Cloud Object Storage
#ibm_cos:
#region: <REGION> # Or set `endpoint` directly
#region: <REGION> # Recommended. Auto-sets endpoint and worker endpoint
#endpoint: <PUBLIC_ENDPOINT> # https://s3.<region>.cloud-object-storage.appdomain.cloud
#private_endpoint: <PRIVATE_ENDPOINT> # Required for ibm_cf, code_engine, ibm_vpc compute backends
#api_key: <API_KEY> # COS service API key
#service_instance_id: <SERVICE_INSTANCE_ID>
#access_key_id: <ACCESS_KEY_ID> # HMAC credentials (alternative to api_key)
#private_endpoint: <WORKER_ENDPOINT> # Auto-set from region when backend is code_engine or ibm_vpc
# # https://s3.direct.<region>.cloud-object-storage.appdomain.cloud
#api_key: <API_KEY> # COS service API key (Option 1)
#service_instance_id: <SERVICE_INSTANCE_ID> # Required if not using HMAC credentials or api_key
#access_key_id: <ACCESS_KEY_ID> # HMAC credentials (Option 2). Bucket can be auto-created
#secret_access_key: <SECRET_ACCESS_KEY>
#storage_bucket: <BUCKET_NAME> # Mandatory unless HMAC credentials allow auto-creation

Expand Down
32 changes: 22 additions & 10 deletions docs/source/compute_config/code_engine.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ python3 -m pip install lithops[ibm]

2. Click `Create an IBM Cloud API Key` and provide the necessary information.

3. Copy the generated IAM API key (You can only see the key the first time you create it, so make sure to copy it).
3. Copy the generated IAM API key (you can only see the key the first time you create it, so make sure to copy it).

4. Navigate to the [resource groups dashboard](https://cloud.ibm.com/account/resource-groups), and copy the desired resource group ID.

Expand All @@ -40,27 +40,25 @@ python3 -m pip install lithops[ibm]
|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|ibm | iam_api_key | |yes | IBM Cloud IAM API key to authenticate against IBM services. Obtain the key [here](https://cloud.ibm.com/iam/apikeys) |
|ibm | region | |yes | IBM Region. One of: `eu-gb`, `eu-de`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd` |
|ibm | resource_group_id | | yes | Resource group id from your IBM Cloud account. Get it from [here](https://cloud.ibm.com/account/resource-groups) |
|ibm | region | |yes | IBM Region. One of: `eu-gb`, `eu-de`, `eu-es`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd` |
|ibm | resource_group_id | | yes | Resource group id from your IBM Cloud account. Get it from [here](https://cloud.ibm.com/account/resource-groups). Required to auto-create a Code Engine project |

### Code Engine:

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|code_engine | project_name | |no | Project name that already exists in Code Engine. If not provided, Lithops will automatically create a new project|
|code_engine | namespace | |no | As an alternative to `project_name`, you can provide `namespace`. Get it from your Code Engine k8s config file.|
|code_engine | region | | no | Cluster region. One of: `eu-gb`, `eu-de`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd`. Lithops will use the `region` set under the `ibm` section if it is not set here |
|code_engine | project_name | |no | Name of an existing Code Engine project. If not provided, Lithops creates `lithops-<region>-<user_key>` automatically |
|code_engine | region | | no | Code Engine region. One of: `eu-gb`, `eu-de`, `eu-es`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd`. Lithops uses the `region` set under the `ibm` section if not set here |
|code_engine | docker_server | docker.io |no | Container registry URL |
|code_engine | docker_user | |no | Container registry user name |
|code_engine | docker_password | |no | Container registry password/token. For Docker Hub, log in to your Docker Hub account and generate a new access token [here](https://hub.docker.com/settings/security)|
|code_engine | max_workers | 1000 | no | Max number of workers per `FunctionExecutor()`|
|code_engine | worker_processes | 1 | no | Number of Lithops processes within a given worker. This can be used to parallelize function activations within a worker. It is recommended to set this value to the same number of CPUs as the container. |
|code_engine | runtime | |no | Docker image name.|
|code_engine | runtime_cpu | 0.125 |no | CPU limit. Default 0.125vCPU. See [valid combinations](https://cloud.ibm.com/docs/codeengine?topic=codeengine-mem-cpu-combo) |
|code_engine | runtime_memory | 256 |no | Memory limit in MB. Default 256Mi. See [valid combinations](https://cloud.ibm.com/docs/codeengine?topic=codeengine-mem-cpu-combo) |
|code_engine | runtime_cpu | 0.125 |no | CPU limit. Default 0.125 vCPU. See [valid combinations](https://cloud.ibm.com/docs/codeengine?topic=codeengine-mem-cpu-combo) |
|code_engine | runtime_memory | 256 |no | Memory limit in MB. Default 256 MB. See [valid combinations](https://cloud.ibm.com/docs/codeengine?topic=codeengine-mem-cpu-combo) |
|code_engine | runtime_timeout | 600 |no | Runtime timeout in seconds. Default 600 seconds |
|code_engine | connection_retries | |no | If specified, number of job invoke retries in case of connection failure with error code 500 |
|code_engine | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (on the order of tens of MB) and the code does not change frequently |
|code_engine | connection_retries | |no | If specified, number of API call retries in case of connection failure with error code 500 |


## Runtime
Expand Down Expand Up @@ -121,3 +119,17 @@ You can view the function executions logs in your local machine using the *litho
```bash
lithops logs poll
```

## Clean up resources

Delete Lithops runtimes, job runs, and cached metadata:

```bash
lithops clean -b code_engine -s ibm_cos
```

To also delete the Code Engine project and local cache:

```bash
lithops clean -b code_engine -s ibm_cos --all
```
78 changes: 44 additions & 34 deletions docs/source/storage_config/ibm_cos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,19 @@ python3 -m pip install lithops[ibm]

## Configuration

1. Create the credentials to access to your COS account (Choose one option):
Choose one authentication option below.

### Option 1 (COS API Key):

2. In the side navigation, click `Service Credentials`.
1. In the side navigation, click `Service Credentials`.

3. Click `New credential +` and provide the necessary information.
2. Click `New credential +` and provide the necessary information.

4. Click `Add` to generate service credential.
3. Click `Add` to generate service credential.

5. Click `View credentials` and copy the *apikey* value.
4. Click `View credentials` and copy the *apikey* value.

6. Edit your Lithops config file and add the following keys:
5. Edit your Lithops config file and add the following keys:

```yaml
lithops:
Expand All @@ -42,19 +42,19 @@ python3 -m pip install lithops[ibm]

### Option 2 (COS HMAC credentials):

2. In the side navigation, click `Service Credentials`.
1. In the side navigation, click `Service Credentials`.

3. Click `New credential +`.
2. Click `New credential +`.

4. Click on advanced options and enable `Include HMAC Credential` button.
3. Click on advanced options and enable `Include HMAC Credential` button.

5. Click `Add` to generate service credential.
4. Click `Add` to generate service credential.

6. Click `View credentials` and copy the *access_key_id* and *secret_access_key* values.
5. Click `View credentials` and copy the *access_key_id* and *secret_access_key* values.

7. When using HMAC credentials, you can omit providing a storage bucket, since Lithops will be able to create it automatically.
6. When using HMAC credentials, you can omit providing a storage bucket, since Lithops will be able to create it automatically.

8. Edit your Lithops config file and add the following keys:
7. Edit your Lithops config file and add the following keys:

```yaml
lithops:
Expand All @@ -63,19 +63,19 @@ python3 -m pip install lithops[ibm]
ibm_cos:
region: <REGION>
access_key_id: <ACCESS_KEY_ID>
secret_access_key: <SECRET_ACCESS_KEY_ID>
secret_access_key: <SECRET_ACCESS_KEY>
```


### Option 3 (IBM IAM API Key):

2. If you don't have an IAM API key created, navigate to the [IBM IAM dashboard](https://cloud.ibm.com/iam/apikeys)
1. If you don't have an IAM API key created, navigate to the [IBM IAM dashboard](https://cloud.ibm.com/iam/apikeys)

3. Click `Create an IBM Cloud API Key` and provide the necessary information.
2. Click `Create an IBM Cloud API Key` and provide the necessary information.

4. Copy the generated IAM API key (You can only see the key the first time you create it, so make sure to copy it).
3. Copy the generated IAM API key (you can only see the key the first time you create it, so make sure to copy it).

5. Edit your Lithops config file and add the following keys:
4. Edit your Lithops config file and add the following keys:

```yaml
lithops:
Expand All @@ -91,30 +91,40 @@ python3 -m pip install lithops[ibm]

## Lithops COS Endpoint configuration

### Using region
The easiest approach is to let Lithops choose the right endpoint by itself. To enable this, just configure Lithops with the region name of your `storage_bucket`, as follows:
### Using region (recommended)

The easiest approach is to let Lithops choose the right endpoint by itself. Configure Lithops with the region name of your `storage_bucket`:

```yaml
ibm_cos:
region : <REGION>
```

Valid region names are: `us-east`, `us-south`, `eu-gb`, `eu-de`, etc.
Valid region names include: `us-east`, `us-south`, `eu-gb`, `eu-de`, `eu-es`, `ca-tor`, `br-sao`, `jp-tok`, `jp-osa`, `au-syd`.

When `region` is set, Lithops automatically configures:

- **Public endpoint** (client): `https://s3.<region>.cloud-object-storage.appdomain.cloud`
- **Worker endpoint** (`code_engine` and `ibm_vpc`): `https://s3.direct.<region>.cloud-object-storage.appdomain.cloud`

You do not need to set `endpoint` or `private_endpoint` manually when using `region` with these backends.

### Using endpoint paths

As an alternative to using `region`, you can configure the public and private endpoints as follows:
As an alternative to using `region`, you can configure the public and worker endpoints explicitly:

1. Login to IBM Cloud and open up your dashboard. Then navigate to your instance of Object Storage.

2. In the side navigation, click `Endpoints` to find your COS endpoints. You must copy both the `public` and `private` endpoints of the region where you created your bucket.
2. In the side navigation, click `Endpoints` to find your COS endpoints. Copy the endpoint for the region where you created your bucket.

```yaml
ibm_cos:
endpoint: https://s3.<region>.cloud-object-storage.appdomain.cloud
private_endpoint: https://s3.private.<region>.cloud-object-storage.appdomain.cloud
private_endpoint: https://s3.direct.<region>.cloud-object-storage.appdomain.cloud
```

Use the **direct** endpoint (`s3.direct`) as `private_endpoint` when running Lithops with `code_engine` or `ibm_vpc`.


## Summary of configuration keys for IBM Cloud:

Expand All @@ -123,18 +133,18 @@ As an alternative to using `region`, you can configure the public and private en
|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|ibm | iam_api_key | |no | IBM Cloud IAM API key to authenticate against IBM services. Obtain the key [here](https://cloud.ibm.com/iam/apikeys) |
|ibm | region | |no | IBM Region. One of: `eu-gb`, `eu-de`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd` |
|ibm | region | |no | IBM Region. One of: `eu-gb`, `eu-de`, `eu-es`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd` |
|ibm | resource_group_id | | no | Resource group id from your IBM Cloud account. Get it from [here](https://cloud.ibm.com/account/resource-groups) |

### IBM Cloud Object Storage:

| Group | Key |Default|Mandatory| Additional info |
|---------|---------------------|---|---|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ibm_cos | region | |yes | Region of your bucket. One of: `eu-gb`, `eu-de`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd`. Lithops will use the region set under the `ibm` section if it is not set here |
| ibm_cos | api_key | |yes | API Key to your COS account. Not needed if using IAM API Key |
| ibm_cos | storage_bucket | | yes | The name of a bucket that exists in your account. This will be used by Lithops for intermediate data. You must provide HMAC credentials if you want the bucket to be automatically created |
| ibm_cos | service_instance_id | |no | The service instance (CRN format) of your COS instance. **Mandatory** if no HMAC Credentials provided. | | | | |
| ibm_cos | access_key_id | |no | HMAC Credentials. **Mandatory** if no api_key. Not needed if using IAM API Key |
| ibm_cos | secret_access_key | |no | HMAC Credentials. **Mandatory** if no api_key. Not needed if using IAM API Key |
| ibm_cos | endpoint | |no | Endpoint to your COS account. **Mandatory** if no region. Make sure to use the full path with 'https://' as prefix |
| ibm_cos | private_endpoint | |no | Private endpoint to your COS account. **Mandatory** if no region. Make sure to use the full path with 'https://' or 'http://' as prefix |
| ibm_cos | region | |yes* | Region of your bucket. One of: `eu-gb`, `eu-de`, `eu-es`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd`. Lithops uses the region set under the `ibm` section if it is not set here. *Not required if `endpoint` is set |
| ibm_cos | api_key | |no | API Key to your COS account. Required for Option 1 |
| ibm_cos | storage_bucket | |yes* | Bucket used by Lithops for intermediate data. *Can be auto-created when using HMAC credentials |
| ibm_cos | service_instance_id | |no | The service instance (CRN format) of your COS instance. Required if neither HMAC credentials nor `api_key` are provided |
| ibm_cos | access_key_id | |no | HMAC credentials. Required for Option 2 |
| ibm_cos | secret_access_key | |no | HMAC credentials. Required for Option 2 |
| ibm_cos | endpoint | |no | Public endpoint to your COS account. Auto-set from `region` if not provided. Must start with `https://` |
| ibm_cos | private_endpoint | |no | Worker endpoint for compute backends. Auto-set from `region` for `code_engine` and `ibm_vpc`. Must start with `https://` or `http://` |
Loading
Loading