[BUG] `--no-interpolate` flag on `docker compose config` has unexpected side effects

[Paraphraser]

Description

Given the example compose file under "Steps To Reproduce", the result of running:

$ docker compose config --no-normalize --no-path-resolution example1

is:

name: demo
services:
  example1:
    container_name: example1
    environment:
      TZ: Australia/Sydney
    image: example/example
    ports:
      - mode: ingress
        target: 3000
        published: "3001"
        protocol: tcp
    restart: unless-stopped
    volumes:
      - type: bind
        source: ./volumes/example1/data
        target: /var/lib/data
        bind: {}

I have chosen:

• --no-normalize because it removes references to networks; and
• --no-path-resolution because it maintains project-relative external paths in volumes: clauses.

The output shows TZ with the value Australia/Sydney (from the .env file). My goal is that that should be displayed in its original form as TZ: ${TZ:-Etc/UTC}.

The --no-interpolate flag is defined as "Don't interpolate environment variables" and seems like a reasonable candidate for achieving this goal. The result of adding that flag is shown in the screen shot in the next section.

The --no-interpolate flag has achieved the expected result with respect to TZ but it has also produced a bunch of unexpected side effects, ranging from relatively unimportant through problematic, none of which would seem to have anything to do with environment variables.

For what I am trying to achieve, the most problematic difference is that the service definition for example2 has appeared in the output. This behaviour suggests that the presence of --no-interpolate causes config to ignore the [SERVICE...] argument.

I assume the name: demo (first line on the left hand side of the screen shot) is the project name. That line disappearing is actually beneficial for what I am trying to achieve but it isn't immediately clear what this has to do with environment variables.

The reverse-ordering of the last three lines of the ports: clause doesn't matter for my purposes. The same applies to the volumes: clause where the ordering and structural differences are just … strange.

Steps To Reproduce

1. Example compose file:

   ---
   
   services:
   
     example1:
       container_name: example1
       image: example/example
       restart: unless-stopped
       ports:
         - "3001:3000"
       environment:
         TZ: ${TZ:-Etc/UTC}
       volumes:
         - ./volumes/example1/data:/var/lib/data
   
     example2:
       container_name: example2
       image: example/example
       restart: unless-stopped
       ports:
         - "3002:3000"
       environment:
         TZ: ${TZ:-Etc/UTC}
       volumes:
         - ./volumes/example2/data:/var/lib/data
   
   ...

2. Run:

   $ diff -y \
     <(docker compose config --no-normalize --no-path-resolution                  example1) \
     <(docker compose config --no-normalize --no-path-resolution --no-interpolate example1)

3. The result is:

   

Compose Version

$ docker compose version
Docker Compose version v5.1.0

$ docker-compose version
Docker Compose version v5.1.0

Docker Environment

$ docker info
Client: Docker Engine - Community
 Version:    29.2.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 6
  Running: 6
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 29.2.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: local
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.21-v8+
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.704GiB
 Name: sec-dev
 ID: f6406dbb-2fc1-4834-9606-5e826871aec6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Anything else?

Older cases raising issues with --no-interpolate (eg #13327, #12292, #12079, #11547) all seem to use examples where the compose file only contains a single service definition. While that scenario does show the unexpected re-ordering of clauses that would not seem to be related to environment variables, it won't reveal the issue with the [SERVICE...] argument being ignored.

[ndeloof]

When using --no-interpolate compose parse compose.yaml but does minimal processing of the compose model (which may be only partially resolved). This typically means it doesn't computes services dependencies, and as such the SERVICES command line option can't be processed the same way as the plain command.

Basically, --no-interpolate is a reminiscence from the past and brings many issues/limitations. I wish we just had dropped this feature from compose as we implemented compose v2.

Please let me know what you're trying to achieve relying on this command

[Paraphraser]

Well, it's a long(ish) story. I'm working on a third menu system for IOTstack. One of the key goals is to implement an apt-like update/upgrade paradigm so that changes to service definitions applied to the GitHub repo have some chance of making it through to active user compose files.

With the first two menu systems, users basically had to "just know" that something had changed on GitHub. Most users didn't bother, with the result that when something broke which had already been fixed on GitHub, there was usually a flurry of issues where the fix advice boiled down to "uninstall and reinstall the affected service, and you're on your own for re-applying any customisations".

To try to ensure that everyone gets up-to-date as soon as possible, part of the design involves a "migration" step where I will iterate the output of docker compose config --services (running against a users existing compose file) to figure out which services the user is actually using, and then "install" those services where "install" essentially means to start using the current template for each such service.

For any given service, then, the "service definition in the current template" and the "service definition in the user's existing compose file" may be different, meaning that when the user "builds" the post-migration stack by assembling current templates into a new compose file, he/she may suddenly get different behaviour. Although the previous compose file will still be there post-migration, it would be handy to be able to save the existing "before" service definition for each migrated service, which is where I was looking at something (roughly) like this:

for S in $(docker compose config --services) ; do
  docker compose config \
    --no-normalize \
    --no-path-resolution \
    --no-interpolate \
    -o $S-baseline.yml $S
done

If there's another/better way of achieving roughly the same outcome, I'm happy to try it.

---

Can I ask you a follow-on question, please? As noted in my original post, --no-interpolate is very tightly defined in the documentation as only affecting environment variables. I didn't necessarily expect its effects to be confined to elements of an environment: clause. If, say, a devices: element referenced an environment variable, I would have expected that to have been covered by the definition too. However, the "minimal processing" you mention is not something I would have predicted from the documentation. Is there some reason why it has to be like that? In other words, couldn't --no-interpolate just behave as documented? Alternatively, a new flag like --no-expand-environment-vars would probably do the trick without interfering with the existing code.

[ndeloof]

To interact with compose.yaml files I'd recommend you don't rely on docker compose command line but use compose-go and use the processed application model.

I didn't necessarily expect its effects to be confined to elements of an environment: clause

I don't get your point. --no-interpolate let compose-go parse yaml files but doesn't interpolate (i.e. "expand variables") values, which also means short notation strings aren't parsed into actual model. The result is a plain yaml tree, and not a compose-go data model where advanced filtering (like: select a subset of services and dependencies) can't be processed.

[Paraphraser]

To interact with compose.yaml files I'd recommend you don't rely on docker compose command line but use compose-go and use the processed application model.

I accept your advice but, in the IOTstack context, it's not really practical to be compiling Go programs on each user's machine, nor to be adding precompiled assets to the project for download onto user machines. At least not for this particular scenario where "migration" only occurs once.

The message I'm taking from both of your thoughtful replies is that "I can't get there from here." Good to know. The idea of being able to snapshot each pre-migration service definition was only ever a "nice to have". If it isn't going to work then I can live without it.

I don't get your point.

I began with the documentation:

I took "environment variables" to mean anything in a compose file represented by ${expression} and, in that context, I took "interpolate" to mean the process of evaluating expression and then replacing ${expression} with the result of the evaluation.

As I noted yesterday, given something like this:

  zigbee2mqtt:
    ...
    environment:
      TZ: ${TZ:-Etc/UTC}
    devices:
      - "${ZIGBEE2MQTT_DEVICE_PATH:?eg echo ZIGBEE2MQTT_DEVICE_PATH=/dev/ttyACM0 >>~/IOTstack/.env}:/dev/ttyACM0"
    ...

I see two "environment variables":

• ${TZ:-Etc/UTC}
• ${ZIGBEE2MQTT_DEVICE_PATH:?eg echo ZIGBEE2MQTT_DEVICE_PATH=/dev/ttyACM0 >>~/IOTstack/.env}

What the words "don't interpolate environment variables" meant to me was that those two "environment variables" would come back "as is" in their ${expression} form, rather than in their interpolated form.

For me, the greatest surprise was all the other changes in the original screen shot. To me, it was all those other changes which meant that docker compose was not behaving according to its documentation, which is why I filed a bug report.

Maybe the documentation should be amended to explain what --no-interpolate actually does ("short notation strings aren't parsed" etc) and how the option is mutually exclusive with [SERVICE...] arguments. Some words about how the option is a not-quite-deprecated pre v2 holdover would also guide readers (like me) away from trying to use it in the first place.