bake: a shared target is serialized once per path in the solve request — exponential in FROM-chain depth (16 MB cap)

[davireis]

Summary

When bake targets share a common base via contexts = { base = "target:base" }, the
base's LLB is inlined into the consumer's definition once for every distinct path
from the base to the target being built. The copies are byte-identical and travel in a
single gRPC message, so the build dies with
ResourceExhausted: trying to send message larger than max (… vs. 16777216) before
anything is built.

Path count, not node count, is what grows. A flat fan-in of N arms costs N copies. But
a FROM-chained tree that fans out 2 ways per level and merges once at the top costs
2^depth copies — three levels of a shared toolchain stage is eight copies of it in one
message. Measured below at exactly 4× and 8×.

The graph is a DAG and buildkit already content-addresses ops, so the daemon would
collapse these on arrival — there is just no way to express the sharing on the wire:
each entry of FrontendInputs is a self-contained pb.Definition carrying its full
transitive closure.

Reproduction

repro.sh <arms> — one shared base, N arms each FROM base, one top that
COPY --from=s every arm. The base is padded with fat-but-instant ops (the padding
rides inside each RUN's command string, so it lands in the marshaled op); RUN false
comes first so a run that slips under the cap aborts instead of building 220 layers.

#!/usr/bin/env bash
set -u
ARMS=${1:-2}
DIR=$(mktemp -d); cd "$DIR" || exit 1

{
  echo "FROM alpine"
  echo "RUN false"
  for i in $(seq 220); do
    printf 'RUN true #%06d%s\n' "$i" "$(head -c 9000 /dev/zero | tr '\0' 'x')"
  done
} > Dockerfile.bottom

echo 'FROM alpine' > Dockerfile.top
cat > bake.hcl <<'EOF'
target "bottom" {
  context = "."
  dockerfile = "Dockerfile.bottom"
}
EOF
for a in $(seq "$ARMS"); do
  printf 'FROM bottom\nRUN true # arm%s\n' "$a" > "Dockerfile.arm$a"
  cat >> bake.hcl <<EOF
target "arm$a" {
  context = "."
  dockerfile = "Dockerfile.arm$a"
  contexts = { bottom = "target:bottom" }
}
EOF
  echo "COPY --from=arm$a /etc/alpine-release /r$a" >> Dockerfile.top
done
{
  echo 'target "top" {'
  echo '  context = "."'
  echo '  dockerfile = "Dockerfile.top"'
  printf '  contexts = { '
  for a in $(seq "$ARMS"); do printf 'arm%s = "target:arm%s", ' "$a" "$a"; done
  printf '}\n}\n'
} >> bake.hcl

echo "--- arms=$ARMS, bottom Dockerfile $(wc -c < Dockerfile.bottom) bytes ---"
docker buildx bake -f bake.hcl --set '*.output=type=cacheonly' --progress quiet top 2>&1 |
  grep -E "ResourceExhausted|did not complete" | head -2
rm -rf "$DIR"

Result

Base Dockerfile is 1,983,762 bytes in every run; only the arm count changes.

arms	solve request bytes	delta
1	under cap (build ran, aborted at RUN false)	—
2	21,433,965	—
3	32,150,801	+10,716,836
4	42,867,637	+10,716,836

The increment is constant to the byte: each additional arm adds exactly one more full
copy of the base's definition. 2 × 10,716,836 = 21,433,672, i.e. the two-arm message is
two copies of the base plus ~293 bytes of arm-specific ops — nothing is shared.

Same behaviour on a standalone daemon (docker-container driver, buildkit v0.18.2):
21,433,918 / 32,150,745, identical increment. Not driver-specific.

Depth compounds it

Replace the flat fan-in with a FROM-chained tree — the base fans out into two
children, each of those into two more, and only the top merges with
COPY --from=<leaf>:

# tree.py <lines> <depth>; then: docker buildx bake -f bake.tree.hcl \
#   --set '*.output=type=cacheonly' --progress quiet top
import sys, os
LINES, DEPTH, PAD = int(sys.argv[1]), int(sys.argv[2]), 9000
with open("Dt.bottom", "w") as f:
    f.write("FROM alpine\n")
    for i in range(LINES):
        f.write(f"RUN true #{i:06d}{'x' * PAD}\n")
hcl = ['target "bottom" {\n  context = "."\n  dockerfile = "Dt.bottom"\n}\n']
frontier = ["bottom"]
for lvl in range(1, DEPTH + 1):
    nxt = []
    for parent in frontier:
        for side in ("a", "b"):
            name = f"t{lvl}{side}_{parent}"
            open(f"Dt.{name}", "w").write(f"FROM base\nRUN true # {name}\n")
            hcl.append(f'target "{name}" {{\n  context = "."\n  dockerfile = "Dt.{name}"\n'
                       f'  contexts = {{ base = "target:{parent}" }}\n}}\n')
            nxt.append(name)
    frontier = nxt
lines, ctxs = ["FROM alpine\n"], {}
for j, leaf in enumerate(frontier):
    lines.append(f"COPY --from=c{j} /etc/alpine-release /f{j}\n")
    ctxs[f"c{j}"] = leaf
open("Dt.top", "w").writelines(lines)
entries = ", ".join(f'{k} = "target:{v}"' for k, v in ctxs.items())
hcl.append(f'target "top" {{\n  context = "."\n  dockerfile = "Dt.top"\n'
           f'  contexts = {{ {entries} }}\n}}\n')
open("bake.tree.hcl", "w").write("".join(hcl))

Per-copy cost of the base scales with its line count
(48,713 bytes/line, from the table above), so the base is shrunk to keep the
prediction in range:

depth	leaves	base LLB	expected (leaves × base)	measured
2	4	5,358,430	21,433,720	21,446,301
3	8	2,679,215	21,433,720	21,479,861

Both land within 0.06% and 0.22% of leaves × base, the residual being the
intermediate nodes' own ops (6 and 14 of them). So the multiplier is exactly the
number of root-to-leaf paths: 2^depth, not the number of nodes.

What the request actually contains

Captured off the wire with a TCP proxy in front of a --driver remote buildkitd
(HTTP/2 DATA frames reassembled into gRPC messages, no HPACK needed). Depth-2 tree,
4 leaves, 20-op base — the top's request is 3,917,180 bytes:

field 3  (frontend attrs) : 13x, total   974,503   max 243,533
field 13 (frontend inputs):  4x, total 2,942,608   max 735,652
  └─ entry: key 16B + Definition 735,630B
       field 1 (def, repeated bytes) : 24x ops        184,711
       field 2 (metadata map)        : 24x            365,944
       field 3 (source)              : 1x             184,808   <- the Dockerfile text

One entry per leaf, each a complete standalone copy of the base — its ops, its
metadata, and its Dockerfile source. Nothing is shared between entries.

Also worth noting the expansion ratio: 1.98 MB of Dockerfile text becomes ~10.7 MB of
LLB (5.4×), so the 16 MB ceiling is reachable with a base that does not look large.

Why it hurts

This is the natural shape for a monorepo that emits one Dockerfile per target and
federates them with contexts = { dep = "target:dep" }. A shared toolchain or
dependency-install stage sits at the bottom of a FROM chain that every consumer
extends, so the request grows with the number of paths through the graph while the
graph itself does not grow at all — and layering (toolchain → deps → build → per-app)
multiplies rather than adds. Today the only
workarounds are to break the diamond by pushing the shared base to a registry and
referencing it by digest, or to split the bake into several invocations so each message
carries fewer arms. Both give up exactly the cross-target sharing that target:
contexts exist to provide.

Suggested direction

Let the solve request carry a shared op pool that FrontendInputs entries can
reference, so an identical subgraph is transmitted once. Plain client-side dedup of the
current message is not an option — a pb.Definition must contain its transitive
closure, so the deduped message would not be a valid one. gRPC compression is not an
escape hatch either: the limit applies to the decompressed size.

Versions

docker         29.1.3
buildx         v0.30.1-desktop.1 (792b8327a475a5d8c9d5f4ea6ce866e7da39ae8b)
buildkit       embedded (docker driver); v0.18.2 (docker-container and remote drivers)
host           macOS / Docker Desktop