`jf docker push` performance regression in v2.98.0+

[g3n35i5]

Describe the bug

jf docker push has a significant performance regression starting in version 2.98.0. Pushing a multi-layer Docker image (~200 MB total, 22 layers × 10 MB) takes over 5× longer compared to all previous versions tested (2.85.0–2.97.0), and over 5× longer than a plain docker push to the same Artifactory registry.

Additional observations

• I cannot reproduce this behavior on my local PC. The slowdown is reproducible on the CI/agent environments described below.
• I attached a btop network usage screenshot from the affected environment. The initial spikes are periodic and roughly equal across runs, but versions 2.98.0 and 2.99.0 continue significantly longer than previous versions before finishing. Same goes for the CPU load.

Current behavior

End-to-end time for jf docker build + jf docker push of a 20-layer image (~200 MB) to a self-hosted Artifactory instance, measured across versions with the attached reproducibility script. The script benchmarks all versions in one run, suppresses build/push command output, prints the exact JFrog CLI version for each step, and prints a final summary table.

Version	Time
2.85.0	15s
2.86.0	15s
2.87.0	15s
2.88.0	17s
2.89.0	16s
2.90.0	16s
2.91.0	16s
2.92.0	17s
2.93.0	15s
2.94.0	21s
2.95.0	21s
2.96.0	21s
2.97.0	22s
2.98.0	1m 23s ⚠️
2.99.0	1m 21s ⚠️
docker push (no JFrog CLI)	15s

The regression is consistent and reproducible across multiple runs. Plain docker push to the same registry is unaffected.

Reproduction steps

1. Build an image with many layers (e.g. 22 × 10 MB layers):

   FROM debian:latest
   
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M
   RUN dd if=/dev/random of=Microhost bs=4k iflag=fullblock,count_bytes count=10M

2. Run the benchmark script (it downloads each CLI version automatically and then runs all measurements):

   ./debug.sh --registry your.artifactory.com --repository your-docker-repository --image image-name

   #!/usr/bin/env bash
   set -euo pipefail
   
   REGISTRY=""
   REPOSITORY=""
   IMAGE_NAME=""
   
   while [[ $# -gt 0 ]]; do
       case "$1" in
           --registry)
               REGISTRY="$2"
               shift 2
               ;;
           --repository)
               REPOSITORY="$2"
               shift 2
               ;;
           --image)
               IMAGE_NAME="$2"
               shift 2
               ;;
           *)
               echo "Unknown option: $1" >&2
               echo "Usage: $0 [--registry <host>] [--repository <repo>] [--image <name>]" >&2
               exit 1
               ;;
       esac
   done
   
   if [[ -z "$REGISTRY" || -z "$REPOSITORY" || -z "$IMAGE_NAME" ]]; then
       echo "Missing required options." >&2
       echo "Usage: $0 --registry <host> --repository <repo> --image <name>" >&2
       exit 1
   fi
   
   VERSIONS=(
       '2.90.0' '2.91.0' '2.92.0' '2.93.0' '2.94.0'
       '2.95.0' '2.96.0' '2.97.0' '2.98.0' '2.99.0'
   )
   
   declare -A TIMINGS
   
   run_benchmark() {
       local label="$1"
       local build_cmd="$2"
       local push_cmd="$3"
       local version_line="${4:-}"
       local tag="${REGISTRY}/${REPOSITORY}/${IMAGE_NAME}:$(uuidgen)"
   
       echo "--- $label ---"
       if [[ -n "$version_line" ]]; then
           echo "$version_line"
       fi
       local start end
       start=$(date +%s%N)
       bash -c "$build_cmd -q --no-cache -t $tag . && $push_cmd -q $tag" > /dev/null 2>&1
       end=$(date +%s%N)
   
       TIMINGS["$label"]=$(( (end - start) / 1000000 ))
   }
   
   for version in "${VERSIONS[@]}"; do
       jf_bin="./jf-${version}"
       if [[ ! -x "$jf_bin" ]]; then
           curl -fsSLo "$jf_bin" "https://releases.jfrog.io/artifactory/jfrog-cli/v2-jf/${version}/jfrog-cli-linux-amd64/jf"
           chmod +x "$jf_bin"
       fi
       run_benchmark "jf ${version}" "$jf_bin docker build" "$jf_bin docker push" "JFrog CLI version: $($jf_bin --version)"
   done
   
   run_benchmark "docker (no jf)" "docker build" "docker push"
   
   echo ""
   echo "=== Summary ==="
   printf "%-20s %s\n" "Version" "Time"
   printf "%-20s %s\n" "-------" "----"
   for version in "${VERSIONS[@]}"; do
       label="jf ${version}"
       ms=${TIMINGS["$label"]}
       printf "%-20s %dm %ds\n" "$label" $(( ms / 60000 )) $(( (ms % 60000) / 1000 ))
   done
   label="docker (no jf)"
   ms=${TIMINGS["$label"]}
   printf "%-20s %dm %ds\n" "$label" $(( ms / 60000 )) $(( (ms % 60000) / 1000 ))
   

3. Observe per-step output and final summary printed by the script, including all JFrog CLI versions and plain Docker baseline. Example format:

   --- jf 2.98.0 ---
   JFrog CLI version: jf version 2.98.0
   
   === Summary ===
   Version              Time
   -------              ----
   jf 2.85.0            0m 15s
   ...
   jf 2.98.0            1m 23s
   jf 2.99.0            1m 21s
   docker (no jf)       0m 15s

Expected behavior

jf docker push performance should be comparable to previous versions (2.85.0–2.97.0), which consistently pushed the same image in 20–30 seconds.

JFrog CLI version

2.98.0 (first affected), 2.99.0 (latest, also affected). Last known good: 2.97.0.

Operating system type and version

Linux (x86_64) — Debian-based CI agents (I tested multiple, phsically different machines. All with the same behavior)

JFrog Artifactory version

7.125.10 (Self-hosted, on-prem)

JFrog Xray version

3.131.31 (Self-hosted, on-prem)

[g3n35i5]

Important

This investigation was conducted with AI assistance.

jf docker push ~5x regression starting v2.98.0 — root cause identified

Environment

• CI: Jenkins on Debian 12, 62 CPUs, 76 GB RAM, Docker 29.4.0
• Registry: Self-hosted Artifactory (on-prem)
• Test image: 23 layers (debian base + 22 × 10 MB dd layers)

Findings

Version	Push time (23 layers)	Push time (2 layers)
jf 2.97.0	17s	—
jf 2.98.0	98s	9s
native docker push	4s	—

The regression scales linearly with layer count at ~4–5s per layer.

Root cause

strace shows jf 2.98.0 opens 25 sequential connections to docker.sock during push (one per layer + config + manifest, each taking ~5s), compared to 4 connections in <1s for jf 2.97.0. PID analysis confirms these come from jf's own Go goroutines, not the docker push subprocess.

The trigger is jfrog/jfrog-cli-artifactory#410 ("RTECO-976 - fixed docker oom issue"), which changed the Id() function in containermanager.go from daemon.Image(ref) to daemon.Image(ref, daemon.WithUnbufferedOpener()). This was intended to fix OOM for large images by streaming instead of buffering into memory.

However, WithUnbufferedOpener() means each access triggers a new docker save (see go-containerregistry daemon/options.go and daemon/image.go). Since Id() is called on every push and triggers a full image export from the Docker daemon for each layer access, this adds ~4–5s per layer on CI machines where Docker daemon I/O is slower.

Note: go-containerregistry v0.21.3 added a WithFileBufferedOpener() option that buffers to a temp file on disk instead of memory — this would fix both the OOM issue and the performance regression.

Relevant changes

Change	Link
OOM fix PR (introduced regression)	jfrog/jfrog-cli-artifactory#410
The one-line change in Id()	containermanager.go L75
go-containerregistry file-backed buffering	google/go-containerregistry#2214
v2.97.0 → v2.98.0 full diff	jfrog/jfrog-cli v2.97.0...v2.98.0
go-containerregistry v0.20.7 → v0.21.3 diff	google/go-containerregistry v0.20.7...v0.21.3

Ruled out

Usage reporting, docker login, Docker version (28 and 29), Docker config format differences, new outbound endpoints, Artifactory-side behavior.

Suggested fix

Replace daemon.WithUnbufferedOpener() with daemon.WithFileBufferedOpener() in Id(). This preserves the OOM fix (no in-memory buffering) while only performing a single docker save per image (buffered to a temp file on disk).

Reproduction

Push any multi-layer image with jf docker push on a CI machine (where Docker daemon I/O latency is higher). The overhead is masked on local machines with fast NVMe storage.

[JtMotoX]

This is causing our pipelines to fail and preventing developers from building and deploying their code changes.

We're seeing the same regression in our Azure DevOps pipelines after the JFrog ADO extension auto-updated to 2.12.8 (bundling CLI 2.99.0) on April 13 (yesterday).

Our affected image is a Flask application at ~1.5 GB with 23 layers. The jf docker push with --build-name/--build-number completes the actual layer push in about 4 minutes, then hangs for 33+ minutes during build-info collection until the pipeline job timeout kills it. The build-info is never published and the entire pipeline fails.

For comparison, a smaller image (~44 MB, 13 layers) in a different pipeline using the same CLI version and extension completes the build-info collection phase in ~30 seconds — it still fails to collect build-info but emits the Failed to collect build-info. No layer(s) was found for image warning and moves on. The large image never reaches that point.

The last successful build of this pipeline was on March 30 using CLI 2.89.0 / extension 2.12.7. That build also got the same "Failed to collect build-info" warning, but it appeared instantly after the push completed (sub-second) instead of hanging.

Timeline from our build logs:

Build	CLI	Image size	Push time	Build-info collection	Result
1229717 (Mar 30)	2.89.0	1.5 GB / 23 layers	~5 min	<1s (warning, continued)	Passed
1240482 (Apr 14)	2.99.0	1.5 GB / 23 layers	~4 min	33 min (hung until timeout)	Failed
1240877 (Apr 15, different pipeline)	2.99.0	44 MB / 13 layers	~10s	~30s (warning, continued)	Passed

[itsmeleela]

Thank you for the detailed analysis and bring this to notice.
We are looking into this on priority!

[fluxxBot]

Hi — update on the slow jf docker push you reported. This issue is fixed and released in version 2.102.0.

1. OOM gone for good — no config needed.
2. Opt-in env var to skip the verification step on push and cut the latency: JFROG_CLI_SKIP_DOCKER_IMAGE_ID_VERIFICATION=true

Please upgrade and confirm on your side. Closing once you verify — feel free to reopen if you still see the issue.
Thanks for the detailed report!

[g3n35i5]

Hi — update on the slow jf docker push you reported. This issue is fixed and released in version 2.102.0.

1. OOM gone for good — no config needed.

2. Opt-in env var to skip the verification step on push and cut the latency: `JFROG_CLI_SKIP_DOCKER_IMAGE_ID_VERIFICATION=true`

Please upgrade and confirm on your side. Closing once you verify — feel free to reopen if you still see the issue. Thanks for the detailed report!

Thanks for the hotfix! Personally, I'd prefer to have the variable being opt-out. I upgraded to 2.102.0 and set the environment variable, I'll give you feedback from our build infra soon.