Adopt Genmedia-style async polling policy in Sume skills

[chasehuh]

Summary

Sume CLI skills should adopt the Genmedia-style async orchestration pattern for paid multi-job workflows: the CLI provides stable primitives, while skills force agents to submit idempotently, poll until terminal state, and complete result/resource readback instead of stopping after an arbitrary watch timeout.

This came out of first-time dogfooding after PR #9 / SUM-619. The current skill direction is good, but the dogfood thread showed one important gap: after submitting 3 paid Avatar jobs, the agent watched for 120 seconds, saw all jobs still processing, and returned a final answer telling the user to check later. The jobs completed shortly afterward. The agent should have treated timeout as a re-evaluation point, not as the end of the workflow.

Related issue: #10 tracks a separate but adjacent command/schema drift where submit examples include --agent even though avatars create / avatar-videos create currently reject it. Fixing this issue should avoid reinforcing that drift; either coordinate with #10 or implement both together if small.

Genmedia reference check

This was verified against the official Genmedia CLI repo, not only local user-created skill files.

• Official repo cloned for inspection: https://github.com/fal-ai-community/genmedia-cli.git
• Checked commit: 63ef5e5b6a72df984c78dfa7fe8e6ed3c03647c0
• Installed local binary observed: /Users/huhchaewon/.local/bin/genmedia
• Installed version observed: 0.7.0

Genmedia CLI itself does not provide a built-in multi-job long-running watch command like Sume's sume jobs watch --ids. Its core primitives are:

genmedia run <endpoint_id> ... --async --json
genmedia status <endpoint_id> <request_id> --json
genmedia status <endpoint_id> <request_id> --result --json
genmedia status <endpoint_id> <request_id> --download ./out/ --json

Official Genmedia source confirms status is a one-shot queue status/result/download command (src/commands/status.ts), not a persistent watcher.

The continuous polling behavior is documented as an agent/workflow loop in the official README:

SUBMIT=$(genmedia run <endpoint_id> --prompt "..." --async --json)
REQ=$(echo "$SUBMIT" | jq -r '.request_id')

while true; do
  RES=$(genmedia status <endpoint_id> "$REQ" --json)
  STATUS=$(echo "$RES" | jq -r '.status')
  [ "$STATUS" = "COMPLETED" ] && break
  [ "$STATUS" = "FAILED" ] && { echo "$RES" | jq '.error'; exit 1; }
  sleep 5
done

genmedia status <endpoint_id> "$REQ" \
  --download "./out/{request_id}_{index}.{ext}" \
  --json

The local Genmedia experiment skill has the same high-concurrency policy:

• submit all independent clips asynchronously first;
• poll all submitted clips concurrently;
• treat non-terminal status as polling state, not immediate failure;
• print only coarse progress;
• after all clips finish, download/assemble/compare;
• keep a sanitized manifest.

This is the pattern Sume skills should mirror.

Sume dogfood evidence

Dogfood thread: Sume CLI first-time skills dogfood (019f35bd-1bc2-75a1-b88b-1143770af841)

User asked the agent to create 3 Korean cosmetics-ad avatars. The agent flow was mostly correct:

1. Detected missing auth and did not submit.
2. Asked for login authorization.
3. Ran browser login and confirmed auth without printing secrets.
4. Asked for explicit paid generation approval.
5. Submitted 3 paid Avatar jobs with stable handles/idempotency intent.
6. Moved into jobs watch.

Problem: the agent chose a bounded 120 second watch and then final-answered when all jobs were still processing, instead of continuing/rechecking/asking whether to keep watching.

Submitted jobs from the dogfood flow:

job_7c98ed3c-da28-45bb-b219-17b2adc9a1be / avtr_9d4f12dc-af62-4604-9fec-c219fa7aa2a6
job_75eeaa27-c6b8-4bd1-b4f9-357df2006f08 / avtr_5ca2c914-5d8f-4571-bb2e-72bc6ef1ed25
job_395c1460-5a97-4e92-87d3-658c472447c1 / avtr_dfd0609a-7e1d-437f-9160-1eedbb4a9c85

A later read-only coordinator poll showed all three were COMPLETED. No additional paid jobs were required to verify that the previous turn stopped too early.

Current Sume advantage

Sume has stronger CLI primitives than Genmedia here:

sume jobs watch --ids job_1,job_2,job_3 --agent --json
sume jobs watch --ids job_1,job_2,job_3 --timeout-seconds 0 --agent --json
sume jobs result <job_id> --agent --json
sume avatars get <avatar_id> --agent --json
sume avatar-videos get <avatar_video_id> --agent --json

So this should be mostly a skill/docs/test policy fix, not a core CLI redesign. The skills should teach agents to use the existing primitives correctly.

Desired Sume skill behavior

After paid async generation submissions, skills should force this policy:

1. Submit all independent jobs first with stable idempotency keys or a batch state file.
2. Record job ids/resource ids in local state or a sanitized manifest.
3. Poll all submitted jobs together with sume jobs watch --ids ... --agent --json, or use --timeout-seconds 0 as a one-shot aggregate inside an agent-managed loop.
4. Treat timeout as a re-evaluation point, not final completion.
5. Do not final-answer as if the workflow is done while paid jobs are still non-terminal, unless the user explicitly asks to stop.
6. On timeout, immediately run a one-shot status/readback pass, then either:
   • continue polling;
   • ask the user whether to keep watching;
   • or provide a clear “resume” command and mark the work incomplete.
7. Once all jobs are terminal, fetch result/resource readback with agent-safe redaction.
8. Report completed/failed/canceled counts, resource ids, cost state if visible, and sanitized artifact availability.
9. Do not submit replacement paid jobs unless the user explicitly approves another paid run under a cap.
10. Do not print API keys, auth codes, signed URLs, private media URLs, full result URLs, provider payloads, or private identifiers.

Suggested implementation direction

Update these skill/docs surfaces first:

• agent/sume-min-latency/SKILL.md
• agent/sume-spend-capped-dogfood/SKILL.md
• agent/sume-avatar/SKILL.md
• agent/sume-avatar-video/SKILL.md
• agent/sume/references/eval-scenarios.md
• docs/agent-workflows.md
• README.md if it describes agent submit/watch behavior
• regenerate src/lib/bundled-skills-data.ts

Recommended wording to add somewhere central:

Timeout is not terminal. After a paid submit, keep ownership of the job set until every job is completed, failed, canceled, or the user explicitly tells you to stop. If `jobs watch` returns timeout, run a one-shot status aggregate, then continue polling or ask the user whether to keep watching. Do not submit duplicate paid jobs while recovering from a timeout.

Recommended examples:

# Submit independent jobs with idempotency/state first.
# Then use one aggregate watcher:
sume jobs watch --ids "$JOB_IDS" --agent --json

# Or agent-managed loop using one-shot aggregate checks:
while true; do
  sume jobs watch --ids "$JOB_IDS" --timeout-seconds 0 --agent --json
  # Parse terminal counts. If all terminal, break and fetch results.
  # If still running, sleep and continue or ask the user if the time cap is reached.
  sleep 10
done

If #10 is fixed by adding --agent support to submit commands, examples can use --agent --json on submit. If #10 is fixed by removing --agent from submit commands, make these examples reflect the actual accepted command shape.

Acceptance criteria

• Skills explicitly state that paid async job timeout is not a final answer condition.
• Skills explain Sume's Genmedia-like model: CLI primitives plus agent-side workflow ownership.
• Multi-job Avatar and Avatar Video examples submit first, then watch/poll all job ids together, then fetch results/resources.
• sume-min-latency optimizes for low latency without losing terminal readback responsibility.
• sume-spend-capped-dogfood requires explicit cap, idempotency/state, no duplicate paid calls, and terminal readback before verdict.
• sume-avatar and sume-avatar-video route multi-candidate work through batch/state or aggregate watch patterns.
• Tests assert skill content includes timeout/re-evaluation/no-duplicate-paid-call guidance.
• Tests assert skill examples do not use unsupported CLI flags, or this PR also fixes Fix --agent drift on paid submit command guidance #10 so the flags are supported.
• src/lib/bundled-skills-data.ts is regenerated from source skill files.
• No old sume.so Brand/Ads/Face Swap/generic generation surfaces are introduced.
• No provider internals, signed URLs, full media URLs, or private identifiers are introduced in skill examples.

Validation plan

• Run focused skill registry/content tests.
• Run CLI tests that cover skills list/export and tool schema consistency.
• Run pnpm run check if practical.
• Run no-cost smoke checks:

./dist/sume --json skills list
./dist/sume --json skills export sume
./dist/sume --json skills export sume-min-latency
./dist/sume --json skills export sume-spend-capped-dogfood
./dist/sume --json tools schema jobs.watch

No paid/provider calls are required for this fix.