[CSV exports] Support universal CSV generation across all paths and file types

[paullizer]

Summary

Provide one universal CSV-generation capability that works across Chat Search, ordinary Chat, streaming responses, selected agents, Analyze, Compare, workflows, and source-free prompts. CSV generation must not depend on whether the input is a PDF, Office document, text, image/media-derived evidence, CSV/XLSX, mixed selected files, a single file, many files within configured limits, or no workspace files.

User Value

Users need a reliable downloadable CSV whenever they ask for structured CSV output. The current behavior varies by route and source type, so equivalent requests can return inline text, a partial relevance-bounded result, or no artifact at all.

Current Failure

1. Select multiple workspace documents, for example 16 PDFs.
2. Leave the chat document action set to Search.
3. Ask: turn these into a single csv.
4. The response can say it contains one row per retrieved document and render inline CSV, but it does not reliably cover every selected file or attach a downloadable CSV artifact.

The shared CSV-intent predicate currently accepts turn these into a csv but rejects turn these into a single csv, so the exact request above does not invoke the artifact publisher.

Proposed Behavior

• Treat CSV as an output capability shared by all response producers, not as a tabular-file-only or Analyze-only feature.
• Recognize natural CSV requests including single CSV, combined CSV, one CSV, download/export/save CSV, and equivalent phrasing.
• For explicit selected sources, preserve the path's evidence semantics while making the requested CSV coverage explicit. When the request requires all selected sources, do not silently export only relevance hits.
• When a requested row shape or schema is ambiguous, ask one durable clarification rather than guessing whether rows represent files, documents, or extracted records.
• For larger requests, use a durable background job and publish the authorized completed artifact to the originating chat. Small results may publish immediately.
• For source-free requests, convert validated model-produced structured rows into the same downloadable CSV artifact contract.
• Produce a valid downloadable CSV or an explicit failure/partial-coverage result; never fabricate unprocessed rows.

Acceptance Criteria

• A request such as turn these into a single csv creates a downloadable CSV when the response contains valid structured rows.
• CSV intent works for Chat Search, ordinary and streaming Chat, selected agents, Analyze, Compare, workflows, and source-free prompts.
• The capability works with every supported queryable input format through its native evidence adapter, including PDF, Office/text, image/media-derived evidence, CSV/XLSX, and mixed selections.
• Explicit selected-source requests disclose and enforce the requested coverage instead of silently reducing to relevance hits.
• Ambiguous row granularity/schema produces one resumable clarification; explicit row/schema instructions bypass it.
• Large work is durable, cancellable, reauthorizes source access and revision before publication, and exposes the completed download without duplicating artifacts.
• CSV output preserves current formula-injection protection, safe headers, quoted/multiline values, and authorized download controls.
• Partial, failed, unsupported, unresolved, canceled, and source-version-changed inputs are surfaced with per-source coverage and do not produce misleading complete results.
• Functional coverage validates no-source, one source, multi-source, mixed formats, all response paths, exact single CSV phrasing, clarification, background completion, cancellation, authorization loss, and downloads.

Notes

• This is a distinct output-capability issue, not a subtask of mixed-source orchestration issue [Mixed-source orchestration] Phase 6: Harden and roll out native mixed-source processing #1061.
• Existing artifacts and downloads are already generic enough to reuse; the main remediation is shared intent/finalization and source-coverage coordination.
• Focused existing CSV serializer tests passed 23/23. The current lossless-analysis artifact test needs harness maintenance because its AST loader does not inject the new cancellation helper and it has an outdated version pin.

[paullizer]

Follow-up Architecture Decision

The durable execution and artifact-publication portion of this issue is intended to establish reusable, format-neutral primitives for generated deliverables:

• persisted job lifecycle: queued, running, completed, failed, and canceled
• ownership, conversation linkage, authorization and source-revision revalidation
• leases, heartbeats, checkpoints, retry/resume, cancellation, and idempotent artifact publication
• bounded progress/status metadata and authorized artifact download/promotion

CSV remains the first format-specific capability on this issue. The implementation should not turn the existing tabular CSV/JSON executor into a generic Office renderer. Instead, it should extract or define the neutral job/artifact contract so format adapters can own their own planning and rendering.

Follow-on consumers of this foundation:

• [Generated documents] Add durable long-form Word document generation #1072: durable long-form Word document generation, with a DOCX outline/section/render adapter
• [Generated presentations] Add durable long-form PowerPoint generation #1073: durable long-form PowerPoint generation, with a deck/slide/layout/render adapter

Template population and editing remain separate concerns under #1045, #968, #969, and #999.

[paullizer]

Performance And Operability Requirements

Observed failure mode

A background CSV export can remain visibly Running at the same percentage/batch for a long period with no reliable indication whether it is actively making progress, waiting for capacity, retrying, stalled, or permanently hung. A progress card that continues to show an old ETA is not sufficient evidence of healthy execution.

Treat this as a first-class product requirement for universal generated artifacts, not only a CSV concern. The same job contract should serve the long-form DOCX and PPTX work tracked by #1072 and #1073.

Required performance model

• Define workload cohorts and versioned performance budgets for small, medium, and large generation requests. Budgets must cover queue delay, time to first durable progress checkpoint, end-to-end completion latency, and artifact publication latency; use measured p50/p95 values rather than one unqualified global timeout.
• Record wall-clock elapsed time separately from active processing time and queue/wait time.
• Record checkpoint throughput: completed units per minute, rows/batches for CSV, and the equivalent planned/rendered units for future artifact adapters.
• Record per-phase timing: authorization/manifest resolution, input staging, planning, each model/tool request, checkpoint persistence, validation, final rendering/serialization, blob upload, and artifact publication.
• Record retry count, retry reason/category, scheduler delay, lease claim/reclaim count, queue depth/concurrency pressure where available, and ETA accuracy (estimated versus actual completion).
• Emit only bounded aggregate telemetry and opaque correlation/run identifiers. Never log source contents, prompts, raw output rows, document text, filenames beyond approved aggregate labels, secrets, or user-visible data solely for performance analysis.

Forward-progress and recovery requirements

• A heartbeat alone must not keep a run healthy. Detect stalling from the last durably committed forward-progress checkpoint such as completed batch/row/unit, not only last_heartbeat_at or a Running status.
• Persist both last_heartbeat_at and last_forward_progress_at, plus the last completed unit and phase. Do not advance the displayed percentage until the corresponding checkpoint is durable.
• When no forward progress occurs within a configurable workload-aware threshold, transition the public status to stalled or an equivalent actionable state, retain the completed checkpoint, and record the diagnostic reason.
• Attempt bounded automatic recovery/reclaim when safe. After the configured retry/reclaim budget is exhausted, surface an explicit recoverable failure or blocked state rather than leaving the run indefinitely Running.
• Reauthorization, revision checks, cancellation, lease ownership, and idempotent final artifact publication must still hold through automatic recovery and manual resume.

User-visible progress requirements

• Show the current phase, completed versus total durable units, queue time, active elapsed time, last forward-progress timestamp, retry/recovery state, and ETA confidence rather than presenting an obsolete ETA as current.
• Mark stale estimates as unavailable or stale. A run that has not crossed a durable checkpoint inside its threshold must not continue to present a normal Running badge without an explanation.
• Provide accessible, actionable controls appropriate to state: Cancel while cancelable; Resume/Retry when recovery is safe; and a clear terminal error when it is not.
• Keep status polling resilient to refresh, reconnect, worker restart, and authorization loss; polling must not make a stale job appear healthy.

Validation and release gates

• Add deterministic tests for a slow-but-progressing run, a hung model/tool call with no checkpoint, stale heartbeat, stale forward progress despite a misleading worker state, lease loss, scheduler recovery, cancellation during a batch, retry/reclaim exhaustion, and idempotent final publication.
• Add a representative large-export regression scenario, including the current style of multi-batch workload, to verify that progress, ETA, recovery, and terminal status remain correct under delay and restart conditions.
• Add benchmark/soak coverage for each workload cohort. Track p50/p95 queue delay, time to first checkpoint, throughput, completion latency, retry rate, stale-run rate, recovery success rate, and artifact-publication latency as release/rollout metrics.
• Do not default-enable a new generation path until its benchmark baseline and stall/recovery behavior meet the documented performance budgets.

Architecture boundary

Extract these metrics, lifecycle, checkpoint, recovery, and publication contracts into the format-neutral generated-artifact job foundation described in #1071. CSV, DOCX (#1072), and PowerPoint (#1073) should report comparable performance metrics while retaining format-specific planning and rendering adapters.

[paullizer]

Implementation Update: Background CSV Export Throughput and Timeout Hardening

Implemented in application version 0.250.070.

What changed

• Increased default post-schema background batch concurrency from 2 to 3 while preserving the existing maximum of 5.
• Kept the first schema-discovery batch serial so later parallel batches still use one validated output schema.
• Added a configurable per-batch model-call timeout with a default of 300 seconds.
• Caps that timeout below the stale-worker threshold, so a genuinely stuck model call becomes a retryable timeout before stale-worker recovery can create competing execution.
• Existing durable checkpoints, ordered output, cancellation, authorization/revision checks, retries, and idempotent artifact publication remain in place.

Expected impact

A six-batch export now completes its post-schema model work in two three-batch windows instead of up to three two-batch windows. A batch that hangs no longer occupies the background worker indefinitely; it is retried from its last durable checkpoint.

Validation

• functional_tests/test_tabular_background_generated_exports.py: 7/7 passed.
• Added a deterministic asynchronous stalled-model-call test that verifies timeout behavior.
• py_compile passed for functions_tabular_generated_exports.py.
• Editor diagnostics and scoped whitespace checks passed.

Remaining issue scope

This is a focused throughput and bounded-wait improvement. The broader work already defined in this issue remains: durable forward-progress detection distinct from heartbeat, workload performance budgets, benchmark/soak gates, ETA confidence, queue-pressure telemetry, and extraction of a format-neutral generated-artifact job foundation for CSV, DOCX, and PPTX.

Release notes and fix documentation were updated in the same version.

[paullizer]

Related background-export resilience work is included in PR #1074: #1074

The PR increases safe post-schema batch concurrency from two to three and adds bounded model-call timeout recovery using the existing durable checkpoint path. It preserves serial schema discovery, authorization, ordered output, and idempotent artifact publication.

This is a related export reliability increment only; it does not claim to complete the broader universal CSV-generation scope in this issue.

[paullizer]

Implementation Update: Universal CSV Generation

Implemented in application version 0.250.071.

PR: #1075

What changed

• Added shared CSV intent recognition for natural requests such as turn these into a single CSV, one CSV, and combined CSV.
• Finalized valid structured rows through the same downloadable CSV artifact contract across standard and streaming Chat, Chat Search, selected agents, Analyze, Compare, workflows, and source-free prompts.
• Added one conversation-persisted row/schema clarification for ambiguous CSV requests. Explicit row or column instructions bypass it.
• Reused the existing owner-authorized immediate upload and durable background queue, including cancellation, reauthorization, source-version checks, checkpointing, and idempotent publication.
• Preserved formula-injection protection, safe headers, quoted/multiline values, and duplicate-artifact suppression.
• Rebased the work onto PR Harden mixed-source orchestration and CSV exports #1074's timeout and throughput hardening so universal CSV exports retain bounded model-call recovery.

Validation

• functional_tests/test_assistant_table_csv_artifact.py: 27/27 passed
• functional_tests/test_mixed_source_analyze_workflow.py: passed
• functional_tests/test_document_action_token_usage_aggregation.py: 5/5 passed
• functional_tests/test_document_action_conversation_scope_metadata.py: 4/4 passed
• functional_tests/test_mixed_source_conversation_continuity.py: 3/3 passed
• functional_tests/test_tabular_row_orchestration_scale.py: passed
• functional_tests/test_tabular_background_generated_exports.py: 7/7 passed
• Analyze/Compare artifact plumbing tests: 2/2 each passed
• Python compilation, editor diagnostics, and whitespace checks passed.

Remaining Scope

This completes the shared CSV intent/finalization and current durable-export integration increment. The broader format-neutral generated-artifact job foundation, forward-progress stall detection, workload performance budgets, and benchmark/soak gates described in this issue remain follow-up work, so the issue remains open.