[Mixed-source orchestration] Phase 3: Make Analyze exhaustive and compositional for mixed sources

[paullizer]

Parent: #1055

Phase 3 - Mixed-Source Analyze

Planning baseline version: 0.250.061

Status: Proposed

Depends on: Phase 1

Objective

Make combined Analyze exhaustive and compositional: narrative documents use document-window analysis, tabular documents use tabular tools, and one final reduction answers the request across all selected sources.

Current Defects Addressed

• _resolve_tabular_document_action_documents(...) returns [] when any selected source is non-tabular.
• _execute_document_analysis_workflow(...) uses an exclusive branch: tabular payload or run_document_analysis(...).
• Mixed combined Analyze therefore treats tables as ordinary chunks and never invokes tabular analysis.
• Per-document mode processes mixed types but only concatenates isolated replies.

Product Semantics

Combined Analyze

• Process every target with its native engine.
• Produce one per-file summary/evidence envelope.
• Run one collective reduction over all completed envelopes.
• Report failures and partial coverage explicitly.

Per-document Analyze

• Preserve separate per-file execution and outputs.
• Continue using the native engine per file.
• Do not imply collective conclusions unless an optional final synthesis is explicitly requested in a future enhancement.

Analyze All Documents

selection_mode: all means exhaustive authorized catalog enumeration, subject to configured document limits and a clear preflight count. It must not be inferred from top-N search results.

Scope

1. Partition the manifest

Replace all-or-nothing tabular admission with:

• narrative_manifest
• tabular_manifest
• terminal unresolved/unsupported entries

2. Native engine execution

Narrative cohort:

• call run_document_analysis(...) only with narrative IDs
• preserve existing ordered windows, retries, progress, and citations

Tabular cohort:

• execute each table through shared tabular analysis
• require at least one successful analytical result per requested table
• support multi-workbook operations when the question spans tables
• keep exhaustive rows in artifacts/checkpoints, not reduction prompts

Branches may run concurrently only with bounded concurrency and independent cancellation checks.

3. Evidence normalization

Convert both branch results into Phase 1 envelopes. Include:

• document identity and source type
• concise analysis summary
• coverage counts
• citations/tool calls
• generated artifacts
• partial failures

4. Collective reduction

Add a mixed Analyze reduction prompt that receives bounded summaries from both cohorts and instructs the model to:

• answer the user's exact request
• identify cross-source relationships
• separate facts by source where needed
• state missing evidence
• avoid treating computed tabular facts as narrative excerpts
• avoid claiming coverage for failed/unprocessed files

5. Coverage merge

Produce one coverage object with every requested manifest entry and totals by engine/status.

Progress should expose:

1. Resolving sources
2. Analyzing narrative documents
3. Analyzing tabular documents
4. Combining findings
5. Complete/partial/failed

6. Artifact behavior

Preserve:

• narrative analysis artifacts
• generated tabular CSV/JSON artifacts
• summary-only handoff for background exports
• assistant-table fallback suppression

Acceptance Criteria

• Combined Analyze of PDF + XLSX invokes both engines.
• Reversing source order does not change engine routing.
• Each selected table receives a successful analytical call or an explicit failed coverage entry.
• Each selected narrative file receives complete window coverage or an explicit partial/failed entry.
• One final response answers collective questions across both cohorts.
• A simple summarize request includes a per-document summary for every source.
• Analyze All Documents enumerates every authorized target within configured limits.
• Generated tabular exports remain attached and do not flood model context.
• One branch failing does not erase successful evidence from the other branch.

Tests

• PDF + XLSX combined Analyze
• DOCX + CSV combined Analyze in both orders
• multiple narrative + multiple tabular sources
• all-tabular and all-narrative backward compatibility
• mixed collective question requiring both evidence types
• per-document mixed mode remains separate
• one tabular failure plus successful PDF analysis
• one narrative failure plus successful table analysis
• Analyze All Documents catalog larger than search top-N
• source limits and preflight rejection
• cancellation during each branch
• model and agent runners
• streaming and non-streaming response parity
• merged citations, artifacts, coverage, and token usage

Rollout

Use enable_mixed_source_analyze. Enable for explicit selections first, then enable Analyze All Documents after catalog enumeration and performance tests pass.

Rollback

Disable mixed execution. Do not restore silent table-as-text fallback; if the old path cannot safely process a mixed selection, return a clear temporary limitation.

Exit Gate

Proceed when combined Analyze demonstrates complete native-engine coverage for mixed selections and one bounded final synthesis, including partial-failure behavior.

Not in This Phase

• Cross-format comparison
• Chat/Search history semantics beyond Phase 2
• Many-to-many comparison
• Broad module extraction unrelated to Analyze

[paullizer]

Phase 3 implementation is complete on branch feature/1059-mixed-source-analyze.\n\nImplemented:\n- Default-off selected combined Analyze rollout via enable_mixed_source_analyze.\n- Fresh Phase 1 authorized manifest resolution, native narrative/tabular partitioning, bounded evidence envelopes, and one collective reduction.\n- Per-document Analyze remains separate; flag-off mixed selections fail clearly instead of silently treating tables as narrative evidence.\n- Preserved existing tabular generated-output/citation handoff and document-window analysis behavior.\n- Version bumped to v0.250.066 with feature documentation and release notes.\n\nValidation:\n- 22 focused functional tests passed.\n- Python compilation, editor diagnostics, BAC scan, XSS scan, and CRLF-aware whitespace validation passed.\n\nFollow-ups / residual scope:\n- enable_mixed_source_analyze_all remains default-off and rejects all-mode until a dedicated exhaustive authorized catalog enumerator, preflight count, limits, and performance validation are added.\n- Cross-table/multi-workbook native calculations remain a follow-up; current tabular execution is per table before collective evidence reduction.\n\nRefs #1055 #1056 #1057

[paullizer]

Status update: automatic mixed-source Analyze (v0.250.069)

Removed the default-off enable_mixed_source_analyze gate for combined Analyze.

• Both agent and direct-model workflow runners now always route explicitly selected narrative and tabular documents through the native authorization-safe mixed-source workflow.
• Removed the obsolete settings defaults/helpers and the legacy "mixed-source Analyze is disabled" rejection path.
• Kept per-document Analyze behavior unchanged.
• Added a regression that rejects reintroduction of the gate.

Validation:

• python -m pytest functional_tests/test_mixed_source_analyze_workflow.py functional_tests/test_tabular_document_actions_workflow.py -q: 9 passed.
• python -m pytest functional_tests/test_document_action_token_usage_aggregation.py -q: 5 passed.
• Python compilation, editor diagnostics, and whitespace checks passed.

Release notes and fix documentation were added for v0.250.069.

Remaining scope: Analyze All Documents remains independently unavailable pending exhaustive authorized catalog enumeration and its performance validation, so this issue remains open.

[paullizer]

Status update (v0.250.069): Removed the default-off enable_mixed_source_analyze gate for combined Analyze. Explicitly selected narrative and tabular sources now always use the native authorization-safe workflow for both agent and direct-model runners; per-document behavior is unchanged. Added regression coverage. Validation: 9 focused mixed-source/tabular tests passed; 5 document-action token-aggregation tests passed; Python compilation, editor diagnostics, and whitespace checks passed. Release notes and fix documentation added. Remaining scope: Analyze All Documents remains independently unavailable pending exhaustive authorized catalog enumeration and performance validation, so this issue remains open.