Skip to content

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525

Open
indietyp wants to merge 4 commits intobm/be-456-hashql-entity-type-model-enrichment-and-graph-store-queryfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres
Open

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525
indietyp wants to merge 4 commits intobm/be-456-hashql-entity-type-model-enrichment-and-graph-store-queryfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres

Conversation

@indietyp
Copy link
Copy Markdown
Member

@indietyp indietyp commented Mar 8, 2026

🌟 What is the purpose of this PR?

Prepares the MIR execution pipeline for consumption by the postgres compiler. The execution analysis now produces a complete IslandGraph (not just a flat island list), the placement solver and island placement accept external allocators, and the traversal system gains the TraversalPathBitMap and as_symbol() APIs that the SQL generator needs. Also adds a backend switch cost to the terminator placement so cross-backend transitions are no longer free.

🔍 What does this change?

Execution analysis (pass/execution/mod.rs):

  • run() becomes run_in() with an explicit allocator parameter, returning ExecutionAnalysisResidual (assignment + island graph). The island graph is now constructed as part of the analysis rather than left to the caller.
  • Adds run_all_in() which runs the execution analysis over all graph-read bodies in a DefIdSlice.

Traversal system (traversal/mod.rs, traversal/entity.rs):

  • TraversalPathBitMap: per-vertex-type collection of TraversalPathBitSets with pointwise lattice operations. The postgres compiler uses this to track which paths each island accesses across all vertex types.
  • TraversalPath::as_symbol(): returns a static symbol for each path variant, used as SQL column aliases so the interpreter can locate result columns by name.
  • TraversalPathBitSet::vertex(): returns the vertex type for a bitset.
  • EntityPath::as_symbol() and EntityPath::column_name(): per-path SQL identifiers.

Terminator placement (terminator_placement/mod.rs):

  • TransMatrix gains AddAssign for element-wise saturating addition.
  • Adds backend_switch_cost() which encodes a fixed overhead for cross-backend transitions (Postgres to Interpreter: 8, Interpreter to Embedding: 4, etc.). Previously cross-backend transitions had zero inherent cost, so empty blocks were arbitrarily assigned to the interpreter even when staying on postgres was free.

Placement solver (placement/solve/):

  • PlacementSolver::run() becomes run_in() with allocator parameter.
  • CSP solver internal allocations use the provided allocator.

Island graph (island/graph/mod.rs):

  • IslandGraph::new_in() now takes an allocator for its output storage.

Pretty printer (pretty/text.rs):

  • TextFormatAnnotations gains annotate_basic_block() and BasicBlockAnnotation associated type, plus a blanket impl for &mut T.

Builder (builder/rvalue.rs):

  • Adds RValueBuilder::opaque_entity_uuid() convenience constructor for the common pattern of extracting an entity's UUID.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

The backend_switch_cost() values (8, 4, etc.) are hand-tuned heuristics. They correctly prevent the solver from arbitrarily switching backends for empty blocks, but a proper cost model would derive these from measured overhead.

🛡 What tests cover this?

  • Updated execution pass end-to-end tests (execution/tests.rs) including entity_uuid_equality, mixed_postgres_embedding_interpreter, projection_and_apply_splits
  • Updated statement placement tests for interpret and postgres backends (eq_opaque_entity_uuid)
  • Updated terminator placement and island tests
  • Placement solver tests updated for allocator API

❓ How to test this?

cargo nextest run -p hashql-mir

feat: checkpoint (II)

feat: checkpoint (III)

feat: snapshot vec

feat: add dedicated filter

feat: checkpoint

feat: filter implementation

feat: filter implementation (mostly) done

chore: environment capture note

chore: always postgres bigint

feat: target clone

feat: simplify lookup

feat: move storage up

feat: eval entity path

chore: checkpoint

chore: checkpoint

chore: find entrypoint

feat: eval context

feat: eval cleanup

chore: cleanup

feat: track index

feat: wire up filter

feat: add error reporting

chore: checkpoint

feat: add traverse, and first postgres compiler outline

feat: traverse bitmap

feat: move traversal out

feat: projections

feat: projections

fix: clippy

feat: subquery projection for lateral

feat: checkpoint

feat: test plan

feat: checkpoint

feat: checkpoint – failing tests ;-;

feat: checkpoint – failing tests ;-;

feat: checkpoint — passing tests

fix: import

fix: entity type

feat: checkpoint

feat: attribute a cost to terminator placement switches

fix: import

feat: checkpoint

feat: checkpoint

chore: lint
@cursor
Copy link
Copy Markdown

cursor Bot commented Mar 8, 2026

PR Summary

Medium Risk
Touches core MIR execution/placement logic by changing allocator threading, island graph construction, and transition cost modeling; incorrect costs or graph wiring could change backend assignment/scheduling outcomes.

Overview
Execution analysis is refactored to return an ExecutionAnalysisResidual containing the final per-block target assignment and a fully-built IslandGraph, with run() replaced by allocator-aware run_in() plus a new run_all_in() for analyzing all GraphReadFilter bodies.

Placement/island infrastructure is updated to accept external allocators (PlacementSolver::run_in, IslandPlacement::run_in, IslandGraph::new_in) and the island graph API is expanded (members, contains, find). Terminator placement now adds a fixed per-backend switch overhead (via backend_switch_cost() and TransMatrix +=) so cross-backend edges are no longer “free”, shifting solver preferences.

Traversal utilities gain SQL-friendly identifiers (TraversalPath::as_symbol, EntityPath::as_symbol) and a new TraversalPathBitMap lattice type for tracking paths across vertex types. MIR building/pretty-printing and tests/benchmarks are updated accordingly (including new opaque aggregate support and new/updated snapshots for entity-uuid equality).

Reviewed by Cursor Bugbot for commit 0bd3b25. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel
Copy link
Copy Markdown

vercel Bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment Apr 21, 2026 5:15pm
petrinaut Ready Ready Preview Apr 21, 2026 5:15pm
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign Ignored Ignored Preview Apr 21, 2026 5:15pm
hashdotdesign-tokens Ignored Ignored Preview Apr 21, 2026 5:15pm

Copy link
Copy Markdown
Member Author

indietyp commented Mar 8, 2026

@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Mar 8, 2026

🤖 Augment PR Summary

Summary: Prepares the HashQL MIR execution pipeline for Postgres compilation by returning a full island dependency graph and exposing traversal/placement APIs needed by the SQL generator.
Key Changes:

  • Execution analysis now takes an explicit allocator (run_in) and returns ExecutionAnalysisResidual (assignment + IslandGraph); adds run_all_in for batch analysis.
  • Traversal adds TraversalPathBitMap, TraversalPath::as_symbol(), and EntityPath::as_symbol()/column_name() for stable SQL column aliases.
  • Placement solver and island placement APIs accept external allocators and route CSP internal allocations through the provided bump allocator.
  • Terminator placement introduces a fixed backend_switch_cost() so cross-backend transitions have non-zero overhead (avoids “free” backend flips).
  • Island graph gains helper queries (members, contains, find) and is now built/resolved within execution analysis.
  • Pretty printer supports per-basic-block annotations; MIR builder gains an opaque aggregate constructor and new tests cover opaque UUID equality.
Tests: Updates/extends execution, statement placement, terminator placement, and allocator-related solver snapshots (incl. entity_uuid_equality). Notes: Backend switch costs are heuristics intended to be refined by a measured cost model.

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. No suggestions at this time.

Comment augment review to trigger a new review at any time.

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 8, 2026

Codecov Report

❌ Patch coverage is 64.24581% with 128 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.74%. Comparing base (bc50eb0) to head (0bd3b25).

Files with missing lines Patch % Lines
...cal/hashql/mir/src/pass/execution/traversal/mod.rs 0.00% 44 Missing ⚠️
libs/@local/hashql/mir/src/pretty/text.rs 21.62% 28 Missing and 1 partial ⚠️
.../hashql/mir/src/pass/execution/traversal/entity.rs 0.00% 28 Missing ⚠️
libs/@local/hashql/mir/src/pass/execution/mod.rs 44.00% 14 Missing ⚠️
.../hashql/mir/src/pass/execution/island/graph/mod.rs 35.29% 11 Missing ⚠️
.../mir/src/pass/execution/placement/solve/csp/mod.rs 90.00% 0 Missing and 1 partial ⚠️
libs/@local/hashql/mir/src/pass/execution/tests.rs 99.07% 1 Missing ⚠️
Additional details and impacted files
@@                                           Coverage Diff                                           @@
##           bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query    #8525      +/-   ##
=======================================================================================================
- Coverage                                                                62.77%   62.74%   -0.03%     
=======================================================================================================
  Files                                                                     1327     1326       -1     
  Lines                                                                   135968   135760     -208     
  Branches                                                                  5530     5526       -4     
=======================================================================================================
- Hits                                                                     85350    85181     -169     
+ Misses                                                                   49706    49664      -42     
- Partials                                                                   912      915       +3     
Flag Coverage Δ
rust.hashql-compiletest 29.69% <ø> (ø)
rust.hashql-mir 92.08% <64.24%> (-0.37%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Mar 8, 2026

Merging this PR will not alter performance

✅ 24 untouched benchmarks
⏩ 56 skipped benchmarks1


Comparing bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres (0bd3b25) with bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query (bc50eb0)

Open in CodSpeed

Footnotes

  1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 6ef6d13 to ef9b858 Compare March 8, 2026 16:15
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 5d95ba7 to 1aa0f1c Compare March 8, 2026 16:15
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from ef9b858 to a91293c Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 1aa0f1c to 87024df Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from a91293c to 43775ba Compare March 8, 2026 17:03
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 87024df to 269a31f Compare March 8, 2026 17:03
@vercel vercel Bot temporarily deployed to Preview – petrinaut March 8, 2026 17:24 Inactive
@graphite-app graphite-app Bot requested review from a team March 8, 2026 17:25
TimDiekmann
TimDiekmann previously approved these changes Mar 12, 2026
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 8d113b2 to c67124e Compare March 31, 2026 20:56
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 3ba579f to 61d002f Compare March 31, 2026 20:56
@vercel
Copy link
Copy Markdown

vercel Bot commented Mar 31, 2026

Deployment failed with the following error:

Invalid request: `attribution.gitUser` should NOT have additional property `isBot`.

@github-actions github-actions Bot dismissed TimDiekmann’s stale review March 31, 2026 20:57

Latest approval commit db87959 is not an ancestor of c67124e, indicating rewritten history after approval

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 61d002f to bc50eb0 Compare April 21, 2026 17:06
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from c67124e to 0bd3b25 Compare April 21, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

2 participants