Optimize writable entity lookup and preserve same-row alias updates

[crdv7]

Summary

This change avoids redundant graphid lookups when SET, REMOVE, or
DELETE writes an entity already read by MATCH.

For writable statements, AGE now carries the matched tuple's ctid as a
hidden internal lookup hint. The executor validates the tuple's snapshot
visibility and graphid before using it, and falls back to the existing
graphid lookup when the hint is missing or stale. Graphid remains the logical
entity identity.

The same executor path also contained a correctness issue: when two aliases
in one result row represented the same entity, a later SET or REMOVE
could rebuild the entity from stale alias-local properties and overwrite an
earlier change. This change refreshes the current properties and keeps
same-graphid aliases and paths synchronized.

The resulting lookup order is:

validated CTID fetch
    -> usable non-partial B-tree index whose first key is entity id
    -> sequential scan when no suitable id index exists

No Cypher syntax or user-visible result columns are added.

Implementation

Hidden CTID propagation

• Add a hidden CTID target entry for named vertices and edges read by
  MATCH, but only when the statement contains SET, REMOVE, or DELETE.
• Carry the hint through simple WITH n and WITH n AS alias projections.
• Stop propagation at aggregation, DISTINCT, computed expressions, and
  parser-generated variables, where a hidden value could change row-shaping
  semantics or become associated with the wrong entity.
• Store the hidden column position in update/delete plan metadata and include
  it in ExtensibleNode copy/out/read support.

PostgreSQL's native tid is packed into a scalar agtype value for transport
through AGE target lists:

BlockNumber (32 bits) << 16 | OffsetNumber (16 bits)

Read-only MATCH/RETURN statements do not receive hidden target entries.

Shared entity lookup

find_entity_tuple() centralizes tuple lookup for the writable executors.

1. If a hidden CTID is available, fetch that row version directly.
2. Accept it only when it is visible to the current snapshot and contains
   the expected graphid.
3. Otherwise, use a valid non-partial B-tree index whose first key is the
   entity id.
4. Fall back to a sequential scan only when no suitable id index exists.

This remains correct if a user drops a default vertex id index. It also
allows a user-created edge id index to serve as the fallback. The default
edge start_id and end_id indexes cannot resolve an edge by graphid.

A statement-local cache retains the tuple fetch slot, opened id index
relation, and negative index-search result. SET and DELETE own that cache
for their custom-scan lifetime.

CTID is always a hint rather than identity. A previous writable clause may
move the tuple and leave another alias with a stale CTID; the validation and
graphid fallback prevent an update or delete from targeting the wrong row.
After an update, the new CTID is propagated for later writes in the same
result row.

Same-row alias consistency

The original apply_update_list() built replacement properties from the
agtype value stored at each update item's tuple position before locating the
current heap tuple. Different aliases were treated as independent positions
even when they contained the same graphid, and the existing synchronization
pass updated entities inside paths but not top-level aliases.

For example:

MATCH (n:T), (m:T)
WHERE id(n) = id(m)
SET n.a = 1
SET m.b = n.a + 1
RETURN n.a, m.a, m.b

The query could return 1, NULL, 2 while the stored entity contained
a = NULL, b = 2.

The updated executor locates the current heap tuple before building the first
update for an entity position, refreshes stale base properties from that
tuple, synchronizes top-level aliases and paths with the same graphid, and
reuses the tuple for the physical update. REMOVE uses the same path and
receives the same consistency fix.

DELETE does not reconstruct properties, but it uses the validated lookup
helper so a stale alias CTID after an earlier SET cannot cause the wrong
tuple to be deleted.

The correctness fix and the lookup optimization are included together
because they share the same apply_update_list() control flow: the executor
must locate the current tuple before rebuilding properties, synchronize
same-entity references, and then reuse that tuple for the physical update.
Splitting them would require implementing and immediately replacing an
intermediate lookup path. If a correctness-only backport is requested, it
can be adapted to use the existing graphid fallback first.

MERGE fallback cache

MERGE uses a separate lateral/eager custom-scan transform that does not
currently carry a reliable entity-to-CTID column position into its SET
metadata, so it does not receive a hidden CTID fast path in this change. When
a plan contains ON MATCH SET or ON CREATE SET, it creates the shared
graphid fallback cache for the statement lifetime and releases it when the
custom scan ends.

Suggested review order

1. src/backend/executor/cypher_utils.c: shared CTID/index/SeqScan lookup.
2. src/backend/parser/cypher_clause.c: hidden CTID generation and WITH
   propagation.
3. src/backend/executor/cypher_set.c: current-property refresh and
   alias/path synchronization.
4. src/backend/executor/cypher_delete.c and
   src/backend/executor/cypher_merge.c: lookup/cache integration.
5. ExtensibleNode definitions and copy/out/read serialization.
6. Regression SQL and expected output.

Performance

Environment and method

• Baseline AGE:
  801417404978823bd8732452c3f7959017584785
• Patched AGE:
  1d1e8fe4a289eae83b7873a0bc534fa2bccd2cad
• PostgreSQL 18.4, built with --disable-debug --disable-cassert and
  CFLAGS=-O2 -g
• Ubuntu 22.04 and GCC 11.4.0 on both hosts
• Intel Xeon 6982P-C, 8 cores / 16 threads, 29 GiB RAM
• AMD EPYC 9T25, 4 cores / 8 threads, 15 GiB RAM
• Local Unix-domain socket
• shared_buffers=1GB, fsync=off, synchronous_commit=off,
  full_page_writes=off, autovacuum=off, jit=off

Each host ran the same complete benchmark matrix. PostgreSQL and its child
backends were pinned to vCPU 2 to avoid migration between virtual CPUs.
Only age.so changed between variants. Data reset, VACUUM ANALYZE, server
restart, and shared-library replacement were outside the timed interval.

The read-only control executes 20 prepared queries in one backend and reports
the per-query time, avoiding client startup and cold-backend costs from
dominating this short control. Controls use 15 measured AB/BA-interleaved
runs; writable and edge workloads use five. Both variants receive an
unreported warm-up. Negative change means the patch is faster.

Workload	Rows	Intel baseline	Intel patched	Change	AMD baseline	AMD patched	Change
Read-only MATCH/count	100,000	18.516 ms	18.689 ms	+0.94%	12.484 ms	12.526 ms	+0.34%
SQL insert control	100,000	170.849 ms	170.205 ms	-0.38%	155.595 ms	155.106 ms	-0.31%
Cypher CREATE control	100,000	197.685 ms	199.836 ms	+1.09%	177.334 ms	178.330 ms	+0.56%
SET	100,000 vertices	12.910 s	12.458 s	-3.51%	9.246 s	9.335 s	+0.96%
REMOVE	100,000 vertices	12.963 s	12.368 s	-4.59%	9.194 s	8.917 s	-3.02%
DELETE	100,000 vertices	489.771 ms	409.075 ms	-16.48%	401.217 ms	335.844 ms	-16.29%
DELETE	500,000 vertices	2.804 s	2.317 s	-17.37%	2.272 s	1.862 s	-18.03%
MERGE ON MATCH SET	100,000 operations	12.159 s	12.243 s	+0.69%	8.927 s	8.885 s	-0.47%
Edge SET, no edge id index	10,000	939.152 ms	212.813 ms	-77.34%	786.110 ms	151.252 ms	-80.76%
Edge SET, no edge id index	20,000	3.593 s	594.376 ms	-83.46%	3.014 s	421.584 ms	-86.01%
Edge SET, no edge id index	100,000	92.965 s	12.668 s	-86.37%	73.210 s	9.276 s	-87.33%
DISTINCT edge SET, edge id index	20,000	644.015 ms	628.293 ms	-2.44%	463.848 ms	458.426 ms	-1.17%

The controls remain within approximately 1% on both hosts. In particular,
read-only statements do not receive hidden CTID target entries and show no
material regression; their elapsed differences are smaller than the measured
run-to-run variation. CV is at most 1.63% for the 100,000-vertex DELETE
results and 3.48% for the 100,000-edge no-index results.

DELETE, which isolates entity lookup most clearly, improves by 16–18% on
both CPUs. REMOVE improves by 3–5%. SET spends most of its time rebuilding
properties and updating PostgreSQL tuples, so its statement-level result is
small and hardware-dependent. MERGE, which uses only the fallback cache in
this change, remains neutral.

The edge-label case shows the largest and most consistent benefit. Without a
user-created id index, the baseline performs a sequential graphid lookup
for every edge. Increasing the label from 10,000 to 100,000 edges increases
baseline time by roughly two orders of magnitude, while the patch reduces
100,000-edge elapsed time by 86–87% on both CPUs.

Benchmark reproduction

A self-contained reviewer script builds the baseline and patched commits,
creates an isolated PostgreSQL 18 cluster, runs the workloads in AB/BA order,
and writes raw and summarized CSV files. It neither installs AGE into the
PostgreSQL prefix nor modifies an existing cluster.

repro_benchmark.sh

From an AGE checkout on the pull request branch:

chmod +x repro_pr_benchmark.sh
SERVER_CPU=2 READ_ONLY_BATCH_COUNT=20 \
EDGE_SIZES="10000 20000 100000" ./repro_pr_benchmark.sh \
  --age-repo "$PWD" \
  --pg-config /path/to/postgresql-18/bin/pg_config \
  --full

This command reproduces the timing matrix above, including 15 control runs,
five writable/edge runs, and the 500,000-vertex DELETE case. Because the
no-index 100,000-edge baseline is intentionally slow, reviewers can first
validate their environment with:

SERVER_CPU=2 READ_ONLY_BATCH_COUNT=20 ./repro_pr_benchmark.sh \
  --age-repo "$PWD" \
  --pg-config /path/to/postgresql-18/bin/pg_config \
  --quick

The main output is all-summary.csv; per-group raw CSV, build logs,
PostgreSQL logs, and an environment manifest are retained alongside it.
Choose a SERVER_CPU available on the test host.

Standalone edge-label benchmark

The following self-contained script creates an edge label with only the
default start_id and end_id indexes, loads the requested number of edges,
prints the actual indexes, runs EXPLAIN ANALYZE, and verifies the updated
row count:

repro_edge_set_no_id_index.sql

# Quick check: 10,000 edges
psql -X -d <database> -f repro_edge_set_no_id_index.sql

# Same scale as the largest result above
psql -X -d <database> -v edge_count=100000 \
  -f repro_edge_set_no_id_index.sql

CPU profile

The 2,000,000-vertex DELETE workload was sampled at 199 Hz with
cycles:u and DWARF call graphs:

Profile	Samples	Sample duration	Sampled cycles
Baseline	2,217	11.158 s	37.43 billion
Patched	1,822	9.144 s	29.70 billion

In the baseline, stacks below process_delete_list() containing
index_getnext_slot account for 16.56% of sampled cycles:

process_delete_list
  index_getnext_slot
    index_getnext_tid
      btgettuple
        _bt_first
          _bt_search

That stack disappears from the patched profile. Its replacement accounts for
3.09% of patched sampled cycles:

process_delete_list
  find_entity_tuple
    try_ctid_fetch_tuple
      table_tuple_fetch_row_version
        heapam_fetch_row_version
          heap_fetch

Baseline flame graph:

Patched flame graph:

Five-run perf stat medians for the 500,000-vertex DELETE workload:

Event	Baseline	Patched	Change
CPU cycles	8,784,745,592	7,027,241,137	-20.01%
Instructions	28,468,929,329	22,572,493,361	-20.71%
Branches	5,228,809,347	4,153,536,763	-20.56%
Branch misses	5,534,090	3,547,249	-35.90%
Cache references	14,515,881	13,605,987	-6.27%
Cache misses	3,389,227	2,924,470	-13.71%

The elapsed-time improvement is accompanied by comparable reductions in
instructions and branches, supporting a CPU-work reduction rather than an
I/O-only effect.

Tests

Built and tested against PostgreSQL 18.4:

• clean release build completed without warnings;
• targeted writable/VLE regression suites passed under debug/cassert:
  cypher_set, cypher_remove, cypher_delete, cypher_with,
  cypher_vle, and cypher_merge;
• targeted writable suites and the complete security/RLS suite passed with
  AddressSanitizer and MEMORY_CONTEXT_CHECKING;
• the current PR commit passed the full debug/cassert AGE regression suite:
  43/43.

New regression coverage includes:

• valid and stale CTID lookup, including fallback after an earlier SET;
• vertex and edge aliases that share a graphid across chained writes;
• dependent right-hand-side expressions and persisted-state verification;
• alias/path synchronization for SET and REMOVE;
• simple WITH references and renames;
• aggregation, DISTINCT, and computed-expression propagation boundaries;
• writable WITH * and RETURN * without leaking hidden CTID columns;
• stale CTID fallback for REMOVE, DELETE, and DETACH DELETE;
• fallback through a user-created edge id B-tree index;
• sequential-scan fallback after dropping a vertex label's id index;
• forced generic prepared-plan reuse across label-table DDL;
• direct TID-wrapper packing, including block/offset boundary values.

Related issues

Closes #2484
Closes #2485