Add incremental invalidation engine

[st0012]

Extracted from #638 (incremental invalidation). This PR contains the core invalidation engine; the incremental resolver and without_resolution flag will follow as separate PRs.

Replaces the old remove_definitions_for_document + invalidate_ancestor_chains approach with a targeted invalidation engine. When a file is updated or deleted, the engine traces through the name_dependents reverse index to invalidate only the affected declarations, names, and references — instead of requiring a full graph rebuild.

How it works

update() and delete_document() now run a three-step process:

1. invalidate — seeds a worklist from old/new definitions and references
2. remove_document_data — cleans up raw data (defs, refs, names, strings) from maps
3. extend — merges new content and accumulates pending work items

The worklist processes two kinds of items:

• Declaration — either removes the declaration (cascading to members, singleton, descendants) or clears its ancestor chain and queues descendants
• UnresolveName / UnresolveReferences — unresolves names or just their references depending on whether the structural dependency broke or only the ancestor context changed

The name_dependents reverse index (built during indexing in LocalGraph) maps each name to the definitions and references that depend on it, enabling efficient invalidation tracing without scanning the full graph.

Note: the resolver still does clear_declarations + full rebuild. Wiring it to drain pending_work incrementally is a follow-up.

[vinistock]

I think it would be nice to have some indexing tests to ensure that we get the right data structures initially.

For example:

# Bar and CONST don't depend on anything. There are no code changes that
# can modify what they mean other than changing them directly
module Bar; end
CONST = 1

# Foo has no dependencies
module Foo
  # This Bar depends on Foo
  # The Baz depends on both Bar and Foo
  class Bar::Baz
    # CONST depends on Foo and Baz
    CONST
  end
end

Note that the key is ensuring that the chain of dependents can be followed. For example, since Baz already depends on Bar, remembering that CONST depends on Baz is enough. We don't need to tie CONST to Bar. If Bar gets invalidated, it will invalidate Baz and that will in turn invalidate CONST.

[vinistock]

No need to do it in this PR, but I think this method now deserves a more descriptive name. Maybe consume_document_changes or ingest_local_graph. It does a whole lot more than just "update".

[vinistock]

It might be worth adding a fast path for the initial indexing on boot (which can trigger no invalidation and no removal of data).

Maybe a boolean flag for skipping invalidation.

[vinistock]

This is doing two data removal operations: removing a definition and clearing diagnostics. Does it make sense to move this to remove_document_data? Or will that result in duplicate work?

[st0012]

Moved

[vinistock]

Can you clarify that document change this is accounting for? I'm having a hard time understanding it.

A constant reference was changed and we're enqueuing invalidation for the reference's nesting declaration.

[st0012]

This is for

class Foo
  include Bar # constant reference, when added we invalidate Foo entirely for now
end

I've added comments for it.

[vinistock]

What does this mean?

[st0012]

Added more comments. Basically, it's triggered in

class Foo
  include Bar # this is added/removed
end

So we simply change ancestors/descendents update in this branch.

[vinistock]

This is also doing data removal. Maybe this is fine, but I'm calling it out because the method documentation mentions a separation between invalidation and removal.

[st0012]

I currently treat declaration removal as invalidation, kinda similar to unresolving a name. We remove (unresolve) the declaration here if we found it has no underlying definitions anymore.

The underlying materials (definitions, constant references, names...etc.) are only removed in remove_document_data.

[vinistock]

I could be misunderstanding and maybe this was an existing bug, but removing self is not enough. The entire ancestor chain of self must be removed from descendants. However, we should absolutely not try to perform this removal because there are module deduping rules that you cannot possibly account for with a removal.

Whenever a declaration gets invalidated, we always need to invalidate the ancestors of all descendants. In both branches of this method.

[st0012]

Both paths will update the ancestors. I've updated the document to make it more clear: invalidate_declaration either "remove/rebuild" or "update" a declaration. In both paths we update ancestors.

This also means there are optimization opportunities in both paths we can do later, which I also included in comments.