[SPARK-50569][CONNECT]: clean up data cached by individual sessions

[psavalle]

What changes were proposed in this pull request?

This tackles part of SPARK-50569 by adding a new configuration option:

spark.connect.session.manager.cleanupCachedData.enabled

We now track for each cached plan the sessions that have cached it. This allows the following:

• When spark.connect.session.manager.cleanupCachedData.enabled is true and the last session referencing a cached plan is closed, the data is unpersisted.

• catalog.clearCache now has an argument allSessions. By default, allSessions=true and it clears dataframes persisted by all sessions. When setting allSessions=false, only the dataframes persisted by the current session are unpersisted.

Why are the changes needed?

With Spark Connect, when a session calls persist() on a dataframe, it remains cached even after the session is closed. However, it can be useful to unpersist dataframes cached by a given session when it closes. Otherwise, persisted data can accumulate.

Today, sparkSession.catalog.clearCache can also clear all dataframes that have been persisted in the Spark Connect server, so it is also not possible for a session to use it to unpersist all the dataframes it persisted, since it would also unpersist all the dataframes persisted by other sessions.

Does this PR introduce any user-facing change?

• A new configuration option. The default behavior is preserved by default.

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Generated-by: codex-cli 0.147.0