[jira] [Commented] (SPARK-59751) Preserve CHAR/VARCHAR scan policy across cache mutation, refresh, and rename

["Serge Rielau (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-59751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118497#comment-18118497 ] 

Serge Rielau commented on SPARK-59751:
--------------------------------------

Needed work from PR 58317 review of {{b8d27c0e}}

The latest comments are non-blocking, but they describe the actual contract this JIRA should implement. Do not patch them onto SPARK-58814.

Design (do this first)

Cache reuse, mutation identity, direct-cache origin, and restoration are still separate helpers. Make one {{CacheManager}} lifecycle API:
 # A descriptor with resolved direct-cache origin, analyzed plan, bound scan mode, and storage level.
 # Route V1, V2, and Hive mutation, refresh, and rename through that API. Ordinary cache substitution stays mode-sensitive; mutation ignores only scan mode.
 # Restore each variant in isolated session configuration that shares {{SharedState}} / {{{}CacheManager{}}}. Do not wrap restoration in caller {{{}withSQLConf{}}}.
 # Keep format/reader-mode work on SPARK-58814.

Must-fix from the latest review
 * Isolate rename restoration from caller {{{}SQLConf{}}}. V2 {{RenameTableExec}} and V1/Hive {{AlterTableRenameCommand}} both use {{{}withSQLConf{}}}; concurrent work can observe restoration-only policy.
 * Match direct V2 caches by resolved table identity, not {{CACHE TABLE}} display name. Unqualified or namespace-qualified caches can be dropped on rename.
 * Bind {{Legacy}} as an explicit policy and preserve it with PreserveNative and SparkStandard through recache, refresh, and rename.
 * Exercise actually bound CHAR/VARCHAR columns in the V1 insert and save regressions. Current fixtures are {{{}INT{}}}/{{{}STRING{}}} and {{{}LONG{}}}, so both config branches stay unbound and a regression to {{recacheByPlan}} still passes.

Nits to include while doing the redesign
 * Check the cheap direct-cache predicate before walking the plan.
 * Bind scan mode inside the existing padding traversal when padding runs; keep a standalone walk only when padding is skipped.
 * Correct {{lookupCacheDescriptorsByV2Relation}} Scaladoc: default {{directNamedCacheOnly = false}} also returns dependent plans.

> Preserve CHAR/VARCHAR scan policy across cache mutation, refresh, and rename
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-59751
>                 URL: https://issues.apache.org/jira/browse/SPARK-59751
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>    Affects Versions: 5.0.0
>            Reporter: Serge Rielau
>            Priority: Major
>
> First-class CHAR/VARCHAR under SPARK-58794 binds an analyzed scan mode ({{{}PreserveNative{}}} / {{{}SparkStandard{}}}) onto relation identity so incompatible scans cannot reuse each other's cached results. SPARK-58814 / PR 58317 covers format round-trips and ORC truncation, plus the cache-lifecycle repairs that were required to keep that identity from serving stale rows on the paths already in that PR.
> This follow-up is the remaining cache-policy contract. Today {{None}} still means "unbound / not first-class". A cache created with first-class flags off can be rebound when a later session with first-class modes recaches or renames the table. V1 and Hive rename still snapshot one cache entry, so coexisting PreserveNative and SparkStandard variants can lose a mode or storage level. Hive refresh coexistence and a simultaneous three-policy mutation matrix are also not complete.
> Scope:
>  * Represent the pre-first-class policy as an explicit bound state (or equivalent), distinct from unbound.
>  * Preserve that policy, plus PreserveNative and SparkStandard, through recache, refreshTable, and rename for V1, V2, and Hive.
>  * Keep ordinary cache substitution mode-sensitive; ignore scan mode only for mutation/rename discovery.
>  * Restore only direct table caches (including analyzer CHAR/VARCHAR padding Projects); do not promote dependent-query or time-travel caches.
>  * Reproduce with coexisting variants in one SharedState, including a mutation/rename from a session using a different policy.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org