[jira] [Created] (SPARK-54025) Support recaching when a table is written via a different table implementation (V1 or V2)

["Gengliang Wang (Jira)" <ji...@apache.org>]

Gengliang Wang created SPARK-54025:
--------------------------------------

             Summary: Support recaching when a table is written via a different table implementation (V1 or V2)
                 Key: SPARK-54025
                 URL: https://issues.apache.org/jira/browse/SPARK-54025
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.1.0
            Reporter: Gengliang Wang


When a table is cached using one table implementation (e.g., V2) and written through the other (e.g., V1), Spark may not automatically trigger recaching. As a result, the cached data can become stale even though the underlying table content has changed.

 

This issue arises because the current recaching mechanism does not consistently handle cross-implementation writes. Given that the community is actively working on Data Source V2 (DSV2), many data sources are expected to have both V1 and V2 implementations for a period of time, making this issue more likely to occur in practice.

 

*Proposed Fix:*

Enhance the cache invalidation logic to detect writes that occur through a different table implementation (V1 ↔ V2) and trigger recaching accordingly.

 

{*}Expected Outcome:{*}{*}{*}
 * Cached data remains up to date when a table is written through either V1 or V2 paths.

 * Both logical-plan-based and file-path-based recaching continue to work as expected for V1&V2 connectors

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org