[jira] [Updated] (SPARK-58330) Prevent silent dropping of dynamic options on same table references in DML

["Anurag Mantripragada (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Anurag Mantripragada updated SPARK-58330:
-----------------------------------------
    Summary: Prevent silent dropping of dynamic options on same table references in DML  (was: Dynamic table options are dropped when the same table is referenced twice in a DML statement)

> Prevent silent dropping of dynamic options on same table references in DML
> --------------------------------------------------------------------------
>
>                 Key: SPARK-58330
>                 URL: https://issues.apache.org/jira/browse/SPARK-58330
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Anurag Mantripragada
>            Priority: Major
>
> Spark supports a per-statement `WITH (key = value)` dynamic table options clause on `SELECT` (SPARK-36680), `INSERT` (SPARK-49098), `UPDATE` (SPARK-57681), and `MERGE` (SPARK-58007). When a single statement references the same table more than once with different options, only the first-resolved reference's options are honored; the other reference's options are silently dropped.
> Examples (each drops one side's options):
> ```sql
> -- INSERT self-select: the target's write options are dropped
> INSERT INTO t WITH (`write.split-size` = 10) SELECT * FROM t
> -- MERGE self-merge: the source's read options are dropped
> MERGE INTO t WITH (`write.split-size` = 10)
> USING t WITH (`split-size` = 5) s
> ON t.id = s.id
> WHEN MATCHED THEN UPDATE SET ...
> -- UPDATE self-reference in a subquery/CTE: the subquery's options are dropped
> UPDATE t WITH (`write.split-size` = 10) SET ...
> WHERE id IN (SELECT id FROM t WITH (`split-size` = 5))
> ```
> No error is raised; an option the user explicitly wrote is silently ignored.
> Root cause: the per-query analyzer relation cache (`RelationResolution`) is keyed by catalog + namespace + name + time-travel spec only -- options are not part of the key. On a cache hit the cached relation is returned as-is, without applying the current reference's options, so the second reference to the same table inherits the first reference's options. Which reference "wins" is a traversal-order accident. The shared (cross-query) relation cache already re-applies the caller's options on a hit, so this is an inconsistency between the two cache paths.
> This affects INSERT, UPDATE, and MERGE and should be fixed once in the shared resolution path so each reference honors exactly its own options. This subsumes the separately-noted INSERT self-select option-drop.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org