[jira] [Commented] (SPARK-58815) Make attribute binding collision-safe in DSv2 row-level operations

["Hudson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-58815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119427#comment-18119427 ] 

Hudson commented on SPARK-58815:
--------------------------------

User 'EamesTrinh' has created a pull request for this issue:
https://github.com/apache/spark/pull/57500

> Make attribute binding collision-safe in DSv2 row-level operations
> ------------------------------------------------------------------
>
>                 Key: SPARK-58815
>                 URL: https://issues.apache.org/jira/browse/SPARK-58815
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Eames Trinh
>            Priority: Major
>              Labels: pull-request-available
>
> DSv2 row-level operation rewrites sometimes resolve attributes by name after connector references and Spark-generated attributes have already been added to the plan. Attribute names are not unique, so a user column can shadow a metadata column or an internal attribute.
> This problem also occurs with ordinary top-level references; it is not limited to nested row IDs.
> For example, a scan may contain:
> {code:java}
> index#1               user data column
> __metadata_index#2    metadata column with logical name "index"{code}
>  
> If {{SupportsDelta.rowId()}} returns {{{}FieldReference("index"){}}}, the rewrite may select the nullable user column instead of the non-null metadata column. This can produce {{NULLABLE_ROW_ID_ATTRIBUTES}} or pass the wrong value to the writer.
> MERGE has a similar problem. Spark generates an attribute named {{{_}__{_}row_id}}  __  for cardinality validation and later locates it by name. If the target table already contains a user column named __ {{{}__row_id{}}}, {{MergeRows}} can bind the user column and incorrectly report {{{}MERGE_CARDINALITY_VIOLATION{}}}.
> The same issue can affect required metadata attributes, saved row-ID values, generated MERGE presence markers, write distribution and ordering, and runtime group filtering.
> Once a connector or system reference has been resolved, row-level rewrites should carry the resolved {{Attribute}} identity through generated plans and bind later projections using {{{}exprId{}}}. They should not recover the attribute by name.
> Expected behavior:
>  * User columns cannot shadow connector metadata or Spark-generated attributes.
>  * Row IDs and required metadata retain their identity through {{{}Project{}}}, {{{}Expand{}}}, {{{}MergeRows{}}}, write preparation, and runtime filtering.
>  * Existing behavior remains unchanged when names are unique.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org