[jira] [Created] (SPARK-58815) Make attribute binding collision-safe in DSv2 row-level operations

["Eames Trinh (Jira)" <ji...@apache.org>]

Eames Trinh created SPARK-58815:
-----------------------------------

             Summary: Make attribute binding collision-safe in DSv2 row-level operations
                 Key: SPARK-58815
                 URL: https://issues.apache.org/jira/browse/SPARK-58815
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.2.0
            Reporter: Eames Trinh


DSv2 row-level operation rewrites sometimes resolve attributes by name after connector references and Spark-generated attributes have already been added to the plan. Attribute names are not unique, so a user column can shadow a metadata column or an internal attribute.

This problem also occurs with ordinary top-level references; it is not limited to nested row IDs.

For example, a scan may contain:

{{}}
{code:java}

{code}
{{index#1               user data column }}
{{__metadata_index#2    metadata column with logical name "index"}}

 

If {{SupportsDelta.rowId()}} returns {{{}FieldReference("index"){}}}, the rewrite may select the nullable user column instead of the non-null metadata column. This can produce {{NULLABLE_ROW_ID_ATTRIBUTES}} or pass the wrong value to the writer.

MERGE has a similar problem. Spark generates an attribute named {{__row_id}} for cardinality validation and later locates it by name. If the target table already contains a user column named {{{}__row_id{}}}, {{MergeRows}} can bind the user column and incorrectly report {{{}MERGE_CARDINALITY_VIOLATION{}}}.

The same issue can affect required metadata attributes, saved row-ID values, generated MERGE presence markers, write distribution and ordering, and runtime group filtering.

Once a connector or system reference has been resolved, row-level rewrites should carry the resolved {{Attribute}} identity through generated plans and bind later projections using {{{}exprId{}}}. They should not recover the attribute by name.

Expected behavior:
 * User columns cannot shadow connector metadata or Spark-generated attributes.
 * Row IDs and required metadata retain their identity through {{{}Project{}}}, {{{}Expand{}}}, {{{}MergeRows{}}}, write preparation, and runtime filtering.
 * Existing behavior remains unchanged when names are unique.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org