[GitHub] [iceberg] singhpk234 commented on pull request #5679: Spark: Fix MERGE INTO Query failure on tables with non-nullable columns

[GitBox <gi...@apache.org>]

singhpk234 commented on PR #5679:
URL: https://github.com/apache/iceberg/pull/5679#issuecomment-1233328664

   Posting what I found so far : 
   In case of MOR whats happening is due to 
   https://github.com/apache/iceberg/blob/5764c93a59d8fe5d1c8a60c6ec2cc4deaf8f23d4/spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala#L267
   
   nullifies all row attrs that are not part of the row ID. 
   
   In mergeRow's output, all rowAttrs that are not part of rowID becomes nullbable, becuase delete actions are not actually ignored here for MOR as they are non-empty, unlike what comment suggests (i think this implies for COW, as we return empty in delete action).
   
   https://github.com/apache/iceberg/blob/5764c93a59d8fe5d1c8a60c6ec2cc4deaf8f23d4/spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala#L353-L360
   
   
   because of this WriteDelta's resolution fails due to table.opt suggesting non-null and projection.schema being nullable.
   
   https://github.com/apache/iceberg/blob/5764c93a59d8fe5d1c8a60c6ec2cc4deaf8f23d4/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/WriteDelta.scala#L107-L113
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org