[GitHub] [iceberg] flyxu1991 opened a new issue #3826: Spark merge into iceberg report UPDATE and DELETE can appear at most once in MATCHED clauses err

[GitBox <gi...@apache.org>]

flyxu1991 opened a new issue #3826:
URL: https://github.com/apache/iceberg/issues/3826


   Refer to iceberg website, use SparkSql-client test merge into iceberg, there is an err in query: UPDATE and DELETE can appear at most once in MATCHED clauses in a MERGE statement.
   Here is my code:
   `create table merge_into_test_a (
       name string,
       score int,
       ts string,
       is_valid int,
       valid_time string
   ) using iceberg
   TBLPROPERTIES (
       'format-version' = '1',
       'write.parallelism' = '1'
   );
   
   create table merge_into_test_b (
       name string,
       score int,
       ts string,
       is_valid int,
       valid_time string
   ) using iceberg
   TBLPROPERTIES (
       'format-version' = '1',
       'write.parallelism' = '1'
   );
   
   merge into merge_into_test_a a
   using (
       select name, score, ts, is_valid, valid_time
       from (
           select name, score, ts, is_valid, valid_time,
               ROW_NUMBER() OVER(partition by name order by ts desc) AS rk
           from merge_into_test_b
       ) t
       where rk = 1
   ) b
   on a.name = b.name
   when matched and a.is_valid = 0 and b.ts > a.ts then update set *
   when matched and a.is_valid = 1 and b.ts > a.ts then update set a.score = b.score, a.ts = b.ts
   when not matched then insert *;
   `
   May you help me to solve this err
   @RussellSpitzer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] RussellSpitzer commented on issue #3826: Spark merge into iceberg report UPDATE and DELETE can appear at most once in MATCHED clauses err

[GitBox <gi...@apache.org>]

RussellSpitzer commented on issue #3826:
URL: https://github.com/apache/iceberg/issues/3826#issuecomment-1002999685


   Please in the future post the versions of the software you are using, this particular error is a Spark 3.0 error that is solved in Spark 3.1. 
   
   Spark 3.0 only supports 2 conditions in a merge statement in the parser and has restrictions on their type while 3.1 supports an unlimited amount.
   https://issues.apache.org/jira/browse/SPARK-32030
   
   https://github.com/apache/spark/pull/28875/files#diff-26fe6b511f4b8aedf5bdffae0d7c03aba69463c09fa4f881d77520125796e0faL476-L477
   
   To fix this issue use Spark 3.1 or greater, ideally you should always use the latest Spark if you want to take advantage of the latest features.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] flyxu1991 commented on issue #3826: Spark merge into iceberg report UPDATE and DELETE can appear at most once in MATCHED clauses err

[GitBox <gi...@apache.org>]

flyxu1991 commented on issue #3826:
URL: https://github.com/apache/iceberg/issues/3826#issuecomment-1003235156


   Thanks for your help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org