You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/09/10 12:29:39 UTC

[GitHub] [iceberg] vsingh99 opened a new issue #3098: MERGE INTO TABLE is not supported temporarily

vsingh99 opened a new issue #3098:
URL: https://github.com/apache/iceberg/issues/3098


   Spark : 3.1.1 (also validated against 3.0.1)
   Scala: 2.12.12
   Iceberg Version: 0.12.0 (also validated against 0.11.x)
   
   **Scenario:**
   1. Table for initial base volume
   spark.sql(s"CREATE TABLE hadoop_prod.logging.customer_initial (${schemaStr}) USING ICEBERG LOCATION 'output/iceberg/warehouse/logging/customer_initial'")
   
   2. Created another table for incremental volume
   spark.sql(s"CREATE TABLE hadoop_prod.logging.customer_incremental (${schemaStr}) USING ICEBERG LOCATION 'output/iceberg/warehouse/logging/customer_incremental'")
   
   3. When I try to MERGE Iceberg throws "MERGE INTO TABLE is not supported temporarily."
   
   **Exception:**
   Exception in thread "main" java.lang.UnsupportedOperationException: MERGE INTO TABLE is not supported temporarily.
   	at org.apache.spark.sql.execution.SparkStrategies$BasicOperators$.apply(SparkStrategies.scala:796)
   	at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$1(QueryPlanner.scala:63)
   	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
   	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
   	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
   
   **Point of Failure:**
   spark.sql("MERGE INTO hadoop_prod.logging.customer_initial AS baseVolume USING hadoop_prod.logging.customer_incremental AS incremental on baseVolume.id = incremental.id WHEN MATCHED THEN UPDATE SET baseVolume.tag= incremental.tag WHEN NOT MATCHED THEN INSERT *")
   
   **Observation:**
   Noticed separate test case and this seems like excepted failure - testMergeWithNonIcebergTargetTableNotSupported.  
   
   https://github.com/apache/iceberg/blob/d1f5ba9dd2b68305b7402e3e79af611f649491a2/spark3-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
   
   Also noticed other testcases for MERGE and not sure how they work i.e. testMergeSinglePartitionPartitioning, testMergeEmptyTable etc.
   
   Tried different combination as per testcase but still getting same exception. 
   
   **Similar Issues;**
   https://www.mail-archive.com/dev@iceberg.apache.org/msg01895.html
   
   But not finding right solution to my problem.
   
   Any pointers why I am getting "MERGE INTO TABLE is not supported temporarily"?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] vsingh99 commented on issue #3098: MERGE INTO TABLE is not supported temporarily

Posted by GitBox <gi...@apache.org>.
vsingh99 commented on issue #3098:
URL: https://github.com/apache/iceberg/issues/3098#issuecomment-916901473


   Thanks for your help.
   
   As suggested added following line and MERGE INTO works as expected.
   
   .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] vsingh99 closed issue #3098: MERGE INTO TABLE is not supported temporarily

Posted by GitBox <gi...@apache.org>.
vsingh99 closed issue #3098:
URL: https://github.com/apache/iceberg/issues/3098


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] RussellSpitzer edited a comment on issue #3098: MERGE INTO TABLE is not supported temporarily

Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on issue #3098:
URL: https://github.com/apache/iceberg/issues/3098#issuecomment-916887328


   My assumption here is that the Spark SQL Extensions are not being configured as specified in the documentation 
   
   ```--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions ```
   
   https://iceberg.apache.org/getting-started/
   
   Without this catalyst would be unable to recognize our custom MERGE command.
   
   See https://iceberg.apache.org/spark-configuration/#sql-extensions
   
   For more info
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] RussellSpitzer commented on issue #3098: MERGE INTO TABLE is not supported temporarily

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #3098:
URL: https://github.com/apache/iceberg/issues/3098#issuecomment-916887328


   My assumption here is that the Spark SQL Extensions are not being configured as specified in the documentation 
   
   ```--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions ```
   
   https://iceberg.apache.org/getting-started/
   
   Without this catalyst would be unable to recognize our custom MERGE command.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org