You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/05/14 14:12:22 UTC
[GitHub] [iceberg] MariusCostin opened a new issue #2593: MERGE INTO TABLE is not supported temporarily
MariusCostin opened a new issue #2593:
URL: https://github.com/apache/iceberg/issues/2593
Hello,
So, i'm trying to execute a Merge operation between 2 iceberg tables and it seems i get the following error:
**java.lang.UnsupportedOperationException: MERGE INTO TABLE is not supported temporarily.**
The setup is the following:
- **Amazon EMR** with **Spark 3.1.0** and also tried it on **EMR** with **Spark 3.0.1**
- Launching the shell with: **spark-shell --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.1,software.amazon.awssdk:bundle:2.15.40,software.amazon.awssdk:url-connection-client:2.15.40**
- Creating the session as follows:
val spark = SparkSession.builder.appName("MyApp")
.config("spark.jars.packages", "org.apache.iceberg:iceberg-spark3-runtime:0.11.0,software.amazon.awssdk:bundle:2.15.40,software.amazon.awssdk:url-connection-client:2.15.40")
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions").config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hive")
.config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.local.type", "hadoop")
.config("spark.sql.catalog.local.warehouse", "s3://random_bucket")
.getOrCreate()
Do you have any idea why it's acting like this? We recently started to research the implementation of iceberg for our data lake and hit this brick wall.
Thanks in advance.
Marius Costin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] MariusCostin edited a comment on issue #2593: MERGE INTO TABLE is not supported temporarily
Posted by GitBox <gi...@apache.org>.
MariusCostin edited a comment on issue #2593:
URL: https://github.com/apache/iceberg/issues/2593#issuecomment-841345789
Thanks Russell for the help. Both tables are iceberg tables as I can read from them and they are saved in S3 with the proper metadata. I might not be doing something correctly or something is not set properly.
I tried to create the app like you suggested:
spark-shell --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.1,software.amazon.awssdk:bundle:2.15.40,software.amazon.awssdk:url-connection-client:2.15.40 --conf spark.sql.extension=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog --conf spark.sql.catalog.spark_catalog.type=hive --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.local.type=hadoop --conf spark.sql.catalog.local.warehouse=s3://randombucket/iceberg
Same output as below:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer edited a comment on issue #2593: MERGE INTO TABLE is not supported temporarily
Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on issue #2593:
URL: https://github.com/apache/iceberg/issues/2593#issuecomment-841375686
Sorry I copied the typo, it's
spark.sql.extensions not spark.sql.extension
https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/SparkSessionExtensions.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] MariusCostin commented on issue #2593: MERGE INTO TABLE is not supported temporarily
Posted by GitBox <gi...@apache.org>.
MariusCostin commented on issue #2593:
URL: https://github.com/apache/iceberg/issues/2593#issuecomment-841345789
Thanks Russell for the help. Both tables are iceberg tables as I can read from them and they are saved in S3 with the proper metadata. I might not be doing something correctly or something is not set properly.
I tried to create the app like you suggested:
spark-shell --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.1,software.amazon.awssdk:bundle:2.15.40,software.amazon.awssdk:url-connection-client:2.15.40 --conf hive.metastore.uris=thrift://ip-172-31-5-53.eu-central-1.compute.internal:9083 --conf spark.sql.extension=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog --conf spark.sql.catalog.spark_catalog.type=hive --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.local.type=hadoop --conf spark.sql.catalog.local.warehouse=s3://randombucket/iceberg
Same output as below:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] MariusCostin commented on issue #2593: MERGE INTO TABLE is not supported temporarily
Posted by GitBox <gi...@apache.org>.
MariusCostin commented on issue #2593:
URL: https://github.com/apache/iceberg/issues/2593#issuecomment-841445736
Thanks Russell, that was it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #2593: MERGE INTO TABLE is not supported temporarily
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #2593:
URL: https://github.com/apache/iceberg/issues/2593#issuecomment-841375686
Sorry I copied the typo, it's
spark.sql.extensions not spark.sql.extensions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #2593: MERGE INTO TABLE is not supported temporarily
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #2593:
URL: https://github.com/apache/iceberg/issues/2593#issuecomment-841291217
This means 1 of two things,
Either the target of the table is not an Iceberg table, or the Spark Extensions was not set when the session was created.
In your case I believe it's number 2, since Spark-Shell already starts up a SparkContext I believe you will in the above code just be deriving a new session from that context, so some of your config options will be treated as runtime rather than as creation time options. This should only be an issue for
`spark.jars.packages` and `spark.sql.extensions`
I believe you should be setting these earlier in the initialization, for spark-shell this means either on the CLI or in a spark-defaults file.
You already cover "spark.jars.packages" with your --packages command so the only thing to add is
"--conf spark.sql.extension=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
Which should fix your issue
See the docs https://iceberg.apache.org/getting-started/#adding-catalogs
For an example.
The catalogs do not need to be configured before the session is initilzaed so you can leave that code as is if you like but I would recommend setting everything on the CLI or by using
spark.conf.set()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer edited a comment on issue #2593: MERGE INTO TABLE is not supported temporarily
Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on issue #2593:
URL: https://github.com/apache/iceberg/issues/2593#issuecomment-841375686
Sorry I copied the typo, it's
spark.sql.extensions not spark.sql.extension
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] MariusCostin closed issue #2593: MERGE INTO TABLE is not supported temporarily
Posted by GitBox <gi...@apache.org>.
MariusCostin closed issue #2593:
URL: https://github.com/apache/iceberg/issues/2593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org