You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/05/18 10:54:34 UTC
[GitHub] [iceberg] liuzx8888 opened a new issue #2604: MERGE INTO TABLE is not supported temporarily.
liuzx8888 opened a new issue #2604:
URL: https://github.com/apache/iceberg/issues/2604
i'm trying to execute a Merge operation between 2 iceberg tables and it seems i get the following error:
The setup is the following:
```
<scala.version>2.12.0</scala.version>
<spark.version>3.0.2</spark.version>
<iceberg.version>0.11.1</iceberg.version>
<hadoop.version>3.1.1</hadoop.version>
<druid.version>1.1.21</druid.version>
<kafka.version>2.0.0</kafka.version>
<hive.version>3.1.1</hive.version>
```
```
val spark = SparkSession
.builder()
.master("local[2]")
.appName("IcebergAPI")
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hive")
.config("spark.sql.catalog.hive_prod", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.hive_prod.type", "hive")
.config("spark.sql.catalog.hive_prod.warehouse", "hdfs://ambari5:8020/warehouse/tablespace/managed/hive")
.config("spark.sql.catalog.hive_prod.uri", "thrift://ambari6:9083")
.config("iceberg.engine.hive.enabled", "true")
.enableHiveSupport()
.getOrCreate()
```
now i can execute following sql susseccful
spark.sql("create database if not exists hive_prod.ods2")
spark.sql("CREATE TABLE IF NOT EXISTS hive_prod.ods.table4 (id bigint, data string) USING iceberg")
spark.sql("INSERT INTO hive_prod.ods.table4 VALUES (1, 'aa'), (2, 'bb'), (3, 'cc')")
spark.sql("select * from hive_prod.ods.table4").show()
Do you have any idea why it's acting like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liuzx8888 commented on issue #2604: MERGE INTO TABLE is not supported temporarily.
Posted by GitBox <gi...@apache.org>.
liuzx8888 commented on issue #2604:
URL: https://github.com/apache/iceberg/issues/2604#issuecomment-843273321
@RussellSpitzer yes,it happened in my notebook idea, i can execute merge susseccful in ${SPARK_HOME}/bin/spark-sql
use like
`./spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.1 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hive \
--conf spark.sql.catalog.local.uri=thrift://ambari6:9083 \
--conf spark.sql.catalog.local.warehouse=hdfs://ambari5:8020/warehouse/tablespace/managed/hive`
but i faild when config the same property in my source code.
table is not iceberg I think Extensions are not loaded is quite possible, but in my IDEA program, I do not know how to detect it. I tried to execute spark.conf.set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"), but it reported an error.
Now I don't know how to load Extensions successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liuzx8888 closed issue #2604: MERGE INTO TABLE is not supported temporarily.
Posted by GitBox <gi...@apache.org>.
liuzx8888 closed issue #2604:
URL: https://github.com/apache/iceberg/issues/2604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liuzx8888 edited a comment on issue #2604: MERGE INTO TABLE is not supported temporarily.
Posted by GitBox <gi...@apache.org>.
liuzx8888 edited a comment on issue #2604:
URL: https://github.com/apache/iceberg/issues/2604#issuecomment-843273321
@RussellSpitzer yes,it happend in my notebook idea, i can execute merge susseccful in ${SPARK_HOME}/bin/spark-sql
use like
`./spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.1 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hive \
--conf spark.sql.catalog.local.uri=thrift://ambari6:9083 \
--conf spark.sql.catalog.local.warehouse=hdfs://ambari5:8020/warehouse/tablespace/managed/hive`
but i faild when config the same property in my source code.
table is not iceberg I think Extensions are not loaded is quite possible, but in my IDEA program, I do not know how to detect it. I tried to execute spark.conf.set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"), but it reported an error.
Now I don't know how to load Extensions successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #2604: MERGE INTO TABLE is not supported temporarily.
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #2604:
URL: https://github.com/apache/iceberg/issues/2604#issuecomment-843212802
I wrote this on another issue recently, I think usually you have two places where this could break.
Target table is not iceberg
Extensions are not loaded
In this case I'm guessing you are possibly making the session before setting the extensions property. This could happen in a notebook or shell which creates the session in an init script. Could you try moving the extensions config into a command line --conf option or some other method which sets initial spark conf?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liuzx8888 edited a comment on issue #2604: MERGE INTO TABLE is not supported temporarily.
Posted by GitBox <gi...@apache.org>.
liuzx8888 edited a comment on issue #2604:
URL: https://github.com/apache/iceberg/issues/2604#issuecomment-843273321
@RussellSpitzer yes,it happend in my notebook idea, i can execute merge susseccful in ${SPARK_HOME}/bin/spark-sql
use like
`./spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.1 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hive \
--conf spark.sql.catalog.local.uri=thrift://ambari6:9083 \
--conf spark.sql.catalog.local.warehouse=hdfs://ambari5:8020/warehouse/tablespace/managed/hive`
but i faild when config the same property in my source code.
table is iceberg I think Extensions are not loaded is quite possible, but in my IDEA program, I do not know how to detect it. I tried to execute spark.conf.set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"), but it reported an error.
Now I don't know how to load Extensions successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liuzx8888 commented on issue #2604: MERGE INTO TABLE is not supported temporarily.
Posted by GitBox <gi...@apache.org>.
liuzx8888 commented on issue #2604:
URL: https://github.com/apache/iceberg/issues/2604#issuecomment-843309591
@RussellSpitzer Thanks Russell, that was it ! Now it can run normally, thank you very, very, very much, you are such a great person
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #2604: MERGE INTO TABLE is not supported temporarily.
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #2604:
URL: https://github.com/apache/iceberg/issues/2604#issuecomment-843280647
The issue here is that SparkSession.getOrCreate will use any session that already exists if one does. In this case the extensions are not applied. You cannot modify the SparkConf after it has been made which is why you get the "can't modify" spark conf error. So in your case you have basically 2 options I think ...
1. find the original session creation in your code and add the property there, if this is a notebook usually you can just set the properties in the kernel definition. If this is your own code you need to find the first init yourself.
2. Force the creation of a new session (newSession) and make sure you are using that. I'm not sure this will pick up the session.conf from the old session but it couldn't hurt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #2604: MERGE INTO TABLE is not supported temporarily.
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #2604:
URL: https://github.com/apache/iceberg/issues/2604#issuecomment-843310367
Ha no problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org