You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/08/27 13:20:53 UTC
[GitHub] [iceberg] RamakrishnaChilaka opened a new issue #3042: Help needed in migrating parquet file to iceberg table.
RamakrishnaChilaka opened a new issue #3042:
URL: https://github.com/apache/iceberg/issues/3042
Can someone point me to some documentation ? Please. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhaoyuxiaoxiao commented on issue #3042: Help needed in migrating parquet file to iceberg table.
Posted by GitBox <gi...@apache.org>.
zhaoyuxiaoxiao commented on issue #3042:
URL: https://github.com/apache/iceberg/issues/3042#issuecomment-915730858
I have a problem. Spark migrated form V1, and I changed it into V2 form with ALTER. But table V2 requires a primary key field. How do I add this field@RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #3042: Help needed in migrating parquet file to iceberg table.
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #3042:
URL: https://github.com/apache/iceberg/issues/3042#issuecomment-907199158
https://iceberg.apache.org/spark-procedures/#table-migration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RamakrishnaChilaka commented on issue #3042: Help needed in migrating parquet file to iceberg table.
Posted by GitBox <gi...@apache.org>.
RamakrishnaChilaka commented on issue #3042:
URL: https://github.com/apache/iceberg/issues/3042#issuecomment-907203894
@RussellSpitzer , Thanks for this, I was wondering if we can write a dataframe in iceberg format into a folder ? please help. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #3042: Help needed in migrating parquet file to iceberg table.
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #3042:
URL: https://github.com/apache/iceberg/issues/3042#issuecomment-907205576
Not sure exactly what you mean.
Iceberg tables have configurable location so you can put them wherever you like.
The link I provided above will let you migrate an existing Spark Compatible hive or pure file based table into an Iceberg table in the same location. Once the table is an iceberg table you can write to it with the "Iceberg" format in Spark or using the Spark3 DSV2 Catalog api.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhaoyuxiaoxiao commented on issue #3042: Help needed in migrating parquet file to iceberg table.
Posted by GitBox <gi...@apache.org>.
zhaoyuxiaoxiao commented on issue #3042:
URL: https://github.com/apache/iceberg/issues/3042#issuecomment-915920903
@openinx ，Sorry, I found the cause of FLINKSQL error, but I have a question is, how to update V1 to V2 table add primary key field, implement CDC function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on issue #3042: Help needed in migrating parquet file to iceberg table.
Posted by GitBox <gi...@apache.org>.
openinx commented on issue #3042:
URL: https://github.com/apache/iceberg/issues/3042#issuecomment-915738786
@zhaoyuxiaoxiao , What's the specific stacktrace ? And which apache iceberg version (git commit) in your environment ? How did you upgrade your v1 table to v2 (according to the description in [PR](https://github.com/apache/iceberg/pull/2887) ) ?
It will be helpful to locate the solution for you if you can provide the above information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhaoyuxiaoxiao commented on issue #3042: Help needed in migrating parquet file to iceberg table.
Posted by GitBox <gi...@apache.org>.
zhaoyuxiaoxiao commented on issue #3042:
URL: https://github.com/apache/iceberg/issues/3042#issuecomment-915801041
@openinx ，I am using the master compiled version, I am upgrading the table form through sparkSQL，SQL：“alter table hive_prod.test.hive2iceberg_table set TBLPROPERTIES ('engine.hive.enabled' = 'true' , 'format-version' = '2', 'write.upsert.enable' = 'true') ;”
I'm not sure there's a problem with that。
After the upgrade I used flinkSQL test to insert data and he reported an error.
[ERROR] Could not execute SQL statement. Reason:
java.lang.IllegalStateException: Equality field columns shouldn't be empty when configuring to use UPSERT data stream.
Error prompt: It seems that I did not specify a primary key。
Please take a look at it for me. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org