You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/11/15 06:47:08 UTC
[GitHub] [iceberg] 0xffmeta opened a new issue, #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
0xffmeta opened a new issue, #6194:
URL: https://github.com/apache/iceberg/issues/6194
### Apache Iceberg version
0.14.1
### Query engine
Spark
### Please describe the bug 🐞
I'm running below spark sql to migrate an iceberg table from v1 to v2 format:
```
alter table tmp.table SET TBLPROPERTIES('format-version'='2')
```
The spark sql can finish with no error, but it still shows `'option.format-version'='1'` while querying form hive `show create table`.
Do I need to use this table property for the foramt upgrade? `option.format-version`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] gaborkaszab commented on issue #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
Posted by GitBox <gi...@apache.org>.
gaborkaszab commented on issue #6194:
URL: https://github.com/apache/iceberg/issues/6194#issuecomment-1324788433
I found the relevant Hive PR for this change: https://github.com/apache/hive/pull/3778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] 0xffmeta commented on issue #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
Posted by GitBox <gi...@apache.org>.
0xffmeta commented on issue #6194:
URL: https://github.com/apache/iceberg/issues/6194#issuecomment-1318240764
Seems I need to use `alter table tmp.table SET TBLPROPERTIES('option.format-version'='2')` to update the format-version. But I'm not very sure why it required prefix `option.`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] gaborkaszab commented on issue #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
Posted by GitBox <gi...@apache.org>.
gaborkaszab commented on issue #6194:
URL: https://github.com/apache/iceberg/issues/6194#issuecomment-1324786357
@0xffmeta I think Hive currently is not using the 'format-version' table property set by Spark and in turn takes another one that is meant to be stored in HMS to check the version of an Iceberg table. I believe there is active work on this to fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] 0xffmeta commented on issue #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
Posted by GitBox <gi...@apache.org>.
0xffmeta commented on issue #6194:
URL: https://github.com/apache/iceberg/issues/6194#issuecomment-1331528398
Understood. Thanks for the explanation. @gaborkaszab
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] gaborkaszab commented on issue #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
Posted by GitBox <gi...@apache.org>.
gaborkaszab commented on issue #6194:
URL: https://github.com/apache/iceberg/issues/6194#issuecomment-1330841257
This might depend on the client you use. Spark and Impala for sure gets the format-version from Iceberg metadata files (actually by using loadTable() API and the result holds this info) while until Hive fixes the ticket I attached they take this info from Hive Metastore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] 0xffmeta closed issue #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
Posted by GitBox <gi...@apache.org>.
0xffmeta closed issue #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
URL: https://github.com/apache/iceberg/issues/6194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] 0xffmeta commented on issue #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
Posted by GitBox <gi...@apache.org>.
0xffmeta commented on issue #6194:
URL: https://github.com/apache/iceberg/issues/6194#issuecomment-1330005531
Many thanks @gaborkaszab for the information. So I think right now the proper way to check for the table's `format-version` is by checking the `.metadata.json` file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #6194: Spark DDL: upgrade format-version from v1 to v2 is not working
Posted by GitBox <gi...@apache.org>.
Fokko commented on issue #6194:
URL: https://github.com/apache/iceberg/issues/6194#issuecomment-1318232667
It works fine with Spark:
In Hive, could it be that you need to refresh the table? Using `REFRESH tmp.table`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org