You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "madeirak (via GitHub)" <gi...@apache.org> on 2024/04/11 08:22:24 UTC
[I] Can Iceberg old partition with no data be deleted? [iceberg]
madeirak opened a new issue, #10121:
URL: https://github.com/apache/iceberg/issues/10121
### Query engine
spark
### Question
As image above, how to delete partition field as "xxx=null/yyy=null/" with no data in it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #10121:
URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058743091
I think Iceberg v1 tables has a concept of void transform and it will keep the dropped partition still as a void after partition evolution.
Try changing the table default format version table property to 2 and try the partition evolution freshly with rewrite data files again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]
Posted by "madeirak (via GitHub)" <gi...@apache.org>.
madeirak closed issue #10121: Can Iceberg old partition with no data be deleted?
URL: https://github.com/apache/iceberg/issues/10121
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #10121:
URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058678837
how is the table partitioned? Can you do describe table or show create table to show the partition scheme? Also, was there any partition evolution? Are you using V1 Iceberg table or v2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]
Posted by "madeirak (via GitHub)" <gi...@apache.org>.
madeirak commented on issue #10121:
URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058796187
> Lastly I think there is no way to delete empty folders in Iceberg. Only files are tracked by Iceberg metadata. So, GC operations only clean up the files.
thx set version of table to 2, it does fix my problem😘
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]
Posted by "madeirak (via GitHub)" <gi...@apache.org>.
madeirak commented on issue #10121:
URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058730627
> how is the table partitioned? Can you do describe table or show create table to show the partition scheme? Also, was there any partition evolution? Are you using V1 Iceberg table or v2?
thx for reply, this table has indeed changed partitions.
Before, show create table on this table , and get "PARTITIONED BY (old_xxx, old_yyy)", with no data in the table.
Then, the partition fields change to "PARTITIONED BY (new_partition_1, new_partition_2)"
I know this form is used for data co-existence between new and old partitions，with old partition field = null.
but in my case, old partition field has no data in it, I am worried that a long path will affect the efficiency of READ and WRITE,
**so I wonder if there is way to delete the old partition path with no data, or the impact of a long path on the efficiency of READ and WRITE**?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Can Iceberg old partition with no data be deleted? [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #10121:
URL: https://github.com/apache/iceberg/issues/10121#issuecomment-2058745161
Lastly I think there is no way to delete empty folders in Iceberg. Only files are tracked by Iceberg metadata. So, GC operations only clean up the files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org