You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/28 07:39:51 UTC
[GitHub] [iceberg] song01010 opened a new issue #3409: After merging small files, the number of files is still large
song01010 opened a new issue #3409:
URL: https://github.com/apache/iceberg/issues/3409
When we successfully write data, the number of small files and the space of small files increase greatly, but after the small file merge on the official website, the number of files does not change, and there is no merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #3409: After merging small files, the number of files is still large
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #3409:
URL: https://github.com/apache/iceberg/issues/3409#issuecomment-953958541
The data rewrite does not remove files, it is a purely additive operation like almost all Iceberg operations. The new snapshot created will refer only to the new files which should be compacted versions of the old small files.
To remove the old small files you must erase the history of the table with `expire snapshots`, only when the table no longer has a history which references the pre-compacted files will it physically remove them. Be sure to check the parameters of `expire snapshots` when using it as the defaults are conservative to prevent erasing history which may be in use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] KnightChess commented on issue #3409: After merging small files, the number of files is still large
Posted by GitBox <gi...@apache.org>.
KnightChess commented on issue #3409:
URL: https://github.com/apache/iceberg/issues/3409#issuecomment-954379239
> thanks for U,but I userd 'rewriteDataFiles' to merge small file,then use 'table.expireSnapshots().expireOlderThan(table.currentSnapshot().timestampMillis())' to delete expire snapshot and data file . But,I found my data directory‘s file is no significant reduction
doubt: actions.expireSnapshots(table).execute(), if you forgot execute()
Confirm that the historical manifest file is deleted.
if not, check parameters of snapshot like` history.expire.min-snapshots-to-keep`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] SongJok commented on issue #3409: After merging small files, the number of files is still large
Posted by GitBox <gi...@apache.org>.
SongJok commented on issue #3409:
URL: https://github.com/apache/iceberg/issues/3409#issuecomment-954353168
thanks for U,but I userd 'rewriteDataFiles' to merge small file,then use 'table.expireSnapshots().expireOlderThan(table.currentSnapshot().timestampMillis())' to delete expire snapshot and data file .
But,I found my data directory‘s file is no significant reduction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] SongJok commented on issue #3409: After merging small files, the number of files is still large
Posted by GitBox <gi...@apache.org>.
SongJok commented on issue #3409:
URL: https://github.com/apache/iceberg/issues/3409#issuecomment-954476265
Excuse me
I simply draw a picture, which describes some of the problems I encountered, because the drawing used Chinese, please forgive me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] KnightChess commented on issue #3409: After merging small files, the number of files is still large
Posted by GitBox <gi...@apache.org>.
KnightChess commented on issue #3409:
URL: https://github.com/apache/iceberg/issues/3409#issuecomment-954491092
it looks that the manifest file references the older data file, the new add datafile reference and delete datafile in then same manifest. you can rewriteManifest to remove the delete references
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] KnightChess commented on issue #3409: After merging small files, the number of files is still large
Posted by GitBox <gi...@apache.org>.
KnightChess commented on issue #3409:
URL: https://github.com/apache/iceberg/issues/3409#issuecomment-954493404
you use deprecated function, I am not sure if SparkActions also has this problem, I will research the generation strategy of the manifest later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] KnightChess commented on issue #3409: After merging small files, the number of files is still large
Posted by GitBox <gi...@apache.org>.
KnightChess commented on issue #3409:
URL: https://github.com/apache/iceberg/issues/3409#issuecomment-954507263
> you use deprecated function, I am not sure if SparkActions also has this problem, I will research the generation strategy of the manifest later.
I think this is no problem. I think that rewriting files generate a snapshot, contain add and delete for a operation. If you want remove older data file , you need to generate a new snapshot exclude delete references, like rewritemanifest? I am not sure if the understanding of snapshots and operations is correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] SongJok commented on issue #3409: After merging small files, the number of files is still large
Posted by GitBox <gi...@apache.org>.
SongJok commented on issue #3409:
URL: https://github.com/apache/iceberg/issues/3409#issuecomment-954457306
This is my program
I don't know what I finish that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org