You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/05/08 19:42:51 UTC
[GitHub] [incubator-iceberg] rdblue opened a new issue #1027: Add an action to compact delete files
rdblue opened a new issue #1027:
URL: https://github.com/apache/incubator-iceberg/issues/1027
Tables with slowly changing data may have many small delete files, with only a small portion of the data affected. In that case, it may not be worth rewriting data files to remove rows. Iceberg should provide an action to compact delete files to minimize delete files in a task that need to be read and to keep the number of files in memory during job planning lower.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on issue #1027: Add an action to compact delete files
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1027:
URL: https://github.com/apache/incubator-iceberg/issues/1027#issuecomment-625988575
This may not need to run in a processing engine. Since position delete files are already sorted, they can be merged together without using much memory. Using an engine like Spark could make it faster, but it isn't necessarily required.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org