[GitHub] [iceberg] RussellSpitzer edited a comment on issue #4074: flink: after rewrite, the two small files are rewritten into the same two small files as before

[GitBox <gi...@apache.org>]

RussellSpitzer edited a comment on issue #4074:
URL: https://github.com/apache/iceberg/issues/4074#issuecomment-1033905799


   This is something that will be fixed a little in the latest release, one of our issues is that the algorithm basically could not handle compacting files which were not of the correct size, but when combined would be too large. Now Files are combined at the offset level so more efficient compaction should be possible. Additionally I believe we have other work that goes through and checks to see if any tasks for rewrite are generated that would end up being a noop and skips them.
   
   #3292  
   
   Ah sorry, I missed this is in the flink implementation but I believe the fix may apply for that engine as well since it's in the split planning code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org