[GitHub] [iceberg] JingsongLi commented on issue #1159: Avoid rewriting big files in RewriteDataFilesAction

[GitBox <gi...@apache.org>]

JingsongLi commented on issue #1159:
URL: https://github.com/apache/iceberg/issues/1159#issuecomment-657389247


   > I think you can specify filter expressions to rewrite files you wanted. For example, you can only rewrite data under newly added partition, not the whole data. I'm not sure if that is enough for you.
   
   Thanks for reply. Yes, using partition just rewrite once is OK.
   I think we optimize further, if the partition is "day_time + type", some type has enough records in its files, but some others may have less records so leads to small files, so I rewrite the all newly partitions for reduce files but can skip big files in some partitions.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org