[GitHub] [incubator-iceberg] bryanck commented on issue #480: Don't group files with different partition specs in BaseRewriteManifests

[GitBox <gi...@apache.org>]

bryanck commented on issue #480: Don't group files with different partition specs in BaseRewriteManifests
URL: https://github.com/apache/incubator-iceberg/issues/480#issuecomment-532221369
 
 
   The downside is that this could potentially result in many small manifests. For example, we have a bucketed table with 10k buckets, which is additionally partitioned by date and hour. So for a week of data we'd have 10k * 24 * 7 = 1.68 million manifests. What we do with RewriteManifests today is combine files into groups of four buckets and ignore date and hour, which results in 10k/4 = 2.5k manifests for a week of data. Performance with file pruning is good even though the manifests have multiple partition specs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org