[I] Core: Enable re-partitioning of old spec data files. [iceberg]

["mukund-thakur (via GitHub)" <gi...@apache.org>]

mukund-thakur opened a new issue, #16189:
URL: https://github.com/apache/iceberg/issues/16189

   ### Feature Request / Improvement
   
   Problem:
   How to efficiently and reliably repartition only the data files belonging to the old partition specification so they conform to the new partition specification, without unnecessarily rewriting or impacting data files already written using the new spec?
   
   Example:
   Suppose we have evolved the table's partition specification by adding a new partition field, day, on top of an existing field, month. After few months, we want to re-partition all the old month data files to follow partitioning by day. Currently if those files are already of desired data sizes, they won't get picked up and thus will remain partitioned by old spec only. 
   
   Solution:
   We want to enable this kind of rewrite using a flag. 
   
   ### Query engine
   
   None
   
   ### Willingness to contribute
   
   - [ ] I can contribute this improvement/feature independently
   - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
   - [ ] I cannot contribute this improvement/feature at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Core: Enable re-partitioning of old spec data files. [iceberg]

["mukund-thakur (via GitHub)" <gi...@apache.org>]

mukund-thakur commented on issue #16189:
URL: https://github.com/apache/iceberg/issues/16189#issuecomment-5006779004

   TL;DR : Currently in Apache Iceberg there is not way to rewrite only old partition spec data files into new spec, thus proposing it to enable it using a new flag.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org