You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Mukund Thakur <mt...@apache.org> on 2026/05/13 18:40:54 UTC

[DISCUSS] CORE: Adding support for repartitioning old partition spec data files

Hi Everyone,
I would like to add support for repartitioning old partition spec data
files, as described in detail below. Please take a look at my PR
https://github.com/apache/iceberg/pull/16190.

Improvement

Problem:
How to efficiently and reliably repartition only the data files belonging
to the old partition specification so they conform to the new partition
specification, without unnecessarily rewriting or impacting data files
already written using the new spec?

Example:
Suppose we have evolved the table's partition specification by adding a new
partition field, day, on top of an existing field, month. After a few
months, we want to re-partition all the old month data files to follow
partitioning by day. Currently if those files are already of the desired
data sizes, they won't get picked up and thus will remain partitioned by
the old spec only.

Explored solution using existing code and feature flags:

As our use case is to rewrite the old partition spec data files to new spec
data files, we have to use rewrite-all=true as rewrite job will skip the
files which are already of desired size for example (512 MB by default) or
if only one file per group rewrite but we would still need to rewrite them
to desired spec.
Based on a suggestion by @pvary <https://github.com/pvary> on an old PR ##12083
(comment)
<https://github.com/apache/iceberg/pull/12083#issuecomment-2751808447> and
looking at the current code, I thought we can use filters to filter only
the old data files after applying rewrite-all=true based on some column
values for example timestamp( month <=2025-06) for rewriting. To
efficiently rewrite a huge number of data files we have to also use
partial-progress.enabled and partial-progress.max-commits such that if job
fails half way we don't need to start from scratch.

Why this won't work?

Suppose there are so many files to rewrite and jobs fail half way. When we
rerun using the same filter, it will again pick up the same files even if
we have rewritten suppose 50% of files successfully. We can somehow improve
the filter to pick only old files after every iteration but it puts a lot
of work on end-user as currently we can't filter data files based on the
spec ID.

Suggested code change

Based on above reasons, I suggest to enable this use case using this new
flag rewrite-partition-spec-mismatch and partial-progress.enabled.

Happy to try out any other suggestion for achieving the use case.


Thanks,

Mukund

Re: [DISCUSS] CORE: Adding support for repartitioning old partition spec data files

Posted by Cheng Pan <pa...@gmail.com>.
+1 for this.

I just experienced the migration from Hive to Iceberg (with new partition spec to adapt SPJ) cause SPJ does not work and finally identified the root cause is the Iceberg table scan mixed the old data files that does not fits the current partition spec.

Thanks,
Cheng Pan



> On Jun 24, 2026, at 10:02, Imran Rashid <ir...@cloudera.com> wrote:
> 
> Hi all,
> 
> I wanted to chime in here and say how this is important for real workloads involving large Iceberg tables.
> 
> 1. While partition evolution and hidden partitioning are great, sometimes you really want to get all the data into the new partitioning scheme for performance.  In particular, Storage Partitioned Joins require tables to have a consistent partitioning strategy.  SPJ can be an enormous performance improvement, especially for ingesting a small additional dataset into an existing large table.  I have a small demo of SPJ's requirements around partitioning here: https://github.com/squito/spark_iceberg_spj_experiments
> 
> 2. If you've got a large table, the existing tools for migrating old data to the new partitioning scheme don't work very well.  You can't make incremental progress, which is a problem when you've already got a large table or a tight window where these kinds of maintenance operations are possible.  We have real customer workloads with both of these constraints -- 15 TB of data and a small amount of downtime on the cluster before resources are consumed by production jobs with tight SLAs
> 
> Full disclosure, I work with Mukund so I'm speaking about the same use cases that motivated his work.
> 
> thanks for considering this
> Imran Rashid
> 
> On Wed, Jun 10, 2026 at 2:20 PM Mukund Thakur <mthakur@apache.org <ma...@apache.org>> wrote:
>> Could someone take a look at this please and provide some feedback. Thanks. 
>> I am also working on an optimized re partitioning algorithm via PR #16515
>> 
>> On Wed, May 13, 2026 at 1:40 PM Mukund Thakur <mthakur@apache.org <ma...@apache.org>> wrote:
>>> Hi Everyone, 
>>> I would like to add support for repartitioning old partition spec data files, as described in detail below. Please take a look at my PR https://github.com/apache/iceberg/pull/16190. 
>>> 
>>> Improvement
>>> 
>>> Problem:
>>> How to efficiently and reliably repartition only the data files belonging to the old partition specification so they conform to the new partition specification, without unnecessarily rewriting or impacting data files already written using the new spec?
>>> 
>>> Example:
>>> Suppose we have evolved the table's partition specification by adding a new partition field, day, on top of an existing field, month. After a few months, we want to re-partition all the old month data files to follow partitioning by day. Currently if those files are already of the desired data sizes, they won't get picked up and thus will remain partitioned by the old spec only.
>>> 
>>> Explored solution using existing code and feature flags:
>>> 
>>> As our use case is to rewrite the old partition spec data files to new spec data files, we have to use rewrite-all=true as rewrite job will skip the files which are already of desired size for example (512 MB by default) or if only one file per group rewrite but we would still need to rewrite them to desired spec.
>>> Based on a suggestion by @pvary <https://github.com/pvary> on an old PR ##12083 (comment) <https://github.com/apache/iceberg/pull/12083#issuecomment-2751808447> and looking at the current code, I thought we can use filters to filter only the old data files after applying rewrite-all=true based on some column values for example timestamp( month <=2025-06) for rewriting. To efficiently rewrite a huge number of data files we have to also use partial-progress.enabled and partial-progress.max-commits such that if job fails half way we don't need to start from scratch.
>>> 
>>> Why this won't work?
>>> 
>>> Suppose there are so many files to rewrite and jobs fail half way. When we rerun using the same filter, it will again pick up the same files even if we have rewritten suppose 50% of files successfully. We can somehow improve the filter to pick only old files after every iteration but it puts a lot of work on end-user as currently we can't filter data files based on the spec ID.
>>> 
>>> Suggested code change
>>> 
>>> Based on above reasons, I suggest to enable this use case using this new flag rewrite-partition-spec-mismatch and partial-progress.enabled. 
>>> 
>>> Happy to try out any other suggestion for achieving the use case.
>>> 
>>> 
>>> 
>>> Thanks, 
>>> 
>>> Mukund
>>> 


Re: [DISCUSS] CORE: Adding support for repartitioning old partition spec data files

Posted by Imran Rashid <ir...@cloudera.com>.
Hi all,

I wanted to chime in here and say how this is important for real workloads
involving large Iceberg tables.

1. While partition evolution and hidden partitioning are great, sometimes
you really want to get all the data into the new partitioning scheme for
performance.  In particular, Storage Partitioned Joins require tables to
have a consistent partitioning strategy.  SPJ can be an enormous
performance improvement, especially for ingesting a small additional
dataset into an existing large table.  I have a small demo of SPJ's
requirements around partitioning here:
https://github.com/squito/spark_iceberg_spj_experiments

2. If you've got a large table, the existing tools for migrating old data
to the new partitioning scheme don't work very well.  You can't make
incremental progress, which is a problem when you've already got a large
table or a tight window where these kinds of maintenance operations are
possible.  We have real customer workloads with both of these constraints
-- 15 TB of data and a small amount of downtime on the cluster before
resources are consumed by production jobs with tight SLAs

Full disclosure, I work with Mukund so I'm speaking about the same use
cases that motivated his work.

thanks for considering this
Imran Rashid

On Wed, Jun 10, 2026 at 2:20 PM Mukund Thakur <mt...@apache.org> wrote:

> Could someone take a look at this please and provide some feedback.
> Thanks.
> I am also working on an optimized re partitioning algorithm via PR #16515
>
> On Wed, May 13, 2026 at 1:40 PM Mukund Thakur <mt...@apache.org> wrote:
>
>> Hi Everyone,
>> I would like to add support for repartitioning old partition spec data
>> files, as described in detail below. Please take a look at my PR
>> https://github.com/apache/iceberg/pull/16190.
>>
>> Improvement
>>
>> Problem:
>> How to efficiently and reliably repartition only the data files belonging
>> to the old partition specification so they conform to the new partition
>> specification, without unnecessarily rewriting or impacting data files
>> already written using the new spec?
>>
>> Example:
>> Suppose we have evolved the table's partition specification by adding a
>> new partition field, day, on top of an existing field, month. After a few
>> months, we want to re-partition all the old month data files to follow
>> partitioning by day. Currently if those files are already of the desired
>> data sizes, they won't get picked up and thus will remain partitioned by
>> the old spec only.
>>
>> Explored solution using existing code and feature flags:
>>
>> As our use case is to rewrite the old partition spec data files to new
>> spec data files, we have to use rewrite-all=true as rewrite job will skip
>> the files which are already of desired size for example (512 MB by default)
>> or if only one file per group rewrite but we would still need to rewrite
>> them to desired spec.
>> Based on a suggestion by @pvary <https://github.com/pvary> on an old PR ##12083
>> (comment)
>> <https://github.com/apache/iceberg/pull/12083#issuecomment-2751808447> and
>> looking at the current code, I thought we can use filters to filter only
>> the old data files after applying rewrite-all=true based on some column
>> values for example timestamp( month <=2025-06) for rewriting. To
>> efficiently rewrite a huge number of data files we have to also use
>> partial-progress.enabled and partial-progress.max-commits such that if job
>> fails half way we don't need to start from scratch.
>>
>> Why this won't work?
>>
>> Suppose there are so many files to rewrite and jobs fail half way. When
>> we rerun using the same filter, it will again pick up the same files even
>> if we have rewritten suppose 50% of files successfully. We can somehow
>> improve the filter to pick only old files after every iteration but it puts
>> a lot of work on end-user as currently we can't filter data files based on
>> the spec ID.
>>
>> Suggested code change
>>
>> Based on above reasons, I suggest to enable this use case using this new
>> flag rewrite-partition-spec-mismatch and partial-progress.enabled.
>>
>> Happy to try out any other suggestion for achieving the use case.
>>
>>
>> Thanks,
>>
>> Mukund
>>
>

Re: [DISCUSS] CORE: Adding support for repartitioning old partition spec data files

Posted by Mukund Thakur <mt...@apache.org>.
Could someone take a look at this please and provide some feedback. Thanks.
I am also working on an optimized re partitioning algorithm via PR #16515

On Wed, May 13, 2026 at 1:40 PM Mukund Thakur <mt...@apache.org> wrote:

> Hi Everyone,
> I would like to add support for repartitioning old partition spec data
> files, as described in detail below. Please take a look at my PR
> https://github.com/apache/iceberg/pull/16190.
>
> Improvement
>
> Problem:
> How to efficiently and reliably repartition only the data files belonging
> to the old partition specification so they conform to the new partition
> specification, without unnecessarily rewriting or impacting data files
> already written using the new spec?
>
> Example:
> Suppose we have evolved the table's partition specification by adding a
> new partition field, day, on top of an existing field, month. After a few
> months, we want to re-partition all the old month data files to follow
> partitioning by day. Currently if those files are already of the desired
> data sizes, they won't get picked up and thus will remain partitioned by
> the old spec only.
>
> Explored solution using existing code and feature flags:
>
> As our use case is to rewrite the old partition spec data files to new
> spec data files, we have to use rewrite-all=true as rewrite job will skip
> the files which are already of desired size for example (512 MB by default)
> or if only one file per group rewrite but we would still need to rewrite
> them to desired spec.
> Based on a suggestion by @pvary <https://github.com/pvary> on an old PR ##12083
> (comment)
> <https://github.com/apache/iceberg/pull/12083#issuecomment-2751808447> and
> looking at the current code, I thought we can use filters to filter only
> the old data files after applying rewrite-all=true based on some column
> values for example timestamp( month <=2025-06) for rewriting. To
> efficiently rewrite a huge number of data files we have to also use
> partial-progress.enabled and partial-progress.max-commits such that if job
> fails half way we don't need to start from scratch.
>
> Why this won't work?
>
> Suppose there are so many files to rewrite and jobs fail half way. When we
> rerun using the same filter, it will again pick up the same files even if
> we have rewritten suppose 50% of files successfully. We can somehow improve
> the filter to pick only old files after every iteration but it puts a lot
> of work on end-user as currently we can't filter data files based on the
> spec ID.
>
> Suggested code change
>
> Based on above reasons, I suggest to enable this use case using this new
> flag rewrite-partition-spec-mismatch and partial-progress.enabled.
>
> Happy to try out any other suggestion for achieving the use case.
>
>
> Thanks,
>
> Mukund
>