Update existing partition spec

[Laszlo Pinter <lp...@cloudera.com.INVALID>]

Hi Iceberg Devs,

I'm using the Iceberg API to drop/add partition transforms, but recently
I've run into an issue.
When I try to add a new partition that previously existed, but it was
already dropped I get *Cannot use partition name more than once *error
message.

Here is what I'm doing:

table.updateSpec().addField(month("ts")).commit();
table.updateSpec().removeField("ts_month").commit();
table.updateSpec().addField(day("ts")).commit();
table.updateSpec().addField(month("ts")).commit();

What is the correct way to implement this sort of partition evolution?

Thanks,
Laszlo

Re: Update existing partition spec

[126 <vi...@126.com>]

Hi Devs,

I had the same problem. and i commit a PR, try to fix it.
https://github.com/apache/iceberg/pull/2691 <https://github.com/apache/iceberg/pull/2691>
 

Re: Update existing partition spec

["Jun H." <ju...@gmail.com>]

Hi Laszlo,

As Jack mentioned, this behavior happened to v1 format.

If you want to stay with v1 format, a short term workaround is to add back
this field with a different name, e.g.

table.updateSpec().addField("ts_month_new", month("ts")).commit();

or rename the field before deleting it

table.updateSpec().renameField("ts_month", "ts_month_deleted").commit();
table.updateSpec().removeField("ts_month_deleted").commit();

I think this may be fixed by reviving the null transform if a newly added
partition field is identical or compatible with the previously dropped one.
I will take a look at it.

Jun

On Wed, Jun 9, 2021 at 8:27 AM Jack Ye <ye...@gmail.com> wrote:

> Hi Laszlo,
>
> I think this is expected behavior of format v1, where a dropped partition
> column is actually converted to a null transform instead of being actually
> dropped. This is because the partition spec is not versioned in v1.
>
> See
> https://github.com/apache/iceberg/blob/efaad975fc9bbeaa27392b333211e99ac444aaba/core/src/main/java/org/apache/iceberg/BaseUpdatePartitionSpec.java#L222-L226
> for more details.
>
> -Jack Ye
>
> On Wed, Jun 9, 2021 at 6:54 AM Laszlo Pinter <lp...@cloudera.com.invalid>
> wrote:
>
>> Hi Iceberg Devs,
>>
>> I'm using the Iceberg API to drop/add partition transforms, but recently
>> I've run into an issue.
>> When I try to add a new partition that previously existed, but it was
>> already dropped I get *Cannot use partition name more than once *error
>> message.
>>
>> Here is what I'm doing:
>>
>> table.updateSpec().addField(month("ts")).commit();
>> table.updateSpec().removeField("ts_month").commit();
>> table.updateSpec().addField(day("ts")).commit();
>> table.updateSpec().addField(month("ts")).commit();
>>
>> What is the correct way to implement this sort of partition evolution?
>>
>> Thanks,
>> Laszlo
>>
>

Re: Update existing partition spec

[Jack Ye <ye...@gmail.com>]

Hi Laszlo,

I think this is expected behavior of format v1, where a dropped partition
column is actually converted to a null transform instead of being actually
dropped. This is because the partition spec is not versioned in v1.

See
https://github.com/apache/iceberg/blob/efaad975fc9bbeaa27392b333211e99ac444aaba/core/src/main/java/org/apache/iceberg/BaseUpdatePartitionSpec.java#L222-L226
for more details.

-Jack Ye

On Wed, Jun 9, 2021 at 6:54 AM Laszlo Pinter <lp...@cloudera.com.invalid>
wrote:

> Hi Iceberg Devs,
>
> I'm using the Iceberg API to drop/add partition transforms, but recently
> I've run into an issue.
> When I try to add a new partition that previously existed, but it was
> already dropped I get *Cannot use partition name more than once *error
> message.
>
> Here is what I'm doing:
>
> table.updateSpec().addField(month("ts")).commit();
> table.updateSpec().removeField("ts_month").commit();
> table.updateSpec().addField(day("ts")).commit();
> table.updateSpec().addField(month("ts")).commit();
>
> What is the correct way to implement this sort of partition evolution?
>
> Thanks,
> Laszlo
>