You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Fokko Driesprong <fo...@apache.org> on 2024/11/14 09:01:43 UTC
[DISCUSS] Removal of last-column-id of public API
Hi everyone,
While reviewing the TableMetadataBuilder PR on Iceberg-Rust
<https://github.com/apache/iceberg-rust/pull/587#discussion_r1834400220>
the other day, I noticed that it exposes the last-column-id to the public
API, but I believe there is no need for it. This field is used to determine
the next field-id when adding new fields to a schema. The last-column-id was
added to the REST spec <https://github.com/apache/iceberg/pull/7445> a
while ago, to make the spec in line with the reference implementation, but
in hindsight, it should have been the other way around.
My suggestion is to deprecate and remove this field from the spec and code
<https://github.com/apache/iceberg/pull/11514/>, as I can't think of any
use case where you want to make jumps in the last-column-id (it has to be
monotonically increasing). This will help clean up the APIs and the
reference implementation.
Would love to hear everyone's thoughts on this!
Kind regards,
Fokko
Re: [DISCUSS] Removal of last-column-id of public API
Posted by Renjie Liu <li...@gmail.com>.
+1 for deprecating and removing this. I don't think it makes sense to ask
users to pass this while we can infer it from existing arguments.
Also I see java code
<https://github.com/apache/iceberg/blob/821aec32bb9be28d9c1905f772d9e3101cc98d9e/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1085>
has
comments to remove this. Since this is a public api, should we annotate it
as deprecated in 1.x, and remove it in 2.0?
On Sat, Nov 16, 2024 at 3:11 AM Kevin Liu <ke...@gmail.com> wrote:
> Thanks for bringing this up Fokko.
> It makes sense to hide `last-column-id` from the public API, as it is an
> implementation detail.
>
> As mentioned in the PR, I checked references to `last-column-id`
> <https://grep.app/search?current=4&q=last-column-id&filter[lang][0]=Python&filter[lang][1]=Rust&filter[lang][2]=Java&filter[lang][3]=Go&filter[lang][4]=C%2B%2B>
> and `last_column_id` <https://grep.app/search?q=last_column_id> and
> didn't find anything that would break due to this change.
>
> We would likely need to also deprecate this in PyIceberg as well.
>
> https://github.com/apache/iceberg-python/blob/b2f0a9e5cd7dd548e19cdcdd7f9205f03454369a/pyiceberg/table/update/__init__.py#L90-L91
>
> Best,
> Kevin Liu
>
>
> On Thu, Nov 14, 2024 at 1:13 AM Fokko Driesprong <fo...@apache.org> wrote:
>
>> Hi everyone,
>>
>> While reviewing the TableMetadataBuilder PR on Iceberg-Rust
>> <https://github.com/apache/iceberg-rust/pull/587#discussion_r1834400220>
>> the other day, I noticed that it exposes the last-column-id to the public
>> API, but I believe there is no need for it. This field is used to determine
>> the next field-id when adding new fields to a schema. The last-column-id was
>> added to the REST spec <https://github.com/apache/iceberg/pull/7445> a
>> while ago, to make the spec in line with the reference implementation, but
>> in hindsight, it should have been the other way around.
>>
>> My suggestion is to deprecate and remove this field from the spec and
>> code <https://github.com/apache/iceberg/pull/11514/>, as I can't think
>> of any use case where you want to make jumps in the last-column-id (it has
>> to be monotonically increasing). This will help clean up the APIs and the
>> reference implementation.
>>
>> Would love to hear everyone's thoughts on this!
>>
>> Kind regards,
>> Fokko
>>
>>
>>
Re: [DISCUSS] Removal of last-column-id of public API
Posted by Kevin Liu <ke...@gmail.com>.
Thanks for bringing this up Fokko.
It makes sense to hide `last-column-id` from the public API, as it is an
implementation detail.
As mentioned in the PR, I checked references to `last-column-id`
<https://grep.app/search?current=4&q=last-column-id&filter[lang][0]=Python&filter[lang][1]=Rust&filter[lang][2]=Java&filter[lang][3]=Go&filter[lang][4]=C%2B%2B>
and `last_column_id` <https://grep.app/search?q=last_column_id> and didn't
find anything that would break due to this change.
We would likely need to also deprecate this in PyIceberg as well.
https://github.com/apache/iceberg-python/blob/b2f0a9e5cd7dd548e19cdcdd7f9205f03454369a/pyiceberg/table/update/__init__.py#L90-L91
Best,
Kevin Liu
On Thu, Nov 14, 2024 at 1:13 AM Fokko Driesprong <fo...@apache.org> wrote:
> Hi everyone,
>
> While reviewing the TableMetadataBuilder PR on Iceberg-Rust
> <https://github.com/apache/iceberg-rust/pull/587#discussion_r1834400220>
> the other day, I noticed that it exposes the last-column-id to the public
> API, but I believe there is no need for it. This field is used to determine
> the next field-id when adding new fields to a schema. The last-column-id was
> added to the REST spec <https://github.com/apache/iceberg/pull/7445> a
> while ago, to make the spec in line with the reference implementation, but
> in hindsight, it should have been the other way around.
>
> My suggestion is to deprecate and remove this field from the spec and code
> <https://github.com/apache/iceberg/pull/11514/>, as I can't think of any
> use case where you want to make jumps in the last-column-id (it has to be
> monotonically increasing). This will help clean up the APIs and the
> reference implementation.
>
> Would love to hear everyone's thoughts on this!
>
> Kind regards,
> Fokko
>
>
>
Re: [DISCUSS] Removal of last-column-id of public API
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Fokko
I think it makes sense to deprecate and remote the field.
+1
Regards
JB
On Thu, Nov 14, 2024 at 10:01 AM Fokko Driesprong <fo...@apache.org> wrote:
>
> Hi everyone,
>
> While reviewing the TableMetadataBuilder PR on Iceberg-Rust the other day, I noticed that it exposes the last-column-id to the public API, but I believe there is no need for it. This field is used to determine the next field-id when adding new fields to a schema. The last-column-id was added to the REST spec a while ago, to make the spec in line with the reference implementation, but in hindsight, it should have been the other way around.
>
> My suggestion is to deprecate and remove this field from the spec and code, as I can't think of any use case where you want to make jumps in the last-column-id (it has to be monotonically increasing). This will help clean up the APIs and the reference implementation.
>
> Would love to hear everyone's thoughts on this!
>
> Kind regards,
> Fokko
>
>