You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Amogh Jahagirdar <2a...@gmail.com> on 2024/08/19 22:18:55 UTC
[DISCUSS] Adding RemovePartitionSpecsUpdate update type to REST
Hi all,
There has been work [1] to enable users to remove historical partition
specs which are not referenced in manifests as a form of metadata cleanup.
As part of this, a new metadata update type RemovePartitionSpecsUpdate
needs to be added to enable REST Catalogs to be able to perform this
operation.
I have a PR [2] with the REST spec change and am starting this thread to
get feedback on the addition of this metadata update to the spec. Note:
REST servers which receive an unknown type must fail with a 400 response as
discussed/voted earlier. [3]
[1] https://github.com/apache/iceberg/pull/10755
<https://github.com/apache/iceberg/pull/10755>
[2] https://github.com/apache/iceberg/pull/10846/
[3] https://lists.apache.org/thread/99lo7stnprchjzosjcq9k3mns1mq8fwc
Thanks,
Amogh Jahagirdar
Re: [DISCUSS] Adding RemovePartitionSpecsUpdate update type to REST
Posted by Amogh Jahagirdar <2a...@gmail.com>.
Thanks everyone, it seems as though there's general support for this. Since
this has been open for a while for discussion, and this is a spec change I
will start a separate vote thread.
On Thu, Aug 22, 2024 at 12:28 PM Ryan Blue <bl...@databricks.com.invalid>
wrote:
> +1
>
> On Tue, Aug 20, 2024 at 1:56 AM Fokko Driesprong <fo...@apache.org> wrote:
>
>> +1 Thanks for working on this
>>
>> Op di 20 aug 2024 om 04:16 schreef xianjin <xi...@apache.org>:
>>
>>> +1 from my side as well.
>>>
>>> Sent from my iPhone
>>>
>>> On Aug 20, 2024, at 9:09 AM, Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>
>>>
>>> +1, the new spec looks good to me. It seems like the client-side
>>> handling the heavy lifting of figuring out which spec to remove is a
>>> reasonable approach.
>>>
>>> Yufei
>>>
>>>
>>>
>>> On Mon, Aug 19, 2024 at 4:01 PM Anton Okolnychyi <ao...@gmail.com>
>>> wrote:
>>>
>>>> Seems reasonable to me.
>>>>
>>>> - Anton
>>>>
>>>> пн, 19 серп. 2024 р. о 15:19 Amogh Jahagirdar <2a...@gmail.com> пише:
>>>>
>>>>> Hi all,
>>>>>
>>>>> There has been work [1] to enable users to remove historical partition
>>>>> specs which are not referenced in manifests as a form of metadata cleanup.
>>>>> As part of this, a new metadata update type RemovePartitionSpecsUpdate
>>>>> needs to be added to enable REST Catalogs to be able to perform this
>>>>> operation.
>>>>>
>>>>> I have a PR [2] with the REST spec change and am starting this thread
>>>>> to get feedback on the addition of this metadata update to the spec. Note:
>>>>> REST servers which receive an unknown type must fail with a 400 response as
>>>>> discussed/voted earlier. [3]
>>>>>
>>>>> [1] https://github.com/apache/iceberg/pull/10755
>>>>> <https://github.com/apache/iceberg/pull/10755>
>>>>> [2] https://github.com/apache/iceberg/pull/10846/
>>>>> [3] https://lists.apache.org/thread/99lo7stnprchjzosjcq9k3mns1mq8fwc
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Amogh Jahagirdar
>>>>>
>>>>
>
> --
> Ryan Blue
> Databricks
>
Re: [DISCUSS] Adding RemovePartitionSpecsUpdate update type to REST
Posted by Ryan Blue <bl...@databricks.com.INVALID>.
+1
On Tue, Aug 20, 2024 at 1:56 AM Fokko Driesprong <fo...@apache.org> wrote:
> +1 Thanks for working on this
>
> Op di 20 aug 2024 om 04:16 schreef xianjin <xi...@apache.org>:
>
>> +1 from my side as well.
>>
>> Sent from my iPhone
>>
>> On Aug 20, 2024, at 9:09 AM, Yufei Gu <fl...@gmail.com> wrote:
>>
>>
>>
>> +1, the new spec looks good to me. It seems like the client-side handling
>> the heavy lifting of figuring out which spec to remove is a reasonable
>> approach.
>>
>> Yufei
>>
>>
>>
>> On Mon, Aug 19, 2024 at 4:01 PM Anton Okolnychyi <ao...@gmail.com>
>> wrote:
>>
>>> Seems reasonable to me.
>>>
>>> - Anton
>>>
>>> пн, 19 серп. 2024 р. о 15:19 Amogh Jahagirdar <2a...@gmail.com> пише:
>>>
>>>> Hi all,
>>>>
>>>> There has been work [1] to enable users to remove historical partition
>>>> specs which are not referenced in manifests as a form of metadata cleanup.
>>>> As part of this, a new metadata update type RemovePartitionSpecsUpdate
>>>> needs to be added to enable REST Catalogs to be able to perform this
>>>> operation.
>>>>
>>>> I have a PR [2] with the REST spec change and am starting this thread
>>>> to get feedback on the addition of this metadata update to the spec. Note:
>>>> REST servers which receive an unknown type must fail with a 400 response as
>>>> discussed/voted earlier. [3]
>>>>
>>>> [1] https://github.com/apache/iceberg/pull/10755
>>>> <https://github.com/apache/iceberg/pull/10755>
>>>> [2] https://github.com/apache/iceberg/pull/10846/
>>>> [3] https://lists.apache.org/thread/99lo7stnprchjzosjcq9k3mns1mq8fwc
>>>>
>>>> Thanks,
>>>>
>>>> Amogh Jahagirdar
>>>>
>>>
--
Ryan Blue
Databricks
Re: [DISCUSS] Adding RemovePartitionSpecsUpdate update type to REST
Posted by Fokko Driesprong <fo...@apache.org>.
+1 Thanks for working on this
Op di 20 aug 2024 om 04:16 schreef xianjin <xi...@apache.org>:
> +1 from my side as well.
>
> Sent from my iPhone
>
> On Aug 20, 2024, at 9:09 AM, Yufei Gu <fl...@gmail.com> wrote:
>
>
>
> +1, the new spec looks good to me. It seems like the client-side handling
> the heavy lifting of figuring out which spec to remove is a reasonable
> approach.
>
> Yufei
>
>
>
> On Mon, Aug 19, 2024 at 4:01 PM Anton Okolnychyi <ao...@gmail.com>
> wrote:
>
>> Seems reasonable to me.
>>
>> - Anton
>>
>> пн, 19 серп. 2024 р. о 15:19 Amogh Jahagirdar <2a...@gmail.com> пише:
>>
>>> Hi all,
>>>
>>> There has been work [1] to enable users to remove historical partition
>>> specs which are not referenced in manifests as a form of metadata cleanup.
>>> As part of this, a new metadata update type RemovePartitionSpecsUpdate
>>> needs to be added to enable REST Catalogs to be able to perform this
>>> operation.
>>>
>>> I have a PR [2] with the REST spec change and am starting this thread to
>>> get feedback on the addition of this metadata update to the spec. Note:
>>> REST servers which receive an unknown type must fail with a 400 response as
>>> discussed/voted earlier. [3]
>>>
>>> [1] https://github.com/apache/iceberg/pull/10755
>>> <https://github.com/apache/iceberg/pull/10755>
>>> [2] https://github.com/apache/iceberg/pull/10846/
>>> [3] https://lists.apache.org/thread/99lo7stnprchjzosjcq9k3mns1mq8fwc
>>>
>>> Thanks,
>>>
>>> Amogh Jahagirdar
>>>
>>
Re: [DISCUSS] Adding RemovePartitionSpecsUpdate update type to REST
Posted by xianjin <xi...@apache.org>.
+1 from my side as well.
Sent from my iPhone
> On Aug 20, 2024, at 9:09 AM, Yufei Gu <fl...@gmail.com> wrote:
>
>
>
>
> +1, the new spec looks good to me. It seems like the client-side handling
> the heavy lifting of figuring out which spec to remove is a reasonable
> approach.
>
> Yufei
>
>
>
>
>
>
>
>
>
> On Mon, Aug 19, 2024 at 4:01 PM Anton Okolnychyi
> <[aokolnychyi@gmail.com](mailto:aokolnychyi@gmail.com)> wrote:
>
>
>> Seems reasonable to me.
>>
>>
>
>>
>> \- Anton
>>
>>
>
>>
>> пн, 19 серп. 2024 р. о 15:19 Amogh Jahagirdar
<[2amogh@gmail.com](mailto:2amogh@gmail.com)> пише:
>
>>
>>> Hi all,
>>>
>>>
>
>>>
>>> There has been work [1] to enable users to remove historical partition
specs which are not referenced in manifests as a form of metadata cleanup. As
part of this, a new metadata update type RemovePartitionSpecsUpdate needs to
be added to enable REST Catalogs to be able to perform this operation.
>>>
>>>
>
>>>
>>> I have a PR [2] with the REST spec change and am starting this thread to
get feedback on the addition of this metadata update to the spec. Note: REST
servers which receive an unknown type must fail with a 400 response as
discussed/voted earlier. [3]
>>>
>>>
>
>>>
>>> [1] <https://github.com/apache/iceberg/pull/10755>[
](https://github.com/apache/iceberg/pull/10755)
>>>
>>> [2] <https://github.com/apache/iceberg/pull/10846/>
>>>
>>> [3] <https://lists.apache.org/thread/99lo7stnprchjzosjcq9k3mns1mq8fwc>
>>>
>>>
>
>>>
>>> Thanks,
>>>
>>>
>
>>>
>>> Amogh Jahagirdar
Re: [DISCUSS] Adding RemovePartitionSpecsUpdate update type to REST
Posted by Yufei Gu <fl...@gmail.com>.
+1, the new spec looks good to me. It seems like the client-side handling
the heavy lifting of figuring out which spec to remove is a reasonable
approach.
Yufei
On Mon, Aug 19, 2024 at 4:01 PM Anton Okolnychyi <ao...@gmail.com>
wrote:
> Seems reasonable to me.
>
> - Anton
>
> пн, 19 серп. 2024 р. о 15:19 Amogh Jahagirdar <2a...@gmail.com> пише:
>
>> Hi all,
>>
>> There has been work [1] to enable users to remove historical partition
>> specs which are not referenced in manifests as a form of metadata cleanup.
>> As part of this, a new metadata update type RemovePartitionSpecsUpdate
>> needs to be added to enable REST Catalogs to be able to perform this
>> operation.
>>
>> I have a PR [2] with the REST spec change and am starting this thread to
>> get feedback on the addition of this metadata update to the spec. Note:
>> REST servers which receive an unknown type must fail with a 400 response as
>> discussed/voted earlier. [3]
>>
>> [1] https://github.com/apache/iceberg/pull/10755
>> <https://github.com/apache/iceberg/pull/10755>
>> [2] https://github.com/apache/iceberg/pull/10846/
>> [3] https://lists.apache.org/thread/99lo7stnprchjzosjcq9k3mns1mq8fwc
>>
>> Thanks,
>>
>> Amogh Jahagirdar
>>
>
Re: [DISCUSS] Adding RemovePartitionSpecsUpdate update type to REST
Posted by Anton Okolnychyi <ao...@gmail.com>.
Seems reasonable to me.
- Anton
пн, 19 серп. 2024 р. о 15:19 Amogh Jahagirdar <2a...@gmail.com> пише:
> Hi all,
>
> There has been work [1] to enable users to remove historical partition
> specs which are not referenced in manifests as a form of metadata cleanup.
> As part of this, a new metadata update type RemovePartitionSpecsUpdate
> needs to be added to enable REST Catalogs to be able to perform this
> operation.
>
> I have a PR [2] with the REST spec change and am starting this thread to
> get feedback on the addition of this metadata update to the spec. Note:
> REST servers which receive an unknown type must fail with a 400 response as
> discussed/voted earlier. [3]
>
> [1] https://github.com/apache/iceberg/pull/10755
> <https://github.com/apache/iceberg/pull/10755>
> [2] https://github.com/apache/iceberg/pull/10846/
> [3] https://lists.apache.org/thread/99lo7stnprchjzosjcq9k3mns1mq8fwc
>
> Thanks,
>
> Amogh Jahagirdar
>