You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Christian Thiel <ch...@gmail.com> on 2025/04/16 12:53:31 UTC

Re: [DISCUSS] Events Endpoint for IRC

Dear all,
after the last Catalog sync I updated the proposal.
Changes are in the original proposal Document [1] and the original PR [2]

Best,
Christian

[1]
https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
[2] https://github.com/apache/iceberg/pull/12584


On Thu, 20 Mar 2025 at 10:49, Christian Thiel <ch...@gmail.com>
wrote:

> Dear all,
>
> We have recently discussed in the Iceberg Catalog Community Sync [1] and
> the Mailing List [2] different ways on how federation between Catalogs
> could be standardized.
>
> This proposal introduces a /events endpoint to the IRC specification. The
> endpoint provides events of modifications to objects managed by the Catalog
> (tables, namespaces, views), allowing consumers to efficiently track
> metadata changes using persistent offsets for reliable consumption and
> resumability.
>
> Proposal Document:
> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>
> I am looking forward to your thoughts and hope we find time in next week's
> Catalog sync to discuss this further.
>
> Best
> Christian
>
> [1] Catalog Community Sync Feb. 2025
> https://www.youtube.com/watch?v=hYcehreE8Nk
> [2] Mailing List, September 2024 - Notifications Endpoint:
> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht
>

Re: [DISCUSS] Events Endpoint for IRC

Posted by Christian Thiel <ch...@gmail.com>.
The motivation is mainly to not add another plan endpoint pre-maturely,
which sees no implementation for a very long time.
The hope was also that during implementation potential problems might be
spotted.

I think if we get 1 or 2 more eyes on the existing PRs we might be able to
get them merged and then also merge the spec. From my side there are no
other blockers.

On Thu, 30 Oct 2025 at 01:28, Yufei Gu <fl...@gmail.com> wrote:

> Thanks, Christian and Ally, for working on the implementation. I
> understand that the implementation is important, but do we need to block
> the REST spec on it?
> Typically, we merge the spec PR first and then work on the implementation.
>
> Yufei
>
>
> On Sun, Oct 19, 2025 at 11:55 PM Christian Thiel <
> christian.t.b101@gmail.com> wrote:
>
>> Thanks @ally for your work on this.
>> Yes, we are currently waiting for Reviews. I am not a Java Maintainer,
>> but maybe Eduard could have another look?
>> @Yufei please have a look at the Java PR as well. Once this is merged, we
>> can proceed with the REST PR.
>>
>> On Sat, 18 Oct 2025 at 09:02, ally heev <al...@gmail.com> wrote:
>>
>>> I think we are waiting on core classes implementation
>>> <https://github.com/apache/iceberg/issues/13580>. I have a PR
>>> <https://github.com/apache/iceberg/pull/14142> for the Request/Response
>>> Objects waiting for review
>>>
>>>
>>> On Sat, Oct 18, 2025 at 5:02 AM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks, do we have updates on this? Are we still pursuing it? I saw
>>>> the PR https://github.com/apache/iceberg/pull/12584/files is in draft
>>>> state. Could we make it ready to review?
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Wed, May 28, 2025 at 10:34 AM Christian Thiel <
>>>> christian.t.b101@gmail.com> wrote:
>>>>
>>>>> Thank you all for the great discussion today!
>>>>> I have updated the proposal. Key changes are:
>>>>>
>>>>>    -
>>>>>
>>>>>    Specify that clients should ignore unknown operation & event types
>>>>>    -
>>>>>
>>>>>    Specify the `actor` field as part of the `Event` schema as an
>>>>>    opaque string. Remove the `Actor` type.
>>>>>    - Remove the `actors` filter from the `GetEventsRequest`, add an
>>>>>    extendable `custom-filters` object instead (`additionalProperties: true`)
>>>>>
>>>>> The diff for the changes since the sync today is available in Github:
>>>>> https://github.com/apache/iceberg/pull/12584/commits/3de9a7c5d128b1100c38ce688603c94491008d35
>>>>> The google doc is also updated,
>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>
>>>>> Looking forward to more feedback, especially regarding custom
>>>>> operations!
>>>>>
>>>>> On Tue, 27 May 2025 at 10:15, Christian Thiel <
>>>>> christian.t.b101@gmail.com> wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I think we have reached mostly consensus here.
>>>>>> There is one more change since our last discussion: We removed the
>>>>>> recursive "assumed-by" field of actors in favor or an "actor-chain" list.
>>>>>>
>>>>>> If there is any more need for discussion please voice it here on the
>>>>>> Mailing List or in the Catalog sync tomorrow. I would otherwise start a
>>>>>> vote.
>>>>>>
>>>>>> Best,
>>>>>> Christian
>>>>>>
>>>>>> On Wed, 7 May 2025 at 11:18, Christian Thiel <
>>>>>> christian.t.b101@gmail.com> wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I worked the changes discussed in the last catalog sync into the
>>>>>>> Events proposal [1].
>>>>>>> Those include:
>>>>>>> - Using request-id instead of transaction
>>>>>>> - A more flexible User (now called Actor)
>>>>>>> - Custom Operation type
>>>>>>>
>>>>>>> The specific diff compared to the last discussion can be best seen
>>>>>>> in my latest commit in git [2].
>>>>>>> It would be good to still comment in Google Docs so that we have
>>>>>>> everything in one place.
>>>>>>>
>>>>>>> Best,
>>>>>>> Christian
>>>>>>>
>>>>>>> [1]:
>>>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>>> [2]:
>>>>>>> https://github.com/apache/iceberg/pull/12584/commits/4d67051e03d5345687566b3900db3af23ce15766
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 16 Apr 2025 at 14:53, Christian Thiel <
>>>>>>> christian.t.b101@gmail.com> wrote:
>>>>>>>
>>>>>>>> Dear all,
>>>>>>>> after the last Catalog sync I updated the proposal.
>>>>>>>> Changes are in the original proposal Document [1] and the original
>>>>>>>> PR [2]
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Christian
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>>>> [2] https://github.com/apache/iceberg/pull/12584
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 20 Mar 2025 at 10:49, Christian Thiel <
>>>>>>>> christian.t.b101@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> We have recently discussed in the Iceberg Catalog Community Sync
>>>>>>>>> [1] and the Mailing List [2] different ways on how federation between
>>>>>>>>> Catalogs could be standardized.
>>>>>>>>>
>>>>>>>>> This proposal introduces a /events endpoint to the IRC
>>>>>>>>> specification. The endpoint provides events of modifications to objects
>>>>>>>>> managed by the Catalog (tables, namespaces, views), allowing consumers to
>>>>>>>>> efficiently track metadata changes using persistent offsets for reliable
>>>>>>>>> consumption and resumability.
>>>>>>>>>
>>>>>>>>> Proposal Document:
>>>>>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>>>>>
>>>>>>>>> I am looking forward to your thoughts and hope we find time in
>>>>>>>>> next week's Catalog sync to discuss this further.
>>>>>>>>>
>>>>>>>>> Best
>>>>>>>>> Christian
>>>>>>>>>
>>>>>>>>> [1] Catalog Community Sync Feb. 2025
>>>>>>>>> https://www.youtube.com/watch?v=hYcehreE8Nk
>>>>>>>>> [2] Mailing List, September 2024 - Notifications Endpoint:
>>>>>>>>> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht
>>>>>>>>>
>>>>>>>>

Re: [DISCUSS] Events Endpoint for IRC

Posted by Yufei Gu <fl...@gmail.com>.
Thanks, Christian and Ally, for working on the implementation. I understand
that the implementation is important, but do we need to block the REST spec
on it?
Typically, we merge the spec PR first and then work on the implementation.

Yufei


On Sun, Oct 19, 2025 at 11:55 PM Christian Thiel <ch...@gmail.com>
wrote:

> Thanks @ally for your work on this.
> Yes, we are currently waiting for Reviews. I am not a Java Maintainer, but
> maybe Eduard could have another look?
> @Yufei please have a look at the Java PR as well. Once this is merged, we
> can proceed with the REST PR.
>
> On Sat, 18 Oct 2025 at 09:02, ally heev <al...@gmail.com> wrote:
>
>> I think we are waiting on core classes implementation
>> <https://github.com/apache/iceberg/issues/13580>. I have a PR
>> <https://github.com/apache/iceberg/pull/14142> for the Request/Response
>> Objects waiting for review
>>
>>
>> On Sat, Oct 18, 2025 at 5:02 AM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks, do we have updates on this? Are we still pursuing it? I saw
>>> the PR https://github.com/apache/iceberg/pull/12584/files is in draft
>>> state. Could we make it ready to review?
>>>
>>> Yufei
>>>
>>>
>>> On Wed, May 28, 2025 at 10:34 AM Christian Thiel <
>>> christian.t.b101@gmail.com> wrote:
>>>
>>>> Thank you all for the great discussion today!
>>>> I have updated the proposal. Key changes are:
>>>>
>>>>    -
>>>>
>>>>    Specify that clients should ignore unknown operation & event types
>>>>    -
>>>>
>>>>    Specify the `actor` field as part of the `Event` schema as an
>>>>    opaque string. Remove the `Actor` type.
>>>>    - Remove the `actors` filter from the `GetEventsRequest`, add an
>>>>    extendable `custom-filters` object instead (`additionalProperties: true`)
>>>>
>>>> The diff for the changes since the sync today is available in Github:
>>>> https://github.com/apache/iceberg/pull/12584/commits/3de9a7c5d128b1100c38ce688603c94491008d35
>>>> The google doc is also updated,
>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>
>>>> Looking forward to more feedback, especially regarding custom
>>>> operations!
>>>>
>>>> On Tue, 27 May 2025 at 10:15, Christian Thiel <
>>>> christian.t.b101@gmail.com> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I think we have reached mostly consensus here.
>>>>> There is one more change since our last discussion: We removed the
>>>>> recursive "assumed-by" field of actors in favor or an "actor-chain" list.
>>>>>
>>>>> If there is any more need for discussion please voice it here on the
>>>>> Mailing List or in the Catalog sync tomorrow. I would otherwise start a
>>>>> vote.
>>>>>
>>>>> Best,
>>>>> Christian
>>>>>
>>>>> On Wed, 7 May 2025 at 11:18, Christian Thiel <
>>>>> christian.t.b101@gmail.com> wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I worked the changes discussed in the last catalog sync into the
>>>>>> Events proposal [1].
>>>>>> Those include:
>>>>>> - Using request-id instead of transaction
>>>>>> - A more flexible User (now called Actor)
>>>>>> - Custom Operation type
>>>>>>
>>>>>> The specific diff compared to the last discussion can be best seen in
>>>>>> my latest commit in git [2].
>>>>>> It would be good to still comment in Google Docs so that we have
>>>>>> everything in one place.
>>>>>>
>>>>>> Best,
>>>>>> Christian
>>>>>>
>>>>>> [1]:
>>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>> [2]:
>>>>>> https://github.com/apache/iceberg/pull/12584/commits/4d67051e03d5345687566b3900db3af23ce15766
>>>>>>
>>>>>>
>>>>>> On Wed, 16 Apr 2025 at 14:53, Christian Thiel <
>>>>>> christian.t.b101@gmail.com> wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>> after the last Catalog sync I updated the proposal.
>>>>>>> Changes are in the original proposal Document [1] and the original
>>>>>>> PR [2]
>>>>>>>
>>>>>>> Best,
>>>>>>> Christian
>>>>>>>
>>>>>>> [1]
>>>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>>> [2] https://github.com/apache/iceberg/pull/12584
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 20 Mar 2025 at 10:49, Christian Thiel <
>>>>>>> christian.t.b101@gmail.com> wrote:
>>>>>>>
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> We have recently discussed in the Iceberg Catalog Community Sync
>>>>>>>> [1] and the Mailing List [2] different ways on how federation between
>>>>>>>> Catalogs could be standardized.
>>>>>>>>
>>>>>>>> This proposal introduces a /events endpoint to the IRC
>>>>>>>> specification. The endpoint provides events of modifications to objects
>>>>>>>> managed by the Catalog (tables, namespaces, views), allowing consumers to
>>>>>>>> efficiently track metadata changes using persistent offsets for reliable
>>>>>>>> consumption and resumability.
>>>>>>>>
>>>>>>>> Proposal Document:
>>>>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>>>>
>>>>>>>> I am looking forward to your thoughts and hope we find time in next
>>>>>>>> week's Catalog sync to discuss this further.
>>>>>>>>
>>>>>>>> Best
>>>>>>>> Christian
>>>>>>>>
>>>>>>>> [1] Catalog Community Sync Feb. 2025
>>>>>>>> https://www.youtube.com/watch?v=hYcehreE8Nk
>>>>>>>> [2] Mailing List, September 2024 - Notifications Endpoint:
>>>>>>>> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht
>>>>>>>>
>>>>>>>

Re: [DISCUSS] Events Endpoint for IRC

Posted by Christian Thiel <ch...@gmail.com>.
Thanks @ally for your work on this.
Yes, we are currently waiting for Reviews. I am not a Java Maintainer, but
maybe Eduard could have another look?
@Yufei please have a look at the Java PR as well. Once this is merged, we
can proceed with the REST PR.

On Sat, 18 Oct 2025 at 09:02, ally heev <al...@gmail.com> wrote:

> I think we are waiting on core classes implementation
> <https://github.com/apache/iceberg/issues/13580>. I have a PR
> <https://github.com/apache/iceberg/pull/14142> for the Request/Response
> Objects waiting for review
>
>
> On Sat, Oct 18, 2025 at 5:02 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks, do we have updates on this? Are we still pursuing it? I saw the
>> PR https://github.com/apache/iceberg/pull/12584/files is in draft state.
>> Could we make it ready to review?
>>
>> Yufei
>>
>>
>> On Wed, May 28, 2025 at 10:34 AM Christian Thiel <
>> christian.t.b101@gmail.com> wrote:
>>
>>> Thank you all for the great discussion today!
>>> I have updated the proposal. Key changes are:
>>>
>>>    -
>>>
>>>    Specify that clients should ignore unknown operation & event types
>>>    -
>>>
>>>    Specify the `actor` field as part of the `Event` schema as an opaque
>>>    string. Remove the `Actor` type.
>>>    - Remove the `actors` filter from the `GetEventsRequest`, add an
>>>    extendable `custom-filters` object instead (`additionalProperties: true`)
>>>
>>> The diff for the changes since the sync today is available in Github:
>>> https://github.com/apache/iceberg/pull/12584/commits/3de9a7c5d128b1100c38ce688603c94491008d35
>>> The google doc is also updated,
>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>
>>> Looking forward to more feedback, especially regarding custom operations!
>>>
>>> On Tue, 27 May 2025 at 10:15, Christian Thiel <
>>> christian.t.b101@gmail.com> wrote:
>>>
>>>> Dear all,
>>>>
>>>> I think we have reached mostly consensus here.
>>>> There is one more change since our last discussion: We removed the
>>>> recursive "assumed-by" field of actors in favor or an "actor-chain" list.
>>>>
>>>> If there is any more need for discussion please voice it here on the
>>>> Mailing List or in the Catalog sync tomorrow. I would otherwise start a
>>>> vote.
>>>>
>>>> Best,
>>>> Christian
>>>>
>>>> On Wed, 7 May 2025 at 11:18, Christian Thiel <
>>>> christian.t.b101@gmail.com> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I worked the changes discussed in the last catalog sync into the
>>>>> Events proposal [1].
>>>>> Those include:
>>>>> - Using request-id instead of transaction
>>>>> - A more flexible User (now called Actor)
>>>>> - Custom Operation type
>>>>>
>>>>> The specific diff compared to the last discussion can be best seen in
>>>>> my latest commit in git [2].
>>>>> It would be good to still comment in Google Docs so that we have
>>>>> everything in one place.
>>>>>
>>>>> Best,
>>>>> Christian
>>>>>
>>>>> [1]:
>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>> [2]:
>>>>> https://github.com/apache/iceberg/pull/12584/commits/4d67051e03d5345687566b3900db3af23ce15766
>>>>>
>>>>>
>>>>> On Wed, 16 Apr 2025 at 14:53, Christian Thiel <
>>>>> christian.t.b101@gmail.com> wrote:
>>>>>
>>>>>> Dear all,
>>>>>> after the last Catalog sync I updated the proposal.
>>>>>> Changes are in the original proposal Document [1] and the original PR
>>>>>> [2]
>>>>>>
>>>>>> Best,
>>>>>> Christian
>>>>>>
>>>>>> [1]
>>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>> [2] https://github.com/apache/iceberg/pull/12584
>>>>>>
>>>>>>
>>>>>> On Thu, 20 Mar 2025 at 10:49, Christian Thiel <
>>>>>> christian.t.b101@gmail.com> wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> We have recently discussed in the Iceberg Catalog Community Sync [1]
>>>>>>> and the Mailing List [2] different ways on how federation between Catalogs
>>>>>>> could be standardized.
>>>>>>>
>>>>>>> This proposal introduces a /events endpoint to the IRC
>>>>>>> specification. The endpoint provides events of modifications to objects
>>>>>>> managed by the Catalog (tables, namespaces, views), allowing consumers to
>>>>>>> efficiently track metadata changes using persistent offsets for reliable
>>>>>>> consumption and resumability.
>>>>>>>
>>>>>>> Proposal Document:
>>>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>>>
>>>>>>> I am looking forward to your thoughts and hope we find time in next
>>>>>>> week's Catalog sync to discuss this further.
>>>>>>>
>>>>>>> Best
>>>>>>> Christian
>>>>>>>
>>>>>>> [1] Catalog Community Sync Feb. 2025
>>>>>>> https://www.youtube.com/watch?v=hYcehreE8Nk
>>>>>>> [2] Mailing List, September 2024 - Notifications Endpoint:
>>>>>>> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht
>>>>>>>
>>>>>>

Re: [DISCUSS] Events Endpoint for IRC

Posted by ally heev <al...@gmail.com>.
I think we are waiting on core classes implementation
<https://github.com/apache/iceberg/issues/13580>. I have a PR
<https://github.com/apache/iceberg/pull/14142> for the Request/Response
Objects waiting for review


On Sat, Oct 18, 2025 at 5:02 AM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks, do we have updates on this? Are we still pursuing it? I saw the
> PR https://github.com/apache/iceberg/pull/12584/files is in draft state.
> Could we make it ready to review?
>
> Yufei
>
>
> On Wed, May 28, 2025 at 10:34 AM Christian Thiel <
> christian.t.b101@gmail.com> wrote:
>
>> Thank you all for the great discussion today!
>> I have updated the proposal. Key changes are:
>>
>>    -
>>
>>    Specify that clients should ignore unknown operation & event types
>>    -
>>
>>    Specify the `actor` field as part of the `Event` schema as an opaque
>>    string. Remove the `Actor` type.
>>    - Remove the `actors` filter from the `GetEventsRequest`, add an
>>    extendable `custom-filters` object instead (`additionalProperties: true`)
>>
>> The diff for the changes since the sync today is available in Github:
>> https://github.com/apache/iceberg/pull/12584/commits/3de9a7c5d128b1100c38ce688603c94491008d35
>> The google doc is also updated,
>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>
>> Looking forward to more feedback, especially regarding custom operations!
>>
>> On Tue, 27 May 2025 at 10:15, Christian Thiel <ch...@gmail.com>
>> wrote:
>>
>>> Dear all,
>>>
>>> I think we have reached mostly consensus here.
>>> There is one more change since our last discussion: We removed the
>>> recursive "assumed-by" field of actors in favor or an "actor-chain" list.
>>>
>>> If there is any more need for discussion please voice it here on the
>>> Mailing List or in the Catalog sync tomorrow. I would otherwise start a
>>> vote.
>>>
>>> Best,
>>> Christian
>>>
>>> On Wed, 7 May 2025 at 11:18, Christian Thiel <ch...@gmail.com>
>>> wrote:
>>>
>>>> Dear all,
>>>>
>>>> I worked the changes discussed in the last catalog sync into the Events
>>>> proposal [1].
>>>> Those include:
>>>> - Using request-id instead of transaction
>>>> - A more flexible User (now called Actor)
>>>> - Custom Operation type
>>>>
>>>> The specific diff compared to the last discussion can be best seen in
>>>> my latest commit in git [2].
>>>> It would be good to still comment in Google Docs so that we have
>>>> everything in one place.
>>>>
>>>> Best,
>>>> Christian
>>>>
>>>> [1]:
>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>> [2]:
>>>> https://github.com/apache/iceberg/pull/12584/commits/4d67051e03d5345687566b3900db3af23ce15766
>>>>
>>>>
>>>> On Wed, 16 Apr 2025 at 14:53, Christian Thiel <
>>>> christian.t.b101@gmail.com> wrote:
>>>>
>>>>> Dear all,
>>>>> after the last Catalog sync I updated the proposal.
>>>>> Changes are in the original proposal Document [1] and the original PR
>>>>> [2]
>>>>>
>>>>> Best,
>>>>> Christian
>>>>>
>>>>> [1]
>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>> [2] https://github.com/apache/iceberg/pull/12584
>>>>>
>>>>>
>>>>> On Thu, 20 Mar 2025 at 10:49, Christian Thiel <
>>>>> christian.t.b101@gmail.com> wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> We have recently discussed in the Iceberg Catalog Community Sync [1]
>>>>>> and the Mailing List [2] different ways on how federation between Catalogs
>>>>>> could be standardized.
>>>>>>
>>>>>> This proposal introduces a /events endpoint to the IRC
>>>>>> specification. The endpoint provides events of modifications to objects
>>>>>> managed by the Catalog (tables, namespaces, views), allowing consumers to
>>>>>> efficiently track metadata changes using persistent offsets for reliable
>>>>>> consumption and resumability.
>>>>>>
>>>>>> Proposal Document:
>>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>>
>>>>>> I am looking forward to your thoughts and hope we find time in next
>>>>>> week's Catalog sync to discuss this further.
>>>>>>
>>>>>> Best
>>>>>> Christian
>>>>>>
>>>>>> [1] Catalog Community Sync Feb. 2025
>>>>>> https://www.youtube.com/watch?v=hYcehreE8Nk
>>>>>> [2] Mailing List, September 2024 - Notifications Endpoint:
>>>>>> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht
>>>>>>
>>>>>

Re: [DISCUSS] Events Endpoint for IRC

Posted by Yufei Gu <fl...@gmail.com>.
Hi folks, do we have updates on this? Are we still pursuing it? I saw the
PR https://github.com/apache/iceberg/pull/12584/files is in draft state.
Could we make it ready to review?

Yufei


On Wed, May 28, 2025 at 10:34 AM Christian Thiel <ch...@gmail.com>
wrote:

> Thank you all for the great discussion today!
> I have updated the proposal. Key changes are:
>
>    -
>
>    Specify that clients should ignore unknown operation & event types
>    -
>
>    Specify the `actor` field as part of the `Event` schema as an opaque
>    string. Remove the `Actor` type.
>    - Remove the `actors` filter from the `GetEventsRequest`, add an
>    extendable `custom-filters` object instead (`additionalProperties: true`)
>
> The diff for the changes since the sync today is available in Github:
> https://github.com/apache/iceberg/pull/12584/commits/3de9a7c5d128b1100c38ce688603c94491008d35
> The google doc is also updated,
> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>
> Looking forward to more feedback, especially regarding custom operations!
>
> On Tue, 27 May 2025 at 10:15, Christian Thiel <ch...@gmail.com>
> wrote:
>
>> Dear all,
>>
>> I think we have reached mostly consensus here.
>> There is one more change since our last discussion: We removed the
>> recursive "assumed-by" field of actors in favor or an "actor-chain" list.
>>
>> If there is any more need for discussion please voice it here on the
>> Mailing List or in the Catalog sync tomorrow. I would otherwise start a
>> vote.
>>
>> Best,
>> Christian
>>
>> On Wed, 7 May 2025 at 11:18, Christian Thiel <ch...@gmail.com>
>> wrote:
>>
>>> Dear all,
>>>
>>> I worked the changes discussed in the last catalog sync into the Events
>>> proposal [1].
>>> Those include:
>>> - Using request-id instead of transaction
>>> - A more flexible User (now called Actor)
>>> - Custom Operation type
>>>
>>> The specific diff compared to the last discussion can be best seen in my
>>> latest commit in git [2].
>>> It would be good to still comment in Google Docs so that we have
>>> everything in one place.
>>>
>>> Best,
>>> Christian
>>>
>>> [1]:
>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>> [2]:
>>> https://github.com/apache/iceberg/pull/12584/commits/4d67051e03d5345687566b3900db3af23ce15766
>>>
>>>
>>> On Wed, 16 Apr 2025 at 14:53, Christian Thiel <
>>> christian.t.b101@gmail.com> wrote:
>>>
>>>> Dear all,
>>>> after the last Catalog sync I updated the proposal.
>>>> Changes are in the original proposal Document [1] and the original PR
>>>> [2]
>>>>
>>>> Best,
>>>> Christian
>>>>
>>>> [1]
>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>> [2] https://github.com/apache/iceberg/pull/12584
>>>>
>>>>
>>>> On Thu, 20 Mar 2025 at 10:49, Christian Thiel <
>>>> christian.t.b101@gmail.com> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> We have recently discussed in the Iceberg Catalog Community Sync [1]
>>>>> and the Mailing List [2] different ways on how federation between Catalogs
>>>>> could be standardized.
>>>>>
>>>>> This proposal introduces a /events endpoint to the IRC specification.
>>>>> The endpoint provides events of modifications to objects managed by the
>>>>> Catalog (tables, namespaces, views), allowing consumers to efficiently
>>>>> track metadata changes using persistent offsets for reliable consumption
>>>>> and resumability.
>>>>>
>>>>> Proposal Document:
>>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>>
>>>>> I am looking forward to your thoughts and hope we find time in next
>>>>> week's Catalog sync to discuss this further.
>>>>>
>>>>> Best
>>>>> Christian
>>>>>
>>>>> [1] Catalog Community Sync Feb. 2025
>>>>> https://www.youtube.com/watch?v=hYcehreE8Nk
>>>>> [2] Mailing List, September 2024 - Notifications Endpoint:
>>>>> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht
>>>>>
>>>>

Re: [DISCUSS] Events Endpoint for IRC

Posted by Christian Thiel <ch...@gmail.com>.
Thank you all for the great discussion today!
I have updated the proposal. Key changes are:

   -

   Specify that clients should ignore unknown operation & event types
   -

   Specify the `actor` field as part of the `Event` schema as an opaque
   string. Remove the `Actor` type.
   - Remove the `actors` filter from the `GetEventsRequest`, add an
   extendable `custom-filters` object instead (`additionalProperties: true`)

The diff for the changes since the sync today is available in Github:
https://github.com/apache/iceberg/pull/12584/commits/3de9a7c5d128b1100c38ce688603c94491008d35
The google doc is also updated,
https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing

Looking forward to more feedback, especially regarding custom operations!

On Tue, 27 May 2025 at 10:15, Christian Thiel <ch...@gmail.com>
wrote:

> Dear all,
>
> I think we have reached mostly consensus here.
> There is one more change since our last discussion: We removed the
> recursive "assumed-by" field of actors in favor or an "actor-chain" list.
>
> If there is any more need for discussion please voice it here on the
> Mailing List or in the Catalog sync tomorrow. I would otherwise start a
> vote.
>
> Best,
> Christian
>
> On Wed, 7 May 2025 at 11:18, Christian Thiel <ch...@gmail.com>
> wrote:
>
>> Dear all,
>>
>> I worked the changes discussed in the last catalog sync into the Events
>> proposal [1].
>> Those include:
>> - Using request-id instead of transaction
>> - A more flexible User (now called Actor)
>> - Custom Operation type
>>
>> The specific diff compared to the last discussion can be best seen in my
>> latest commit in git [2].
>> It would be good to still comment in Google Docs so that we have
>> everything in one place.
>>
>> Best,
>> Christian
>>
>> [1]:
>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>> [2]:
>> https://github.com/apache/iceberg/pull/12584/commits/4d67051e03d5345687566b3900db3af23ce15766
>>
>>
>> On Wed, 16 Apr 2025 at 14:53, Christian Thiel <ch...@gmail.com>
>> wrote:
>>
>>> Dear all,
>>> after the last Catalog sync I updated the proposal.
>>> Changes are in the original proposal Document [1] and the original PR [2]
>>>
>>> Best,
>>> Christian
>>>
>>> [1]
>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>> [2] https://github.com/apache/iceberg/pull/12584
>>>
>>>
>>> On Thu, 20 Mar 2025 at 10:49, Christian Thiel <
>>> christian.t.b101@gmail.com> wrote:
>>>
>>>> Dear all,
>>>>
>>>> We have recently discussed in the Iceberg Catalog Community Sync [1]
>>>> and the Mailing List [2] different ways on how federation between Catalogs
>>>> could be standardized.
>>>>
>>>> This proposal introduces a /events endpoint to the IRC specification.
>>>> The endpoint provides events of modifications to objects managed by the
>>>> Catalog (tables, namespaces, views), allowing consumers to efficiently
>>>> track metadata changes using persistent offsets for reliable consumption
>>>> and resumability.
>>>>
>>>> Proposal Document:
>>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>>
>>>> I am looking forward to your thoughts and hope we find time in next
>>>> week's Catalog sync to discuss this further.
>>>>
>>>> Best
>>>> Christian
>>>>
>>>> [1] Catalog Community Sync Feb. 2025
>>>> https://www.youtube.com/watch?v=hYcehreE8Nk
>>>> [2] Mailing List, September 2024 - Notifications Endpoint:
>>>> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht
>>>>
>>>

Re: [DISCUSS] Events Endpoint for IRC

Posted by Christian Thiel <ch...@gmail.com>.
Dear all,

I think we have reached mostly consensus here.
There is one more change since our last discussion: We removed the
recursive "assumed-by" field of actors in favor or an "actor-chain" list.

If there is any more need for discussion please voice it here on the
Mailing List or in the Catalog sync tomorrow. I would otherwise start a
vote.

Best,
Christian

On Wed, 7 May 2025 at 11:18, Christian Thiel <ch...@gmail.com>
wrote:

> Dear all,
>
> I worked the changes discussed in the last catalog sync into the Events
> proposal [1].
> Those include:
> - Using request-id instead of transaction
> - A more flexible User (now called Actor)
> - Custom Operation type
>
> The specific diff compared to the last discussion can be best seen in my
> latest commit in git [2].
> It would be good to still comment in Google Docs so that we have
> everything in one place.
>
> Best,
> Christian
>
> [1]:
> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
> [2]:
> https://github.com/apache/iceberg/pull/12584/commits/4d67051e03d5345687566b3900db3af23ce15766
>
>
> On Wed, 16 Apr 2025 at 14:53, Christian Thiel <ch...@gmail.com>
> wrote:
>
>> Dear all,
>> after the last Catalog sync I updated the proposal.
>> Changes are in the original proposal Document [1] and the original PR [2]
>>
>> Best,
>> Christian
>>
>> [1]
>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>> [2] https://github.com/apache/iceberg/pull/12584
>>
>>
>> On Thu, 20 Mar 2025 at 10:49, Christian Thiel <ch...@gmail.com>
>> wrote:
>>
>>> Dear all,
>>>
>>> We have recently discussed in the Iceberg Catalog Community Sync [1] and
>>> the Mailing List [2] different ways on how federation between Catalogs
>>> could be standardized.
>>>
>>> This proposal introduces a /events endpoint to the IRC specification.
>>> The endpoint provides events of modifications to objects managed by the
>>> Catalog (tables, namespaces, views), allowing consumers to efficiently
>>> track metadata changes using persistent offsets for reliable consumption
>>> and resumability.
>>>
>>> Proposal Document:
>>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>>
>>> I am looking forward to your thoughts and hope we find time in next
>>> week's Catalog sync to discuss this further.
>>>
>>> Best
>>> Christian
>>>
>>> [1] Catalog Community Sync Feb. 2025
>>> https://www.youtube.com/watch?v=hYcehreE8Nk
>>> [2] Mailing List, September 2024 - Notifications Endpoint:
>>> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht
>>>
>>

Re: [DISCUSS] Events Endpoint for IRC

Posted by Christian Thiel <ch...@gmail.com>.
Dear all,

I worked the changes discussed in the last catalog sync into the Events
proposal [1].
Those include:
- Using request-id instead of transaction
- A more flexible User (now called Actor)
- Custom Operation type

The specific diff compared to the last discussion can be best seen in my
latest commit in git [2].
It would be good to still comment in Google Docs so that we have
everything in one place.

Best,
Christian

[1]:
https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
[2]:
https://github.com/apache/iceberg/pull/12584/commits/4d67051e03d5345687566b3900db3af23ce15766


On Wed, 16 Apr 2025 at 14:53, Christian Thiel <ch...@gmail.com>
wrote:

> Dear all,
> after the last Catalog sync I updated the proposal.
> Changes are in the original proposal Document [1] and the original PR [2]
>
> Best,
> Christian
>
> [1]
> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
> [2] https://github.com/apache/iceberg/pull/12584
>
>
> On Thu, 20 Mar 2025 at 10:49, Christian Thiel <ch...@gmail.com>
> wrote:
>
>> Dear all,
>>
>> We have recently discussed in the Iceberg Catalog Community Sync [1] and
>> the Mailing List [2] different ways on how federation between Catalogs
>> could be standardized.
>>
>> This proposal introduces a /events endpoint to the IRC specification.
>> The endpoint provides events of modifications to objects managed by the
>> Catalog (tables, namespaces, views), allowing consumers to efficiently
>> track metadata changes using persistent offsets for reliable consumption
>> and resumability.
>>
>> Proposal Document:
>> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing
>>
>> I am looking forward to your thoughts and hope we find time in next
>> week's Catalog sync to discuss this further.
>>
>> Best
>> Christian
>>
>> [1] Catalog Community Sync Feb. 2025
>> https://www.youtube.com/watch?v=hYcehreE8Nk
>> [2] Mailing List, September 2024 - Notifications Endpoint:
>> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht
>>
>