You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by vignesh a <im...@gmail.com> on 2026/10/04 09:45:45 UTC
[DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
\Hi all,
Robert and EJ have both asked that we settle a shared event envelope
before merging the webhook listener [1]. I agree with that sequencing
and would like a quick list consensus on shape and placement before I
open the prerequisite PR.
What each path emits today (from current main / [1]):
- Kafka (extensions/events/kafka/.../EventToJsonMapper):
flat snake_case: event_type, event_id, timestamp, realm_id, principal,
activated_roles, request_id, plus a fairly rich attribute set
(table_identifier as a string list, namespace/catalog/view fields,
rename source/destination, and similar).
- Webhook ([1], WebhookEventListener.toJson):
CloudEvents-shaped: specversion, id, type, source, time, plus
extensions deliverytime, realmid, tableidentifier (single string),
principal, activatedroles, requestid. Thinner attribute coverage than
Kafka.
- CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
flat map: event_type, optional table_identifier (string), realm_id,
principal, activated_roles, request_id. No event_id in the JSON body;
timestamp is on InputLogEvent.
So this is not a mechanical "extract EventToJsonMapper." Naming,
structure, identifier encoding, and attribute breadth already diverge,
and the shared code should not live inside the kafka extension.
Proposal:
1) Land a small prerequisite that introduces a shared, versioned,
CloudEvents-shaped mapper (core fields + a documented extension set),
placed outside any one transport extension.
2) Rebase [1] onto that shared mapper.
3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
compatibility plan so existing consumers are not silently broken
(version field and/or dual-publish window; exact mechanism in the
prerequisite PR description).
4) Attribute breadth for v1: start from the webhook/CloudWatch-thin
common set; add Kafka's richer attributes behind the same versioned
envelope rather than blocking v1 on full Kafka parity.
If that direction is wrong, the useful alternatives seem to be:
- B) Prefer Kafka's flat snake_case as the shared shape instead of
CloudEvents-shaped.
- C) Fold the shared mapper into [1] itself (larger single PR).
Unless someone prefers B or C, I will open the prerequisite PR along the
lines of (1)-(4) early next week.
Thanks,
Vignesh
[1] https://github.com/apache/polaris/pull/5084
Re: [DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
Posted by vignesh a <im...@gmail.com>.
Hi all,
The mapper [1] is on main.
I've rebased the webhook listener [2] onto it. The POST body is the
shared envelope, and the delivery time goes in the
X-Polaris-Delivery-Time header. [2] is the first consumer of the
mapper and the next PR to land.
Quick recap on Kafka and CloudWatch from this thread:
- Kafka already ships a flat payload in 1.7 / 1.8, so that listener
stays on the current mapper for now. The follow-up is a
per-listener format flag: flat payload stays the default and the
shared envelope is opt-in, so existing consumers keep working.
- CloudWatch moves to the same envelope after the webhook listener.
Thanks,
*Vignesh*
[1] https://github.com/apache/polaris/pull/5717
[2] https://github.com/apache/polaris/pull/5084
Re: [DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
Posted by vignesh a <im...@gmail.com>.
Hi all,
Prerequisite PR is up [1].
Shared CloudEvents mapper next to the events SPI in runtime/service.
CE context at the top, Kafka-parity fields in data, and lowercased
type (org.apache.polaris.after_create_table) as discussed.
Webhook rebase on [2] can follow after this lands.
Thanks
*Vignesh*
[1] https://github.com/apache/polaris/pull/5717
[2] https://github.com/apache/polaris/pull/5084
On Wed, 7 Oct 2026 at 02:24, vignesh a <im...@gmail.com> wrote:
> Hi Dmitri,
>
> Thanks - Noted. +1 on lowercasing type..
>
> I'll open the prerequisite PR next (mapper next to the SPI in
> runtime/service).
>
> Thanks
> *Vignesh*
>
> On Tue, 6 Oct 2026 at 21:32, Dmitri Bourlatchkov <di...@apache.org> wrote:
>
>> Hi Vignesh,
>>
>> The proposal LGTM.
>>
>> Minor comment: SInce CloudEvents apparently prefer lowercase names, shall
>> we lowercase the type namne (in JSON)?
>>
>> Re: backward compatibility for Kafka. Having a flag to send old payloads
>> works for me. However, I believe this tends to prolong adoption and users
>> have to face a hard change anyway (when their admin flips the switch).
>> This
>> may not be suitable for rolling upgrade scenarios. That said, we can
>> probably debate this later, after establishing the common format.
>>
>> Cheers,
>> Dmitri.
>>
>> On Tue, Oct 6, 2026 at 8:11 AM vignesh a <im...@gmail.com> wrote:
>>
>> > Hi Dmitri,
>> >
>> > Thanks - good questions, and yes at a high level that's the direction.
>> > Wanted to answer them against what JB and I already talked through, and
>> > I've added the draft v1 JSON again so we're looking at the same thing.
>> >
>> > 1) Same JSON for all JSON-producing listeners?
>> >
>> > For the transport listeners in this thread (Kafka, webhook, CloudWatch),
>> > yes - same CloudEvents-shaped payload is the end goal. OpenTelemetry and
>> > the persistence listener keep their own representations for now; those
>> > stay out of scope for the first PR.
>> >
>> > 2) Adhere to CloudEvents?
>> >
>> > Yes. v1 means CloudEvents 1.0 context attributes at the top only, and
>> the
>> > Polaris fields in "data". That also keeps us honest with CE rules on
>> > context attrs (lowercase alphanumerics / scalars) - so things like
>> > activated_roles live in "data", not as top-level CE extensions. If a
>> > newer CE version becomes relevant later we can bump then..
>> >
>> > 3) Migrating existing consumers that aren't 1:1 with the new shape?
>> >
>> > Union / dual-publish for a few releases is one way to do it. On the list
>> > JB leaned toward a per-listener format option instead (today's flat
>> Kafka
>> > payload as the default, shared envelope opt-in), mainly so we don't
>> > double volume on the topic. That felt right to me too for 1.7 / 1.8
>> > consumers - they keep working until they flip the knob. Webhook [1] can
>> > land on the shared envelope from day one since it isn't released that
>> > way yet. Open to other ideas if you see a cleaner path.. LMK
>> >
>> > 4) Dedicated SPI module?
>> >
>> > Also +1 (Alex mentioned the same). Kafka depending on runtime/service
>> > just for the SPI is tricky, and having the mapper live in that SPI
>> > module later makes a lot of sense. For the first PR I was thinking of
>> > putting the shared mapper next to the existing events SPI in
>> > runtime/service, then extracting SPI + mapper as a follow-up - mostly to
>> > avoid getting stuck on module cycles before the envelope lands (JB had
>> > flagged that). Will do the SPI split earlier if that feels better. WDYT
>> ??
>> >
>> > Draft v1 JSON (CE context + today's Kafka body in "data"):
>> >
>> > {
>> > "specversion": "1.0",
>> > "id": "550e8400-e29b-41d4-a716-446655440000",
>> > "type": "org.apache.polaris.AFTER_CREATE_TABLE",
>> > "source": "org.apache.polaris",
>> > "time": "2026-10-04T12:34:56.789Z",
>> > "datacontenttype": "application/json",
>> > "data": {
>> > "event_type": "AFTER_CREATE_TABLE",
>> > "event_id": "550e8400-e29b-41d4-a716-446655440000",
>> > "timestamp": "2026-10-04T12:34:56.789Z",
>> > "realm_id": "default-realm",
>> > "principal": "alice",
>> > "activated_roles": ["catalog_admin"],
>> > "request_id": "req-123",
>> > "catalog_name": "sales",
>> > "namespace": ["analytics"],
>> > "table_name": "orders",
>> > "table_identifier": ["analytics", "orders"]
>> > }
>> > }
>> >
>> > A couple notes:
>> > - "data" is basically today's Kafka EventToJsonMapper body, so id/time
>> > show up both as CE context and again inside data (event_id /
>> > timestamp) for Kafka parity.
>> > - Type is a namespaced PolarisEventType name; source is a constant for
>> > now - happy to bike-shed those if needed.
>> > - Webhook delivery time stays transport metadata (HTTP header), not in
>> > the shared envelope.
>> > - Rename events would also carry source / destination arrays in "data",
>> > same as Kafka today; omitted above because this example is
>> > AFTER_CREATE_TABLE.
>> >
>> > If this looks right, I'll open the prerequisite PR on that basis...
>> >
>> > Thanks
>> > *Vignesh*
>> >
>> > [1] https://github.com/apache/polaris/pull/5084
>> >
>> > On Tue, 6 Oct 2026 at 05:53, Dmitri Bourlatchkov <di...@apache.org>
>> wrote:
>> >
>> > > Hi Vgnesh,
>> > >
>> > > Your proposal sounds good to be at high level.
>> > >
>> > > Just to clarify: is our end goal here to use the same JSON payload for
>> > all
>> > > event listeners that produce JSON?
>> > >
>> > > Are we going to adhere to the CloudEvents spec for this JSON?
>> Obviously
>> > > starting with 1.0 and migrating to newer version as the become
>> > available...
>> > > Is that the plan?
>> > >
>> > > How are we going to migrare existing use cases that are not 1:1 with
>> > > the ideal JSON schema? Will be publish a union of attributes for a few
>> > > releases to allow consumers to switch to the new schema?
>> > >
>> > > +1 to adding a dedicated SPI module. I suppose it can also contain the
>> > > common java-to-JSON mapping code.
>> > >
>> > > Thanks,
>> > > Dmitri.
>> > >
>> > > On Mon, Oct 5, 2026 at 4:29 AM Alex Dutra <ad...@apache.org> wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1.
>> > > >
>> > > > I also welcome the idea of isolating the Events SPI in its own
>> module.
>> > > > Having the Kafka module depend on runtime/service just for the SPI
>> is
>> > > > concerning.
>> > > >
>> > > > Looking forward to the first PR!
>> > > >
>> > > > Thanks,
>> > > > Alex
>> > > >
>> > > > On Mon, Oct 5, 2026 at 10:02 AM vignesh a <im...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > Hi JB,
>> > > > >
>> > > > > Thanks - this is really useful, and I'm with you on the sequencing
>> > > > > (shared envelope first, then rebase the webhook).
>> > > > >
>> > > > > Your points on shape make sense to me too:
>> > > > > 1. CloudEvents context attrs at the top only, Polaris payload in
>> > "data"
>> > > > > (so things like activated_roles stay valid)
>> > > > > 2. reuse the existing EventAttributes / Kafka snake_case keys in
>> > "data"
>> > > > > 3. v1 at Kafka parity, not the thinner webhook set I suggested
>> > earlier
>> > > > > 4. mapper next to the events SPI in runtime/service for now
>> > > > > 5. deliverytime as webhook HTTP metadata, not in the shared
>> envelope
>> > > > > 6. OpenTelemetry + persistence listeners out of scope for this
>> > > > >
>> > > > > Also +1 on settling Kafka compat on the list. per-listener format
>> > > option
>> > > > > with today's flat payload as the default (shared envelope opt-in)
>> > > sounds
>> > > > > better than dual-publish.
>> > > > >
>> > > > > Draft v1 JSON I have in mind:
>> > > > >
>> > > > > {
>> > > > > "specversion": "1.0",
>> > > > > "id": "550e8400-e29b-41d4-a716-446655440000",
>> > > > > "type": "org.apache.polaris.AFTER_CREATE_TABLE",
>> > > > > "source": "org.apache.polaris",
>> > > > > "time": "2026-10-04T12:34:56.789Z",
>> > > > > "datacontenttype": "application/json",
>> > > > > "data": {
>> > > > > "event_type": "AFTER_CREATE_TABLE",
>> > > > > "event_id": "550e8400-e29b-41d4-a716-446655440000",
>> > > > > "timestamp": "2026-10-04T12:34:56.789Z",
>> > > > > "realm_id": "default-realm",
>> > > > > "principal": "alice",
>> > > > > "activated_roles": ["catalog_admin"],
>> > > > > "request_id": "req-123",
>> > > > > "catalog_name": "sales",
>> > > > > "namespace": ["analytics"],
>> > > > > "table_name": "orders",
>> > > > > "table_identifier": ["analytics", "orders"]
>> > > > > }
>> > > > > }
>> > > > >
>> > > > > A couple notes on purpose:
>> > > > > - "data" is basically today's Kafka body, so id/time show up both
>> as
>> > CE
>> > > > > context and again inside data (event_id / timestamp) for Kafka
>> > parity
>> > > > > - type is a namespaced PolarisEventType name; source is a constant
>> > for
>> > > > > now - will change if needed..
>> > > > > - webhook would put delivery time on an HTTP header
>> > > > > (e.g. X-Polaris-Delivery-Time), not in the envelope
>> > > > >
>> > > > > if this is roughly the right shape, I'll draft the prerequisite PR
>> > > around
>> > > > > it. if you'd rather nail type/source naming first, tell me which
>> > forms
>> > > > > you prefer.
>> > > > >
>> > > > > Thanks,
>> > > > > *Vignesh*
>> > > > >
>> > > > > [1] https://github.com/apache/polaris/pull/5084
>> > > > >
>> > > > > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <
>> jb@nanthrax.net>
>> > > > wrote:
>> > > > >
>> > > > > > Hi Vignesh
>> > > > > >
>> > > > > > Thanks for starting this thread.
>> > > > > >
>> > > > > > I'm +1 on the sequence: shared envelope first and then rebase
>> the
>> > > > webhook
>> > > > > > PR. I slightly prefer your proposal over B or C.
>> > > > > >
>> > > > > > Some general comments:
>> > > > > > 1. CloudEvents requires attribute names to be lowercase
>> > alphanumerics
>> > > > and
>> > > > > > attribute values to be scalars afair. So a flat set of
>> extensions
>> > > > doesn't
>> > > > > > really work for us: activatedroles as an array is not valid for
>> > > > > > instance. Or we have to transform somehow. I would keep only the
>> > > > > > CloudEvents context attributes at the top level and put the
>> Polaris
>> > > > fields
>> > > > > > in "data".
>> > > > > > 2. The EventAttributes keys already have underscored names
>> > > > (catalog_name,
>> > > > > > table_identifier, ...) and they match what Kafka emits. I would
>> > reuse
>> > > > them
>> > > > > > as is. Then the current Kafka body basically becomes "data",
>> which
>> > > also
>> > > > > > makes the migration easier.
>> > > > > > 3. I would not start from the thin set: migrating Kafka to it
>> would
>> > > > drop
>> > > > > > fields that consumers already get (namespace, catalog, view,
>> ..).
>> > > Kafka
>> > > > > > only has nine optional fields, so I think v1 should start at
>> Kafka
>> > > > parity.
>> > > > > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I
>> > > think
>> > > > we
>> > > > > > should agree on the mechanism here on the list rather than in
>> the
>> > PR
>> > > > > > description (if it doesn't happen on the mailing list, it never
>> > > happens
>> > > > > > :)). I would prefer a per-listener format option, with the
>> current
>> > > > format
>> > > > > > as default, over dual-publish (which doubles the volume on the
>> > > topic).
>> > > > > > 5. The events SPI and the CloudWatch listener are both in
>> > > > runtime/service,
>> > > > > > and the Kafka extension depends on it. A new module would
>> > introduce a
>> > > > cycle
>> > > > > > unless we extract the SPI first, so I would put the mapper next
>> to
>> > > the
>> > > > SPI
>> > > > > > in runtime/service for now, and move it when the SPI is moved.
>> > > > > > 6. deliverytime is a transport metadata, I would move it to a
>> HTTP
>> > > > header
>> > > > > > instead of the shared envelope. We should also decide what
>> "type"
>> > and
>> > > > > > "source" look like (today it's a bare enum name and a constant).
>> > > > > >
>> > > > > > Could you share the v1 JSON you have in mind on this thread
>> before
>> > > > opening
>> > > > > > the PR? I think it will be easier to converge on an example.
>> > > > > >
>> > > > > > The OpenTelemetry and persistence listeners have their own
>> > > > representations
>> > > > > > as well. I'm fine keeping them out of scope, but it's worth
>> stating
>> > > it
>> > > > > > explicitly.
>> > > > > >
>> > > > > > Regards
>> > > > > > JB
>> > > > > >
>> > > > > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a <
>> imavignesh27@gmail.com>
>> > > > wrote:
>> > > > > >
>> > > > > > > \Hi all,
>> > > > > > >
>> > > > > > > Robert and EJ have both asked that we settle a shared event
>> > > envelope
>> > > > > > > before merging the webhook listener [1]. I agree with that
>> > > sequencing
>> > > > > > > and would like a quick list consensus on shape and placement
>> > > before I
>> > > > > > > open the prerequisite PR.
>> > > > > > >
>> > > > > > > What each path emits today (from current main / [1]):
>> > > > > > >
>> > > > > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper):
>> > > > > > > flat snake_case: event_type, event_id, timestamp, realm_id,
>> > > > principal,
>> > > > > > > activated_roles, request_id, plus a fairly rich attribute
>> set
>> > > > > > > (table_identifier as a string list, namespace/catalog/view
>> > > fields,
>> > > > > > > rename source/destination, and similar).
>> > > > > > >
>> > > > > > > - Webhook ([1], WebhookEventListener.toJson):
>> > > > > > > CloudEvents-shaped: specversion, id, type, source, time,
>> plus
>> > > > > > > extensions deliverytime, realmid, tableidentifier (single
>> > > string),
>> > > > > > > principal, activatedroles, requestid. Thinner attribute
>> > coverage
>> > > > than
>> > > > > > > Kafka.
>> > > > > > >
>> > > > > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
>> > > > > > > flat map: event_type, optional table_identifier (string),
>> > > realm_id,
>> > > > > > > principal, activated_roles, request_id. No event_id in the
>> JSON
>> > > > body;
>> > > > > > > timestamp is on InputLogEvent.
>> > > > > > >
>> > > > > > > So this is not a mechanical "extract EventToJsonMapper."
>> Naming,
>> > > > > > > structure, identifier encoding, and attribute breadth already
>> > > > diverge,
>> > > > > > > and the shared code should not live inside the kafka
>> extension.
>> > > > > > >
>> > > > > > > Proposal:
>> > > > > > >
>> > > > > > > 1) Land a small prerequisite that introduces a shared,
>> versioned,
>> > > > > > > CloudEvents-shaped mapper (core fields + a documented
>> > extension
>> > > > set),
>> > > > > > > placed outside any one transport extension.
>> > > > > > >
>> > > > > > > 2) Rebase [1] onto that shared mapper.
>> > > > > > >
>> > > > > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an
>> explicit
>> > > > > > > compatibility plan so existing consumers are not silently
>> > broken
>> > > > > > > (version field and/or dual-publish window; exact mechanism
>> in
>> > > the
>> > > > > > > prerequisite PR description).
>> > > > > > >
>> > > > > > > 4) Attribute breadth for v1: start from the
>> > webhook/CloudWatch-thin
>> > > > > > > common set; add Kafka's richer attributes behind the same
>> > > > versioned
>> > > > > > > envelope rather than blocking v1 on full Kafka parity.
>> > > > > > >
>> > > > > > > If that direction is wrong, the useful alternatives seem to
>> be:
>> > > > > > >
>> > > > > > > - B) Prefer Kafka's flat snake_case as the shared shape
>> instead
>> > of
>> > > > > > > CloudEvents-shaped.
>> > > > > > > - C) Fold the shared mapper into [1] itself (larger single
>> PR).
>> > > > > > >
>> > > > > > > Unless someone prefers B or C, I will open the prerequisite PR
>> > > along
>> > > > the
>> > > > > > > lines of (1)-(4) early next week.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Vignesh
>> > > > > > >
>> > > > > > > [1] https://github.com/apache/polaris/pull/5084
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
Posted by vignesh a <im...@gmail.com>.
Hi Dmitri,
Thanks - Noted. +1 on lowercasing type..
I'll open the prerequisite PR next (mapper next to the SPI in
runtime/service).
Thanks
*Vignesh*
On Tue, 6 Oct 2026 at 21:32, Dmitri Bourlatchkov <di...@apache.org> wrote:
> Hi Vignesh,
>
> The proposal LGTM.
>
> Minor comment: SInce CloudEvents apparently prefer lowercase names, shall
> we lowercase the type namne (in JSON)?
>
> Re: backward compatibility for Kafka. Having a flag to send old payloads
> works for me. However, I believe this tends to prolong adoption and users
> have to face a hard change anyway (when their admin flips the switch). This
> may not be suitable for rolling upgrade scenarios. That said, we can
> probably debate this later, after establishing the common format.
>
> Cheers,
> Dmitri.
>
> On Tue, Oct 6, 2026 at 8:11 AM vignesh a <im...@gmail.com> wrote:
>
> > Hi Dmitri,
> >
> > Thanks - good questions, and yes at a high level that's the direction.
> > Wanted to answer them against what JB and I already talked through, and
> > I've added the draft v1 JSON again so we're looking at the same thing.
> >
> > 1) Same JSON for all JSON-producing listeners?
> >
> > For the transport listeners in this thread (Kafka, webhook, CloudWatch),
> > yes - same CloudEvents-shaped payload is the end goal. OpenTelemetry and
> > the persistence listener keep their own representations for now; those
> > stay out of scope for the first PR.
> >
> > 2) Adhere to CloudEvents?
> >
> > Yes. v1 means CloudEvents 1.0 context attributes at the top only, and the
> > Polaris fields in "data". That also keeps us honest with CE rules on
> > context attrs (lowercase alphanumerics / scalars) - so things like
> > activated_roles live in "data", not as top-level CE extensions. If a
> > newer CE version becomes relevant later we can bump then..
> >
> > 3) Migrating existing consumers that aren't 1:1 with the new shape?
> >
> > Union / dual-publish for a few releases is one way to do it. On the list
> > JB leaned toward a per-listener format option instead (today's flat Kafka
> > payload as the default, shared envelope opt-in), mainly so we don't
> > double volume on the topic. That felt right to me too for 1.7 / 1.8
> > consumers - they keep working until they flip the knob. Webhook [1] can
> > land on the shared envelope from day one since it isn't released that
> > way yet. Open to other ideas if you see a cleaner path.. LMK
> >
> > 4) Dedicated SPI module?
> >
> > Also +1 (Alex mentioned the same). Kafka depending on runtime/service
> > just for the SPI is tricky, and having the mapper live in that SPI
> > module later makes a lot of sense. For the first PR I was thinking of
> > putting the shared mapper next to the existing events SPI in
> > runtime/service, then extracting SPI + mapper as a follow-up - mostly to
> > avoid getting stuck on module cycles before the envelope lands (JB had
> > flagged that). Will do the SPI split earlier if that feels better. WDYT
> ??
> >
> > Draft v1 JSON (CE context + today's Kafka body in "data"):
> >
> > {
> > "specversion": "1.0",
> > "id": "550e8400-e29b-41d4-a716-446655440000",
> > "type": "org.apache.polaris.AFTER_CREATE_TABLE",
> > "source": "org.apache.polaris",
> > "time": "2026-10-04T12:34:56.789Z",
> > "datacontenttype": "application/json",
> > "data": {
> > "event_type": "AFTER_CREATE_TABLE",
> > "event_id": "550e8400-e29b-41d4-a716-446655440000",
> > "timestamp": "2026-10-04T12:34:56.789Z",
> > "realm_id": "default-realm",
> > "principal": "alice",
> > "activated_roles": ["catalog_admin"],
> > "request_id": "req-123",
> > "catalog_name": "sales",
> > "namespace": ["analytics"],
> > "table_name": "orders",
> > "table_identifier": ["analytics", "orders"]
> > }
> > }
> >
> > A couple notes:
> > - "data" is basically today's Kafka EventToJsonMapper body, so id/time
> > show up both as CE context and again inside data (event_id /
> > timestamp) for Kafka parity.
> > - Type is a namespaced PolarisEventType name; source is a constant for
> > now - happy to bike-shed those if needed.
> > - Webhook delivery time stays transport metadata (HTTP header), not in
> > the shared envelope.
> > - Rename events would also carry source / destination arrays in "data",
> > same as Kafka today; omitted above because this example is
> > AFTER_CREATE_TABLE.
> >
> > If this looks right, I'll open the prerequisite PR on that basis...
> >
> > Thanks
> > *Vignesh*
> >
> > [1] https://github.com/apache/polaris/pull/5084
> >
> > On Tue, 6 Oct 2026 at 05:53, Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > > Hi Vgnesh,
> > >
> > > Your proposal sounds good to be at high level.
> > >
> > > Just to clarify: is our end goal here to use the same JSON payload for
> > all
> > > event listeners that produce JSON?
> > >
> > > Are we going to adhere to the CloudEvents spec for this JSON? Obviously
> > > starting with 1.0 and migrating to newer version as the become
> > available...
> > > Is that the plan?
> > >
> > > How are we going to migrare existing use cases that are not 1:1 with
> > > the ideal JSON schema? Will be publish a union of attributes for a few
> > > releases to allow consumers to switch to the new schema?
> > >
> > > +1 to adding a dedicated SPI module. I suppose it can also contain the
> > > common java-to-JSON mapping code.
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Mon, Oct 5, 2026 at 4:29 AM Alex Dutra <ad...@apache.org> wrote:
> > >
> > > > Hi all,
> > > >
> > > > +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1.
> > > >
> > > > I also welcome the idea of isolating the Events SPI in its own
> module.
> > > > Having the Kafka module depend on runtime/service just for the SPI is
> > > > concerning.
> > > >
> > > > Looking forward to the first PR!
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > > On Mon, Oct 5, 2026 at 10:02 AM vignesh a <im...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hi JB,
> > > > >
> > > > > Thanks - this is really useful, and I'm with you on the sequencing
> > > > > (shared envelope first, then rebase the webhook).
> > > > >
> > > > > Your points on shape make sense to me too:
> > > > > 1. CloudEvents context attrs at the top only, Polaris payload in
> > "data"
> > > > > (so things like activated_roles stay valid)
> > > > > 2. reuse the existing EventAttributes / Kafka snake_case keys in
> > "data"
> > > > > 3. v1 at Kafka parity, not the thinner webhook set I suggested
> > earlier
> > > > > 4. mapper next to the events SPI in runtime/service for now
> > > > > 5. deliverytime as webhook HTTP metadata, not in the shared
> envelope
> > > > > 6. OpenTelemetry + persistence listeners out of scope for this
> > > > >
> > > > > Also +1 on settling Kafka compat on the list. per-listener format
> > > option
> > > > > with today's flat payload as the default (shared envelope opt-in)
> > > sounds
> > > > > better than dual-publish.
> > > > >
> > > > > Draft v1 JSON I have in mind:
> > > > >
> > > > > {
> > > > > "specversion": "1.0",
> > > > > "id": "550e8400-e29b-41d4-a716-446655440000",
> > > > > "type": "org.apache.polaris.AFTER_CREATE_TABLE",
> > > > > "source": "org.apache.polaris",
> > > > > "time": "2026-10-04T12:34:56.789Z",
> > > > > "datacontenttype": "application/json",
> > > > > "data": {
> > > > > "event_type": "AFTER_CREATE_TABLE",
> > > > > "event_id": "550e8400-e29b-41d4-a716-446655440000",
> > > > > "timestamp": "2026-10-04T12:34:56.789Z",
> > > > > "realm_id": "default-realm",
> > > > > "principal": "alice",
> > > > > "activated_roles": ["catalog_admin"],
> > > > > "request_id": "req-123",
> > > > > "catalog_name": "sales",
> > > > > "namespace": ["analytics"],
> > > > > "table_name": "orders",
> > > > > "table_identifier": ["analytics", "orders"]
> > > > > }
> > > > > }
> > > > >
> > > > > A couple notes on purpose:
> > > > > - "data" is basically today's Kafka body, so id/time show up both
> as
> > CE
> > > > > context and again inside data (event_id / timestamp) for Kafka
> > parity
> > > > > - type is a namespaced PolarisEventType name; source is a constant
> > for
> > > > > now - will change if needed..
> > > > > - webhook would put delivery time on an HTTP header
> > > > > (e.g. X-Polaris-Delivery-Time), not in the envelope
> > > > >
> > > > > if this is roughly the right shape, I'll draft the prerequisite PR
> > > around
> > > > > it. if you'd rather nail type/source naming first, tell me which
> > forms
> > > > > you prefer.
> > > > >
> > > > > Thanks,
> > > > > *Vignesh*
> > > > >
> > > > > [1] https://github.com/apache/polaris/pull/5084
> > > > >
> > > > > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > > > wrote:
> > > > >
> > > > > > Hi Vignesh
> > > > > >
> > > > > > Thanks for starting this thread.
> > > > > >
> > > > > > I'm +1 on the sequence: shared envelope first and then rebase the
> > > > webhook
> > > > > > PR. I slightly prefer your proposal over B or C.
> > > > > >
> > > > > > Some general comments:
> > > > > > 1. CloudEvents requires attribute names to be lowercase
> > alphanumerics
> > > > and
> > > > > > attribute values to be scalars afair. So a flat set of extensions
> > > > doesn't
> > > > > > really work for us: activatedroles as an array is not valid for
> > > > > > instance. Or we have to transform somehow. I would keep only the
> > > > > > CloudEvents context attributes at the top level and put the
> Polaris
> > > > fields
> > > > > > in "data".
> > > > > > 2. The EventAttributes keys already have underscored names
> > > > (catalog_name,
> > > > > > table_identifier, ...) and they match what Kafka emits. I would
> > reuse
> > > > them
> > > > > > as is. Then the current Kafka body basically becomes "data",
> which
> > > also
> > > > > > makes the migration easier.
> > > > > > 3. I would not start from the thin set: migrating Kafka to it
> would
> > > > drop
> > > > > > fields that consumers already get (namespace, catalog, view, ..).
> > > Kafka
> > > > > > only has nine optional fields, so I think v1 should start at
> Kafka
> > > > parity.
> > > > > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I
> > > think
> > > > we
> > > > > > should agree on the mechanism here on the list rather than in the
> > PR
> > > > > > description (if it doesn't happen on the mailing list, it never
> > > happens
> > > > > > :)). I would prefer a per-listener format option, with the
> current
> > > > format
> > > > > > as default, over dual-publish (which doubles the volume on the
> > > topic).
> > > > > > 5. The events SPI and the CloudWatch listener are both in
> > > > runtime/service,
> > > > > > and the Kafka extension depends on it. A new module would
> > introduce a
> > > > cycle
> > > > > > unless we extract the SPI first, so I would put the mapper next
> to
> > > the
> > > > SPI
> > > > > > in runtime/service for now, and move it when the SPI is moved.
> > > > > > 6. deliverytime is a transport metadata, I would move it to a
> HTTP
> > > > header
> > > > > > instead of the shared envelope. We should also decide what "type"
> > and
> > > > > > "source" look like (today it's a bare enum name and a constant).
> > > > > >
> > > > > > Could you share the v1 JSON you have in mind on this thread
> before
> > > > opening
> > > > > > the PR? I think it will be easier to converge on an example.
> > > > > >
> > > > > > The OpenTelemetry and persistence listeners have their own
> > > > representations
> > > > > > as well. I'm fine keeping them out of scope, but it's worth
> stating
> > > it
> > > > > > explicitly.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a <
> imavignesh27@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > \Hi all,
> > > > > > >
> > > > > > > Robert and EJ have both asked that we settle a shared event
> > > envelope
> > > > > > > before merging the webhook listener [1]. I agree with that
> > > sequencing
> > > > > > > and would like a quick list consensus on shape and placement
> > > before I
> > > > > > > open the prerequisite PR.
> > > > > > >
> > > > > > > What each path emits today (from current main / [1]):
> > > > > > >
> > > > > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper):
> > > > > > > flat snake_case: event_type, event_id, timestamp, realm_id,
> > > > principal,
> > > > > > > activated_roles, request_id, plus a fairly rich attribute set
> > > > > > > (table_identifier as a string list, namespace/catalog/view
> > > fields,
> > > > > > > rename source/destination, and similar).
> > > > > > >
> > > > > > > - Webhook ([1], WebhookEventListener.toJson):
> > > > > > > CloudEvents-shaped: specversion, id, type, source, time, plus
> > > > > > > extensions deliverytime, realmid, tableidentifier (single
> > > string),
> > > > > > > principal, activatedroles, requestid. Thinner attribute
> > coverage
> > > > than
> > > > > > > Kafka.
> > > > > > >
> > > > > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
> > > > > > > flat map: event_type, optional table_identifier (string),
> > > realm_id,
> > > > > > > principal, activated_roles, request_id. No event_id in the
> JSON
> > > > body;
> > > > > > > timestamp is on InputLogEvent.
> > > > > > >
> > > > > > > So this is not a mechanical "extract EventToJsonMapper."
> Naming,
> > > > > > > structure, identifier encoding, and attribute breadth already
> > > > diverge,
> > > > > > > and the shared code should not live inside the kafka extension.
> > > > > > >
> > > > > > > Proposal:
> > > > > > >
> > > > > > > 1) Land a small prerequisite that introduces a shared,
> versioned,
> > > > > > > CloudEvents-shaped mapper (core fields + a documented
> > extension
> > > > set),
> > > > > > > placed outside any one transport extension.
> > > > > > >
> > > > > > > 2) Rebase [1] onto that shared mapper.
> > > > > > >
> > > > > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
> > > > > > > compatibility plan so existing consumers are not silently
> > broken
> > > > > > > (version field and/or dual-publish window; exact mechanism
> in
> > > the
> > > > > > > prerequisite PR description).
> > > > > > >
> > > > > > > 4) Attribute breadth for v1: start from the
> > webhook/CloudWatch-thin
> > > > > > > common set; add Kafka's richer attributes behind the same
> > > > versioned
> > > > > > > envelope rather than blocking v1 on full Kafka parity.
> > > > > > >
> > > > > > > If that direction is wrong, the useful alternatives seem to be:
> > > > > > >
> > > > > > > - B) Prefer Kafka's flat snake_case as the shared shape instead
> > of
> > > > > > > CloudEvents-shaped.
> > > > > > > - C) Fold the shared mapper into [1] itself (larger single PR).
> > > > > > >
> > > > > > > Unless someone prefers B or C, I will open the prerequisite PR
> > > along
> > > > the
> > > > > > > lines of (1)-(4) early next week.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Vignesh
> > > > > > >
> > > > > > > [1] https://github.com/apache/polaris/pull/5084
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Vignesh,
The proposal LGTM.
Minor comment: SInce CloudEvents apparently prefer lowercase names, shall
we lowercase the type namne (in JSON)?
Re: backward compatibility for Kafka. Having a flag to send old payloads
works for me. However, I believe this tends to prolong adoption and users
have to face a hard change anyway (when their admin flips the switch). This
may not be suitable for rolling upgrade scenarios. That said, we can
probably debate this later, after establishing the common format.
Cheers,
Dmitri.
On Tue, Oct 6, 2026 at 8:11 AM vignesh a <im...@gmail.com> wrote:
> Hi Dmitri,
>
> Thanks - good questions, and yes at a high level that's the direction.
> Wanted to answer them against what JB and I already talked through, and
> I've added the draft v1 JSON again so we're looking at the same thing.
>
> 1) Same JSON for all JSON-producing listeners?
>
> For the transport listeners in this thread (Kafka, webhook, CloudWatch),
> yes - same CloudEvents-shaped payload is the end goal. OpenTelemetry and
> the persistence listener keep their own representations for now; those
> stay out of scope for the first PR.
>
> 2) Adhere to CloudEvents?
>
> Yes. v1 means CloudEvents 1.0 context attributes at the top only, and the
> Polaris fields in "data". That also keeps us honest with CE rules on
> context attrs (lowercase alphanumerics / scalars) - so things like
> activated_roles live in "data", not as top-level CE extensions. If a
> newer CE version becomes relevant later we can bump then..
>
> 3) Migrating existing consumers that aren't 1:1 with the new shape?
>
> Union / dual-publish for a few releases is one way to do it. On the list
> JB leaned toward a per-listener format option instead (today's flat Kafka
> payload as the default, shared envelope opt-in), mainly so we don't
> double volume on the topic. That felt right to me too for 1.7 / 1.8
> consumers - they keep working until they flip the knob. Webhook [1] can
> land on the shared envelope from day one since it isn't released that
> way yet. Open to other ideas if you see a cleaner path.. LMK
>
> 4) Dedicated SPI module?
>
> Also +1 (Alex mentioned the same). Kafka depending on runtime/service
> just for the SPI is tricky, and having the mapper live in that SPI
> module later makes a lot of sense. For the first PR I was thinking of
> putting the shared mapper next to the existing events SPI in
> runtime/service, then extracting SPI + mapper as a follow-up - mostly to
> avoid getting stuck on module cycles before the envelope lands (JB had
> flagged that). Will do the SPI split earlier if that feels better. WDYT ??
>
> Draft v1 JSON (CE context + today's Kafka body in "data"):
>
> {
> "specversion": "1.0",
> "id": "550e8400-e29b-41d4-a716-446655440000",
> "type": "org.apache.polaris.AFTER_CREATE_TABLE",
> "source": "org.apache.polaris",
> "time": "2026-10-04T12:34:56.789Z",
> "datacontenttype": "application/json",
> "data": {
> "event_type": "AFTER_CREATE_TABLE",
> "event_id": "550e8400-e29b-41d4-a716-446655440000",
> "timestamp": "2026-10-04T12:34:56.789Z",
> "realm_id": "default-realm",
> "principal": "alice",
> "activated_roles": ["catalog_admin"],
> "request_id": "req-123",
> "catalog_name": "sales",
> "namespace": ["analytics"],
> "table_name": "orders",
> "table_identifier": ["analytics", "orders"]
> }
> }
>
> A couple notes:
> - "data" is basically today's Kafka EventToJsonMapper body, so id/time
> show up both as CE context and again inside data (event_id /
> timestamp) for Kafka parity.
> - Type is a namespaced PolarisEventType name; source is a constant for
> now - happy to bike-shed those if needed.
> - Webhook delivery time stays transport metadata (HTTP header), not in
> the shared envelope.
> - Rename events would also carry source / destination arrays in "data",
> same as Kafka today; omitted above because this example is
> AFTER_CREATE_TABLE.
>
> If this looks right, I'll open the prerequisite PR on that basis...
>
> Thanks
> *Vignesh*
>
> [1] https://github.com/apache/polaris/pull/5084
>
> On Tue, 6 Oct 2026 at 05:53, Dmitri Bourlatchkov <di...@apache.org> wrote:
>
> > Hi Vgnesh,
> >
> > Your proposal sounds good to be at high level.
> >
> > Just to clarify: is our end goal here to use the same JSON payload for
> all
> > event listeners that produce JSON?
> >
> > Are we going to adhere to the CloudEvents spec for this JSON? Obviously
> > starting with 1.0 and migrating to newer version as the become
> available...
> > Is that the plan?
> >
> > How are we going to migrare existing use cases that are not 1:1 with
> > the ideal JSON schema? Will be publish a union of attributes for a few
> > releases to allow consumers to switch to the new schema?
> >
> > +1 to adding a dedicated SPI module. I suppose it can also contain the
> > common java-to-JSON mapping code.
> >
> > Thanks,
> > Dmitri.
> >
> > On Mon, Oct 5, 2026 at 4:29 AM Alex Dutra <ad...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1.
> > >
> > > I also welcome the idea of isolating the Events SPI in its own module.
> > > Having the Kafka module depend on runtime/service just for the SPI is
> > > concerning.
> > >
> > > Looking forward to the first PR!
> > >
> > > Thanks,
> > > Alex
> > >
> > > On Mon, Oct 5, 2026 at 10:02 AM vignesh a <im...@gmail.com>
> > wrote:
> > > >
> > > > Hi JB,
> > > >
> > > > Thanks - this is really useful, and I'm with you on the sequencing
> > > > (shared envelope first, then rebase the webhook).
> > > >
> > > > Your points on shape make sense to me too:
> > > > 1. CloudEvents context attrs at the top only, Polaris payload in
> "data"
> > > > (so things like activated_roles stay valid)
> > > > 2. reuse the existing EventAttributes / Kafka snake_case keys in
> "data"
> > > > 3. v1 at Kafka parity, not the thinner webhook set I suggested
> earlier
> > > > 4. mapper next to the events SPI in runtime/service for now
> > > > 5. deliverytime as webhook HTTP metadata, not in the shared envelope
> > > > 6. OpenTelemetry + persistence listeners out of scope for this
> > > >
> > > > Also +1 on settling Kafka compat on the list. per-listener format
> > option
> > > > with today's flat payload as the default (shared envelope opt-in)
> > sounds
> > > > better than dual-publish.
> > > >
> > > > Draft v1 JSON I have in mind:
> > > >
> > > > {
> > > > "specversion": "1.0",
> > > > "id": "550e8400-e29b-41d4-a716-446655440000",
> > > > "type": "org.apache.polaris.AFTER_CREATE_TABLE",
> > > > "source": "org.apache.polaris",
> > > > "time": "2026-10-04T12:34:56.789Z",
> > > > "datacontenttype": "application/json",
> > > > "data": {
> > > > "event_type": "AFTER_CREATE_TABLE",
> > > > "event_id": "550e8400-e29b-41d4-a716-446655440000",
> > > > "timestamp": "2026-10-04T12:34:56.789Z",
> > > > "realm_id": "default-realm",
> > > > "principal": "alice",
> > > > "activated_roles": ["catalog_admin"],
> > > > "request_id": "req-123",
> > > > "catalog_name": "sales",
> > > > "namespace": ["analytics"],
> > > > "table_name": "orders",
> > > > "table_identifier": ["analytics", "orders"]
> > > > }
> > > > }
> > > >
> > > > A couple notes on purpose:
> > > > - "data" is basically today's Kafka body, so id/time show up both as
> CE
> > > > context and again inside data (event_id / timestamp) for Kafka
> parity
> > > > - type is a namespaced PolarisEventType name; source is a constant
> for
> > > > now - will change if needed..
> > > > - webhook would put delivery time on an HTTP header
> > > > (e.g. X-Polaris-Delivery-Time), not in the envelope
> > > >
> > > > if this is roughly the right shape, I'll draft the prerequisite PR
> > around
> > > > it. if you'd rather nail type/source naming first, tell me which
> forms
> > > > you prefer.
> > > >
> > > > Thanks,
> > > > *Vignesh*
> > > >
> > > > [1] https://github.com/apache/polaris/pull/5084
> > > >
> > > > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > > >
> > > > > Hi Vignesh
> > > > >
> > > > > Thanks for starting this thread.
> > > > >
> > > > > I'm +1 on the sequence: shared envelope first and then rebase the
> > > webhook
> > > > > PR. I slightly prefer your proposal over B or C.
> > > > >
> > > > > Some general comments:
> > > > > 1. CloudEvents requires attribute names to be lowercase
> alphanumerics
> > > and
> > > > > attribute values to be scalars afair. So a flat set of extensions
> > > doesn't
> > > > > really work for us: activatedroles as an array is not valid for
> > > > > instance. Or we have to transform somehow. I would keep only the
> > > > > CloudEvents context attributes at the top level and put the Polaris
> > > fields
> > > > > in "data".
> > > > > 2. The EventAttributes keys already have underscored names
> > > (catalog_name,
> > > > > table_identifier, ...) and they match what Kafka emits. I would
> reuse
> > > them
> > > > > as is. Then the current Kafka body basically becomes "data", which
> > also
> > > > > makes the migration easier.
> > > > > 3. I would not start from the thin set: migrating Kafka to it would
> > > drop
> > > > > fields that consumers already get (namespace, catalog, view, ..).
> > Kafka
> > > > > only has nine optional fields, so I think v1 should start at Kafka
> > > parity.
> > > > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I
> > think
> > > we
> > > > > should agree on the mechanism here on the list rather than in the
> PR
> > > > > description (if it doesn't happen on the mailing list, it never
> > happens
> > > > > :)). I would prefer a per-listener format option, with the current
> > > format
> > > > > as default, over dual-publish (which doubles the volume on the
> > topic).
> > > > > 5. The events SPI and the CloudWatch listener are both in
> > > runtime/service,
> > > > > and the Kafka extension depends on it. A new module would
> introduce a
> > > cycle
> > > > > unless we extract the SPI first, so I would put the mapper next to
> > the
> > > SPI
> > > > > in runtime/service for now, and move it when the SPI is moved.
> > > > > 6. deliverytime is a transport metadata, I would move it to a HTTP
> > > header
> > > > > instead of the shared envelope. We should also decide what "type"
> and
> > > > > "source" look like (today it's a bare enum name and a constant).
> > > > >
> > > > > Could you share the v1 JSON you have in mind on this thread before
> > > opening
> > > > > the PR? I think it will be easier to converge on an example.
> > > > >
> > > > > The OpenTelemetry and persistence listeners have their own
> > > representations
> > > > > as well. I'm fine keeping them out of scope, but it's worth stating
> > it
> > > > > explicitly.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a <im...@gmail.com>
> > > wrote:
> > > > >
> > > > > > \Hi all,
> > > > > >
> > > > > > Robert and EJ have both asked that we settle a shared event
> > envelope
> > > > > > before merging the webhook listener [1]. I agree with that
> > sequencing
> > > > > > and would like a quick list consensus on shape and placement
> > before I
> > > > > > open the prerequisite PR.
> > > > > >
> > > > > > What each path emits today (from current main / [1]):
> > > > > >
> > > > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper):
> > > > > > flat snake_case: event_type, event_id, timestamp, realm_id,
> > > principal,
> > > > > > activated_roles, request_id, plus a fairly rich attribute set
> > > > > > (table_identifier as a string list, namespace/catalog/view
> > fields,
> > > > > > rename source/destination, and similar).
> > > > > >
> > > > > > - Webhook ([1], WebhookEventListener.toJson):
> > > > > > CloudEvents-shaped: specversion, id, type, source, time, plus
> > > > > > extensions deliverytime, realmid, tableidentifier (single
> > string),
> > > > > > principal, activatedroles, requestid. Thinner attribute
> coverage
> > > than
> > > > > > Kafka.
> > > > > >
> > > > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
> > > > > > flat map: event_type, optional table_identifier (string),
> > realm_id,
> > > > > > principal, activated_roles, request_id. No event_id in the JSON
> > > body;
> > > > > > timestamp is on InputLogEvent.
> > > > > >
> > > > > > So this is not a mechanical "extract EventToJsonMapper." Naming,
> > > > > > structure, identifier encoding, and attribute breadth already
> > > diverge,
> > > > > > and the shared code should not live inside the kafka extension.
> > > > > >
> > > > > > Proposal:
> > > > > >
> > > > > > 1) Land a small prerequisite that introduces a shared, versioned,
> > > > > > CloudEvents-shaped mapper (core fields + a documented
> extension
> > > set),
> > > > > > placed outside any one transport extension.
> > > > > >
> > > > > > 2) Rebase [1] onto that shared mapper.
> > > > > >
> > > > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
> > > > > > compatibility plan so existing consumers are not silently
> broken
> > > > > > (version field and/or dual-publish window; exact mechanism in
> > the
> > > > > > prerequisite PR description).
> > > > > >
> > > > > > 4) Attribute breadth for v1: start from the
> webhook/CloudWatch-thin
> > > > > > common set; add Kafka's richer attributes behind the same
> > > versioned
> > > > > > envelope rather than blocking v1 on full Kafka parity.
> > > > > >
> > > > > > If that direction is wrong, the useful alternatives seem to be:
> > > > > >
> > > > > > - B) Prefer Kafka's flat snake_case as the shared shape instead
> of
> > > > > > CloudEvents-shaped.
> > > > > > - C) Fold the shared mapper into [1] itself (larger single PR).
> > > > > >
> > > > > > Unless someone prefers B or C, I will open the prerequisite PR
> > along
> > > the
> > > > > > lines of (1)-(4) early next week.
> > > > > >
> > > > > > Thanks,
> > > > > > Vignesh
> > > > > >
> > > > > > [1] https://github.com/apache/polaris/pull/5084
> > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
Posted by vignesh a <im...@gmail.com>.
Hi Dmitri,
Thanks - good questions, and yes at a high level that's the direction.
Wanted to answer them against what JB and I already talked through, and
I've added the draft v1 JSON again so we're looking at the same thing.
1) Same JSON for all JSON-producing listeners?
For the transport listeners in this thread (Kafka, webhook, CloudWatch),
yes - same CloudEvents-shaped payload is the end goal. OpenTelemetry and
the persistence listener keep their own representations for now; those
stay out of scope for the first PR.
2) Adhere to CloudEvents?
Yes. v1 means CloudEvents 1.0 context attributes at the top only, and the
Polaris fields in "data". That also keeps us honest with CE rules on
context attrs (lowercase alphanumerics / scalars) - so things like
activated_roles live in "data", not as top-level CE extensions. If a
newer CE version becomes relevant later we can bump then..
3) Migrating existing consumers that aren't 1:1 with the new shape?
Union / dual-publish for a few releases is one way to do it. On the list
JB leaned toward a per-listener format option instead (today's flat Kafka
payload as the default, shared envelope opt-in), mainly so we don't
double volume on the topic. That felt right to me too for 1.7 / 1.8
consumers - they keep working until they flip the knob. Webhook [1] can
land on the shared envelope from day one since it isn't released that
way yet. Open to other ideas if you see a cleaner path.. LMK
4) Dedicated SPI module?
Also +1 (Alex mentioned the same). Kafka depending on runtime/service
just for the SPI is tricky, and having the mapper live in that SPI
module later makes a lot of sense. For the first PR I was thinking of
putting the shared mapper next to the existing events SPI in
runtime/service, then extracting SPI + mapper as a follow-up - mostly to
avoid getting stuck on module cycles before the envelope lands (JB had
flagged that). Will do the SPI split earlier if that feels better. WDYT ??
Draft v1 JSON (CE context + today's Kafka body in "data"):
{
"specversion": "1.0",
"id": "550e8400-e29b-41d4-a716-446655440000",
"type": "org.apache.polaris.AFTER_CREATE_TABLE",
"source": "org.apache.polaris",
"time": "2026-10-04T12:34:56.789Z",
"datacontenttype": "application/json",
"data": {
"event_type": "AFTER_CREATE_TABLE",
"event_id": "550e8400-e29b-41d4-a716-446655440000",
"timestamp": "2026-10-04T12:34:56.789Z",
"realm_id": "default-realm",
"principal": "alice",
"activated_roles": ["catalog_admin"],
"request_id": "req-123",
"catalog_name": "sales",
"namespace": ["analytics"],
"table_name": "orders",
"table_identifier": ["analytics", "orders"]
}
}
A couple notes:
- "data" is basically today's Kafka EventToJsonMapper body, so id/time
show up both as CE context and again inside data (event_id /
timestamp) for Kafka parity.
- Type is a namespaced PolarisEventType name; source is a constant for
now - happy to bike-shed those if needed.
- Webhook delivery time stays transport metadata (HTTP header), not in
the shared envelope.
- Rename events would also carry source / destination arrays in "data",
same as Kafka today; omitted above because this example is
AFTER_CREATE_TABLE.
If this looks right, I'll open the prerequisite PR on that basis...
Thanks
*Vignesh*
[1] https://github.com/apache/polaris/pull/5084
On Tue, 6 Oct 2026 at 05:53, Dmitri Bourlatchkov <di...@apache.org> wrote:
> Hi Vgnesh,
>
> Your proposal sounds good to be at high level.
>
> Just to clarify: is our end goal here to use the same JSON payload for all
> event listeners that produce JSON?
>
> Are we going to adhere to the CloudEvents spec for this JSON? Obviously
> starting with 1.0 and migrating to newer version as the become available...
> Is that the plan?
>
> How are we going to migrare existing use cases that are not 1:1 with
> the ideal JSON schema? Will be publish a union of attributes for a few
> releases to allow consumers to switch to the new schema?
>
> +1 to adding a dedicated SPI module. I suppose it can also contain the
> common java-to-JSON mapping code.
>
> Thanks,
> Dmitri.
>
> On Mon, Oct 5, 2026 at 4:29 AM Alex Dutra <ad...@apache.org> wrote:
>
> > Hi all,
> >
> > +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1.
> >
> > I also welcome the idea of isolating the Events SPI in its own module.
> > Having the Kafka module depend on runtime/service just for the SPI is
> > concerning.
> >
> > Looking forward to the first PR!
> >
> > Thanks,
> > Alex
> >
> > On Mon, Oct 5, 2026 at 10:02 AM vignesh a <im...@gmail.com>
> wrote:
> > >
> > > Hi JB,
> > >
> > > Thanks - this is really useful, and I'm with you on the sequencing
> > > (shared envelope first, then rebase the webhook).
> > >
> > > Your points on shape make sense to me too:
> > > 1. CloudEvents context attrs at the top only, Polaris payload in "data"
> > > (so things like activated_roles stay valid)
> > > 2. reuse the existing EventAttributes / Kafka snake_case keys in "data"
> > > 3. v1 at Kafka parity, not the thinner webhook set I suggested earlier
> > > 4. mapper next to the events SPI in runtime/service for now
> > > 5. deliverytime as webhook HTTP metadata, not in the shared envelope
> > > 6. OpenTelemetry + persistence listeners out of scope for this
> > >
> > > Also +1 on settling Kafka compat on the list. per-listener format
> option
> > > with today's flat payload as the default (shared envelope opt-in)
> sounds
> > > better than dual-publish.
> > >
> > > Draft v1 JSON I have in mind:
> > >
> > > {
> > > "specversion": "1.0",
> > > "id": "550e8400-e29b-41d4-a716-446655440000",
> > > "type": "org.apache.polaris.AFTER_CREATE_TABLE",
> > > "source": "org.apache.polaris",
> > > "time": "2026-10-04T12:34:56.789Z",
> > > "datacontenttype": "application/json",
> > > "data": {
> > > "event_type": "AFTER_CREATE_TABLE",
> > > "event_id": "550e8400-e29b-41d4-a716-446655440000",
> > > "timestamp": "2026-10-04T12:34:56.789Z",
> > > "realm_id": "default-realm",
> > > "principal": "alice",
> > > "activated_roles": ["catalog_admin"],
> > > "request_id": "req-123",
> > > "catalog_name": "sales",
> > > "namespace": ["analytics"],
> > > "table_name": "orders",
> > > "table_identifier": ["analytics", "orders"]
> > > }
> > > }
> > >
> > > A couple notes on purpose:
> > > - "data" is basically today's Kafka body, so id/time show up both as CE
> > > context and again inside data (event_id / timestamp) for Kafka parity
> > > - type is a namespaced PolarisEventType name; source is a constant for
> > > now - will change if needed..
> > > - webhook would put delivery time on an HTTP header
> > > (e.g. X-Polaris-Delivery-Time), not in the envelope
> > >
> > > if this is roughly the right shape, I'll draft the prerequisite PR
> around
> > > it. if you'd rather nail type/source naming first, tell me which forms
> > > you prefer.
> > >
> > > Thanks,
> > > *Vignesh*
> > >
> > > [1] https://github.com/apache/polaris/pull/5084
> > >
> > > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > >
> > > > Hi Vignesh
> > > >
> > > > Thanks for starting this thread.
> > > >
> > > > I'm +1 on the sequence: shared envelope first and then rebase the
> > webhook
> > > > PR. I slightly prefer your proposal over B or C.
> > > >
> > > > Some general comments:
> > > > 1. CloudEvents requires attribute names to be lowercase alphanumerics
> > and
> > > > attribute values to be scalars afair. So a flat set of extensions
> > doesn't
> > > > really work for us: activatedroles as an array is not valid for
> > > > instance. Or we have to transform somehow. I would keep only the
> > > > CloudEvents context attributes at the top level and put the Polaris
> > fields
> > > > in "data".
> > > > 2. The EventAttributes keys already have underscored names
> > (catalog_name,
> > > > table_identifier, ...) and they match what Kafka emits. I would reuse
> > them
> > > > as is. Then the current Kafka body basically becomes "data", which
> also
> > > > makes the migration easier.
> > > > 3. I would not start from the thin set: migrating Kafka to it would
> > drop
> > > > fields that consumers already get (namespace, catalog, view, ..).
> Kafka
> > > > only has nine optional fields, so I think v1 should start at Kafka
> > parity.
> > > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I
> think
> > we
> > > > should agree on the mechanism here on the list rather than in the PR
> > > > description (if it doesn't happen on the mailing list, it never
> happens
> > > > :)). I would prefer a per-listener format option, with the current
> > format
> > > > as default, over dual-publish (which doubles the volume on the
> topic).
> > > > 5. The events SPI and the CloudWatch listener are both in
> > runtime/service,
> > > > and the Kafka extension depends on it. A new module would introduce a
> > cycle
> > > > unless we extract the SPI first, so I would put the mapper next to
> the
> > SPI
> > > > in runtime/service for now, and move it when the SPI is moved.
> > > > 6. deliverytime is a transport metadata, I would move it to a HTTP
> > header
> > > > instead of the shared envelope. We should also decide what "type" and
> > > > "source" look like (today it's a bare enum name and a constant).
> > > >
> > > > Could you share the v1 JSON you have in mind on this thread before
> > opening
> > > > the PR? I think it will be easier to converge on an example.
> > > >
> > > > The OpenTelemetry and persistence listeners have their own
> > representations
> > > > as well. I'm fine keeping them out of scope, but it's worth stating
> it
> > > > explicitly.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a <im...@gmail.com>
> > wrote:
> > > >
> > > > > \Hi all,
> > > > >
> > > > > Robert and EJ have both asked that we settle a shared event
> envelope
> > > > > before merging the webhook listener [1]. I agree with that
> sequencing
> > > > > and would like a quick list consensus on shape and placement
> before I
> > > > > open the prerequisite PR.
> > > > >
> > > > > What each path emits today (from current main / [1]):
> > > > >
> > > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper):
> > > > > flat snake_case: event_type, event_id, timestamp, realm_id,
> > principal,
> > > > > activated_roles, request_id, plus a fairly rich attribute set
> > > > > (table_identifier as a string list, namespace/catalog/view
> fields,
> > > > > rename source/destination, and similar).
> > > > >
> > > > > - Webhook ([1], WebhookEventListener.toJson):
> > > > > CloudEvents-shaped: specversion, id, type, source, time, plus
> > > > > extensions deliverytime, realmid, tableidentifier (single
> string),
> > > > > principal, activatedroles, requestid. Thinner attribute coverage
> > than
> > > > > Kafka.
> > > > >
> > > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
> > > > > flat map: event_type, optional table_identifier (string),
> realm_id,
> > > > > principal, activated_roles, request_id. No event_id in the JSON
> > body;
> > > > > timestamp is on InputLogEvent.
> > > > >
> > > > > So this is not a mechanical "extract EventToJsonMapper." Naming,
> > > > > structure, identifier encoding, and attribute breadth already
> > diverge,
> > > > > and the shared code should not live inside the kafka extension.
> > > > >
> > > > > Proposal:
> > > > >
> > > > > 1) Land a small prerequisite that introduces a shared, versioned,
> > > > > CloudEvents-shaped mapper (core fields + a documented extension
> > set),
> > > > > placed outside any one transport extension.
> > > > >
> > > > > 2) Rebase [1] onto that shared mapper.
> > > > >
> > > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
> > > > > compatibility plan so existing consumers are not silently broken
> > > > > (version field and/or dual-publish window; exact mechanism in
> the
> > > > > prerequisite PR description).
> > > > >
> > > > > 4) Attribute breadth for v1: start from the webhook/CloudWatch-thin
> > > > > common set; add Kafka's richer attributes behind the same
> > versioned
> > > > > envelope rather than blocking v1 on full Kafka parity.
> > > > >
> > > > > If that direction is wrong, the useful alternatives seem to be:
> > > > >
> > > > > - B) Prefer Kafka's flat snake_case as the shared shape instead of
> > > > > CloudEvents-shaped.
> > > > > - C) Fold the shared mapper into [1] itself (larger single PR).
> > > > >
> > > > > Unless someone prefers B or C, I will open the prerequisite PR
> along
> > the
> > > > > lines of (1)-(4) early next week.
> > > > >
> > > > > Thanks,
> > > > > Vignesh
> > > > >
> > > > > [1] https://github.com/apache/polaris/pull/5084
> > > > >
> > > >
> >
>
Re: [DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Vgnesh,
Your proposal sounds good to be at high level.
Just to clarify: is our end goal here to use the same JSON payload for all
event listeners that produce JSON?
Are we going to adhere to the CloudEvents spec for this JSON? Obviously
starting with 1.0 and migrating to newer version as the become available...
Is that the plan?
How are we going to migrare existing use cases that are not 1:1 with
the ideal JSON schema? Will be publish a union of attributes for a few
releases to allow consumers to switch to the new schema?
+1 to adding a dedicated SPI module. I suppose it can also contain the
common java-to-JSON mapping code.
Thanks,
Dmitri.
On Mon, Oct 5, 2026 at 4:29 AM Alex Dutra <ad...@apache.org> wrote:
> Hi all,
>
> +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1.
>
> I also welcome the idea of isolating the Events SPI in its own module.
> Having the Kafka module depend on runtime/service just for the SPI is
> concerning.
>
> Looking forward to the first PR!
>
> Thanks,
> Alex
>
> On Mon, Oct 5, 2026 at 10:02 AM vignesh a <im...@gmail.com> wrote:
> >
> > Hi JB,
> >
> > Thanks - this is really useful, and I'm with you on the sequencing
> > (shared envelope first, then rebase the webhook).
> >
> > Your points on shape make sense to me too:
> > 1. CloudEvents context attrs at the top only, Polaris payload in "data"
> > (so things like activated_roles stay valid)
> > 2. reuse the existing EventAttributes / Kafka snake_case keys in "data"
> > 3. v1 at Kafka parity, not the thinner webhook set I suggested earlier
> > 4. mapper next to the events SPI in runtime/service for now
> > 5. deliverytime as webhook HTTP metadata, not in the shared envelope
> > 6. OpenTelemetry + persistence listeners out of scope for this
> >
> > Also +1 on settling Kafka compat on the list. per-listener format option
> > with today's flat payload as the default (shared envelope opt-in) sounds
> > better than dual-publish.
> >
> > Draft v1 JSON I have in mind:
> >
> > {
> > "specversion": "1.0",
> > "id": "550e8400-e29b-41d4-a716-446655440000",
> > "type": "org.apache.polaris.AFTER_CREATE_TABLE",
> > "source": "org.apache.polaris",
> > "time": "2026-10-04T12:34:56.789Z",
> > "datacontenttype": "application/json",
> > "data": {
> > "event_type": "AFTER_CREATE_TABLE",
> > "event_id": "550e8400-e29b-41d4-a716-446655440000",
> > "timestamp": "2026-10-04T12:34:56.789Z",
> > "realm_id": "default-realm",
> > "principal": "alice",
> > "activated_roles": ["catalog_admin"],
> > "request_id": "req-123",
> > "catalog_name": "sales",
> > "namespace": ["analytics"],
> > "table_name": "orders",
> > "table_identifier": ["analytics", "orders"]
> > }
> > }
> >
> > A couple notes on purpose:
> > - "data" is basically today's Kafka body, so id/time show up both as CE
> > context and again inside data (event_id / timestamp) for Kafka parity
> > - type is a namespaced PolarisEventType name; source is a constant for
> > now - will change if needed..
> > - webhook would put delivery time on an HTTP header
> > (e.g. X-Polaris-Delivery-Time), not in the envelope
> >
> > if this is roughly the right shape, I'll draft the prerequisite PR around
> > it. if you'd rather nail type/source naming first, tell me which forms
> > you prefer.
> >
> > Thanks,
> > *Vignesh*
> >
> > [1] https://github.com/apache/polaris/pull/5084
> >
> > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> > > Hi Vignesh
> > >
> > > Thanks for starting this thread.
> > >
> > > I'm +1 on the sequence: shared envelope first and then rebase the
> webhook
> > > PR. I slightly prefer your proposal over B or C.
> > >
> > > Some general comments:
> > > 1. CloudEvents requires attribute names to be lowercase alphanumerics
> and
> > > attribute values to be scalars afair. So a flat set of extensions
> doesn't
> > > really work for us: activatedroles as an array is not valid for
> > > instance. Or we have to transform somehow. I would keep only the
> > > CloudEvents context attributes at the top level and put the Polaris
> fields
> > > in "data".
> > > 2. The EventAttributes keys already have underscored names
> (catalog_name,
> > > table_identifier, ...) and they match what Kafka emits. I would reuse
> them
> > > as is. Then the current Kafka body basically becomes "data", which also
> > > makes the migration easier.
> > > 3. I would not start from the thin set: migrating Kafka to it would
> drop
> > > fields that consumers already get (namespace, catalog, view, ..). Kafka
> > > only has nine optional fields, so I think v1 should start at Kafka
> parity.
> > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I think
> we
> > > should agree on the mechanism here on the list rather than in the PR
> > > description (if it doesn't happen on the mailing list, it never happens
> > > :)). I would prefer a per-listener format option, with the current
> format
> > > as default, over dual-publish (which doubles the volume on the topic).
> > > 5. The events SPI and the CloudWatch listener are both in
> runtime/service,
> > > and the Kafka extension depends on it. A new module would introduce a
> cycle
> > > unless we extract the SPI first, so I would put the mapper next to the
> SPI
> > > in runtime/service for now, and move it when the SPI is moved.
> > > 6. deliverytime is a transport metadata, I would move it to a HTTP
> header
> > > instead of the shared envelope. We should also decide what "type" and
> > > "source" look like (today it's a bare enum name and a constant).
> > >
> > > Could you share the v1 JSON you have in mind on this thread before
> opening
> > > the PR? I think it will be easier to converge on an example.
> > >
> > > The OpenTelemetry and persistence listeners have their own
> representations
> > > as well. I'm fine keeping them out of scope, but it's worth stating it
> > > explicitly.
> > >
> > > Regards
> > > JB
> > >
> > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a <im...@gmail.com>
> wrote:
> > >
> > > > \Hi all,
> > > >
> > > > Robert and EJ have both asked that we settle a shared event envelope
> > > > before merging the webhook listener [1]. I agree with that sequencing
> > > > and would like a quick list consensus on shape and placement before I
> > > > open the prerequisite PR.
> > > >
> > > > What each path emits today (from current main / [1]):
> > > >
> > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper):
> > > > flat snake_case: event_type, event_id, timestamp, realm_id,
> principal,
> > > > activated_roles, request_id, plus a fairly rich attribute set
> > > > (table_identifier as a string list, namespace/catalog/view fields,
> > > > rename source/destination, and similar).
> > > >
> > > > - Webhook ([1], WebhookEventListener.toJson):
> > > > CloudEvents-shaped: specversion, id, type, source, time, plus
> > > > extensions deliverytime, realmid, tableidentifier (single string),
> > > > principal, activatedroles, requestid. Thinner attribute coverage
> than
> > > > Kafka.
> > > >
> > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
> > > > flat map: event_type, optional table_identifier (string), realm_id,
> > > > principal, activated_roles, request_id. No event_id in the JSON
> body;
> > > > timestamp is on InputLogEvent.
> > > >
> > > > So this is not a mechanical "extract EventToJsonMapper." Naming,
> > > > structure, identifier encoding, and attribute breadth already
> diverge,
> > > > and the shared code should not live inside the kafka extension.
> > > >
> > > > Proposal:
> > > >
> > > > 1) Land a small prerequisite that introduces a shared, versioned,
> > > > CloudEvents-shaped mapper (core fields + a documented extension
> set),
> > > > placed outside any one transport extension.
> > > >
> > > > 2) Rebase [1] onto that shared mapper.
> > > >
> > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
> > > > compatibility plan so existing consumers are not silently broken
> > > > (version field and/or dual-publish window; exact mechanism in the
> > > > prerequisite PR description).
> > > >
> > > > 4) Attribute breadth for v1: start from the webhook/CloudWatch-thin
> > > > common set; add Kafka's richer attributes behind the same
> versioned
> > > > envelope rather than blocking v1 on full Kafka parity.
> > > >
> > > > If that direction is wrong, the useful alternatives seem to be:
> > > >
> > > > - B) Prefer Kafka's flat snake_case as the shared shape instead of
> > > > CloudEvents-shaped.
> > > > - C) Fold the shared mapper into [1] itself (larger single PR).
> > > >
> > > > Unless someone prefers B or C, I will open the prerequisite PR along
> the
> > > > lines of (1)-(4) early next week.
> > > >
> > > > Thanks,
> > > > Vignesh
> > > >
> > > > [1] https://github.com/apache/polaris/pull/5084
> > > >
> > >
>
Re: [DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
Posted by Alex Dutra <ad...@apache.org>.
Hi all,
+1 for the proposed v1 JSON, and +1 for having Kafka parity in v1.
I also welcome the idea of isolating the Events SPI in its own module.
Having the Kafka module depend on runtime/service just for the SPI is
concerning.
Looking forward to the first PR!
Thanks,
Alex
On Mon, Oct 5, 2026 at 10:02 AM vignesh a <im...@gmail.com> wrote:
>
> Hi JB,
>
> Thanks - this is really useful, and I'm with you on the sequencing
> (shared envelope first, then rebase the webhook).
>
> Your points on shape make sense to me too:
> 1. CloudEvents context attrs at the top only, Polaris payload in "data"
> (so things like activated_roles stay valid)
> 2. reuse the existing EventAttributes / Kafka snake_case keys in "data"
> 3. v1 at Kafka parity, not the thinner webhook set I suggested earlier
> 4. mapper next to the events SPI in runtime/service for now
> 5. deliverytime as webhook HTTP metadata, not in the shared envelope
> 6. OpenTelemetry + persistence listeners out of scope for this
>
> Also +1 on settling Kafka compat on the list. per-listener format option
> with today's flat payload as the default (shared envelope opt-in) sounds
> better than dual-publish.
>
> Draft v1 JSON I have in mind:
>
> {
> "specversion": "1.0",
> "id": "550e8400-e29b-41d4-a716-446655440000",
> "type": "org.apache.polaris.AFTER_CREATE_TABLE",
> "source": "org.apache.polaris",
> "time": "2026-10-04T12:34:56.789Z",
> "datacontenttype": "application/json",
> "data": {
> "event_type": "AFTER_CREATE_TABLE",
> "event_id": "550e8400-e29b-41d4-a716-446655440000",
> "timestamp": "2026-10-04T12:34:56.789Z",
> "realm_id": "default-realm",
> "principal": "alice",
> "activated_roles": ["catalog_admin"],
> "request_id": "req-123",
> "catalog_name": "sales",
> "namespace": ["analytics"],
> "table_name": "orders",
> "table_identifier": ["analytics", "orders"]
> }
> }
>
> A couple notes on purpose:
> - "data" is basically today's Kafka body, so id/time show up both as CE
> context and again inside data (event_id / timestamp) for Kafka parity
> - type is a namespaced PolarisEventType name; source is a constant for
> now - will change if needed..
> - webhook would put delivery time on an HTTP header
> (e.g. X-Polaris-Delivery-Time), not in the envelope
>
> if this is roughly the right shape, I'll draft the prerequisite PR around
> it. if you'd rather nail type/source naming first, tell me which forms
> you prefer.
>
> Thanks,
> *Vignesh*
>
> [1] https://github.com/apache/polaris/pull/5084
>
> On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> > Hi Vignesh
> >
> > Thanks for starting this thread.
> >
> > I'm +1 on the sequence: shared envelope first and then rebase the webhook
> > PR. I slightly prefer your proposal over B or C.
> >
> > Some general comments:
> > 1. CloudEvents requires attribute names to be lowercase alphanumerics and
> > attribute values to be scalars afair. So a flat set of extensions doesn't
> > really work for us: activatedroles as an array is not valid for
> > instance. Or we have to transform somehow. I would keep only the
> > CloudEvents context attributes at the top level and put the Polaris fields
> > in "data".
> > 2. The EventAttributes keys already have underscored names (catalog_name,
> > table_identifier, ...) and they match what Kafka emits. I would reuse them
> > as is. Then the current Kafka body basically becomes "data", which also
> > makes the migration easier.
> > 3. I would not start from the thin set: migrating Kafka to it would drop
> > fields that consumers already get (namespace, catalog, view, ..). Kafka
> > only has nine optional fields, so I think v1 should start at Kafka parity.
> > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I think we
> > should agree on the mechanism here on the list rather than in the PR
> > description (if it doesn't happen on the mailing list, it never happens
> > :)). I would prefer a per-listener format option, with the current format
> > as default, over dual-publish (which doubles the volume on the topic).
> > 5. The events SPI and the CloudWatch listener are both in runtime/service,
> > and the Kafka extension depends on it. A new module would introduce a cycle
> > unless we extract the SPI first, so I would put the mapper next to the SPI
> > in runtime/service for now, and move it when the SPI is moved.
> > 6. deliverytime is a transport metadata, I would move it to a HTTP header
> > instead of the shared envelope. We should also decide what "type" and
> > "source" look like (today it's a bare enum name and a constant).
> >
> > Could you share the v1 JSON you have in mind on this thread before opening
> > the PR? I think it will be easier to converge on an example.
> >
> > The OpenTelemetry and persistence listeners have their own representations
> > as well. I'm fine keeping them out of scope, but it's worth stating it
> > explicitly.
> >
> > Regards
> > JB
> >
> > On Sun, Oct 4, 2026 at 11:47 AM vignesh a <im...@gmail.com> wrote:
> >
> > > \Hi all,
> > >
> > > Robert and EJ have both asked that we settle a shared event envelope
> > > before merging the webhook listener [1]. I agree with that sequencing
> > > and would like a quick list consensus on shape and placement before I
> > > open the prerequisite PR.
> > >
> > > What each path emits today (from current main / [1]):
> > >
> > > - Kafka (extensions/events/kafka/.../EventToJsonMapper):
> > > flat snake_case: event_type, event_id, timestamp, realm_id, principal,
> > > activated_roles, request_id, plus a fairly rich attribute set
> > > (table_identifier as a string list, namespace/catalog/view fields,
> > > rename source/destination, and similar).
> > >
> > > - Webhook ([1], WebhookEventListener.toJson):
> > > CloudEvents-shaped: specversion, id, type, source, time, plus
> > > extensions deliverytime, realmid, tableidentifier (single string),
> > > principal, activatedroles, requestid. Thinner attribute coverage than
> > > Kafka.
> > >
> > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
> > > flat map: event_type, optional table_identifier (string), realm_id,
> > > principal, activated_roles, request_id. No event_id in the JSON body;
> > > timestamp is on InputLogEvent.
> > >
> > > So this is not a mechanical "extract EventToJsonMapper." Naming,
> > > structure, identifier encoding, and attribute breadth already diverge,
> > > and the shared code should not live inside the kafka extension.
> > >
> > > Proposal:
> > >
> > > 1) Land a small prerequisite that introduces a shared, versioned,
> > > CloudEvents-shaped mapper (core fields + a documented extension set),
> > > placed outside any one transport extension.
> > >
> > > 2) Rebase [1] onto that shared mapper.
> > >
> > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
> > > compatibility plan so existing consumers are not silently broken
> > > (version field and/or dual-publish window; exact mechanism in the
> > > prerequisite PR description).
> > >
> > > 4) Attribute breadth for v1: start from the webhook/CloudWatch-thin
> > > common set; add Kafka's richer attributes behind the same versioned
> > > envelope rather than blocking v1 on full Kafka parity.
> > >
> > > If that direction is wrong, the useful alternatives seem to be:
> > >
> > > - B) Prefer Kafka's flat snake_case as the shared shape instead of
> > > CloudEvents-shaped.
> > > - C) Fold the shared mapper into [1] itself (larger single PR).
> > >
> > > Unless someone prefers B or C, I will open the prerequisite PR along the
> > > lines of (1)-(4) early next week.
> > >
> > > Thanks,
> > > Vignesh
> > >
> > > [1] https://github.com/apache/polaris/pull/5084
> > >
> >
Re: [DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
Posted by vignesh a <im...@gmail.com>.
Hi JB,
Thanks - this is really useful, and I'm with you on the sequencing
(shared envelope first, then rebase the webhook).
Your points on shape make sense to me too:
1. CloudEvents context attrs at the top only, Polaris payload in "data"
(so things like activated_roles stay valid)
2. reuse the existing EventAttributes / Kafka snake_case keys in "data"
3. v1 at Kafka parity, not the thinner webhook set I suggested earlier
4. mapper next to the events SPI in runtime/service for now
5. deliverytime as webhook HTTP metadata, not in the shared envelope
6. OpenTelemetry + persistence listeners out of scope for this
Also +1 on settling Kafka compat on the list. per-listener format option
with today's flat payload as the default (shared envelope opt-in) sounds
better than dual-publish.
Draft v1 JSON I have in mind:
{
"specversion": "1.0",
"id": "550e8400-e29b-41d4-a716-446655440000",
"type": "org.apache.polaris.AFTER_CREATE_TABLE",
"source": "org.apache.polaris",
"time": "2026-10-04T12:34:56.789Z",
"datacontenttype": "application/json",
"data": {
"event_type": "AFTER_CREATE_TABLE",
"event_id": "550e8400-e29b-41d4-a716-446655440000",
"timestamp": "2026-10-04T12:34:56.789Z",
"realm_id": "default-realm",
"principal": "alice",
"activated_roles": ["catalog_admin"],
"request_id": "req-123",
"catalog_name": "sales",
"namespace": ["analytics"],
"table_name": "orders",
"table_identifier": ["analytics", "orders"]
}
}
A couple notes on purpose:
- "data" is basically today's Kafka body, so id/time show up both as CE
context and again inside data (event_id / timestamp) for Kafka parity
- type is a namespaced PolarisEventType name; source is a constant for
now - will change if needed..
- webhook would put delivery time on an HTTP header
(e.g. X-Polaris-Delivery-Time), not in the envelope
if this is roughly the right shape, I'll draft the prerequisite PR around
it. if you'd rather nail type/source naming first, tell me which forms
you prefer.
Thanks,
*Vignesh*
[1] https://github.com/apache/polaris/pull/5084
On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi Vignesh
>
> Thanks for starting this thread.
>
> I'm +1 on the sequence: shared envelope first and then rebase the webhook
> PR. I slightly prefer your proposal over B or C.
>
> Some general comments:
> 1. CloudEvents requires attribute names to be lowercase alphanumerics and
> attribute values to be scalars afair. So a flat set of extensions doesn't
> really work for us: activatedroles as an array is not valid for
> instance. Or we have to transform somehow. I would keep only the
> CloudEvents context attributes at the top level and put the Polaris fields
> in "data".
> 2. The EventAttributes keys already have underscored names (catalog_name,
> table_identifier, ...) and they match what Kafka emits. I would reuse them
> as is. Then the current Kafka body basically becomes "data", which also
> makes the migration easier.
> 3. I would not start from the thin set: migrating Kafka to it would drop
> fields that consumers already get (namespace, catalog, view, ..). Kafka
> only has nine optional fields, so I think v1 should start at Kafka parity.
> 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I think we
> should agree on the mechanism here on the list rather than in the PR
> description (if it doesn't happen on the mailing list, it never happens
> :)). I would prefer a per-listener format option, with the current format
> as default, over dual-publish (which doubles the volume on the topic).
> 5. The events SPI and the CloudWatch listener are both in runtime/service,
> and the Kafka extension depends on it. A new module would introduce a cycle
> unless we extract the SPI first, so I would put the mapper next to the SPI
> in runtime/service for now, and move it when the SPI is moved.
> 6. deliverytime is a transport metadata, I would move it to a HTTP header
> instead of the shared envelope. We should also decide what "type" and
> "source" look like (today it's a bare enum name and a constant).
>
> Could you share the v1 JSON you have in mind on this thread before opening
> the PR? I think it will be easier to converge on an example.
>
> The OpenTelemetry and persistence listeners have their own representations
> as well. I'm fine keeping them out of scope, but it's worth stating it
> explicitly.
>
> Regards
> JB
>
> On Sun, Oct 4, 2026 at 11:47 AM vignesh a <im...@gmail.com> wrote:
>
> > \Hi all,
> >
> > Robert and EJ have both asked that we settle a shared event envelope
> > before merging the webhook listener [1]. I agree with that sequencing
> > and would like a quick list consensus on shape and placement before I
> > open the prerequisite PR.
> >
> > What each path emits today (from current main / [1]):
> >
> > - Kafka (extensions/events/kafka/.../EventToJsonMapper):
> > flat snake_case: event_type, event_id, timestamp, realm_id, principal,
> > activated_roles, request_id, plus a fairly rich attribute set
> > (table_identifier as a string list, namespace/catalog/view fields,
> > rename source/destination, and similar).
> >
> > - Webhook ([1], WebhookEventListener.toJson):
> > CloudEvents-shaped: specversion, id, type, source, time, plus
> > extensions deliverytime, realmid, tableidentifier (single string),
> > principal, activatedroles, requestid. Thinner attribute coverage than
> > Kafka.
> >
> > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
> > flat map: event_type, optional table_identifier (string), realm_id,
> > principal, activated_roles, request_id. No event_id in the JSON body;
> > timestamp is on InputLogEvent.
> >
> > So this is not a mechanical "extract EventToJsonMapper." Naming,
> > structure, identifier encoding, and attribute breadth already diverge,
> > and the shared code should not live inside the kafka extension.
> >
> > Proposal:
> >
> > 1) Land a small prerequisite that introduces a shared, versioned,
> > CloudEvents-shaped mapper (core fields + a documented extension set),
> > placed outside any one transport extension.
> >
> > 2) Rebase [1] onto that shared mapper.
> >
> > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
> > compatibility plan so existing consumers are not silently broken
> > (version field and/or dual-publish window; exact mechanism in the
> > prerequisite PR description).
> >
> > 4) Attribute breadth for v1: start from the webhook/CloudWatch-thin
> > common set; add Kafka's richer attributes behind the same versioned
> > envelope rather than blocking v1 on full Kafka parity.
> >
> > If that direction is wrong, the useful alternatives seem to be:
> >
> > - B) Prefer Kafka's flat snake_case as the shared shape instead of
> > CloudEvents-shaped.
> > - C) Fold the shared mapper into [1] itself (larger single PR).
> >
> > Unless someone prefers B or C, I will open the prerequisite PR along the
> > lines of (1)-(4) early next week.
> >
> > Thanks,
> > Vignesh
> >
> > [1] https://github.com/apache/polaris/pull/5084
> >
>
Re: [DISCUSS] Shared event JSON envelope vs webhook listener (#5084)
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Vignesh
Thanks for starting this thread.
I'm +1 on the sequence: shared envelope first and then rebase the webhook
PR. I slightly prefer your proposal over B or C.
Some general comments:
1. CloudEvents requires attribute names to be lowercase alphanumerics and
attribute values to be scalars afair. So a flat set of extensions doesn't
really work for us: activatedroles as an array is not valid for
instance. Or we have to transform somehow. I would keep only the
CloudEvents context attributes at the top level and put the Polaris fields
in "data".
2. The EventAttributes keys already have underscored names (catalog_name,
table_identifier, ...) and they match what Kafka emits. I would reuse them
as is. Then the current Kafka body basically becomes "data", which also
makes the migration easier.
3. I would not start from the thin set: migrating Kafka to it would drop
fields that consumers already get (namespace, catalog, view, ..). Kafka
only has nine optional fields, so I think v1 should start at Kafka parity.
4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I think we
should agree on the mechanism here on the list rather than in the PR
description (if it doesn't happen on the mailing list, it never happens
:)). I would prefer a per-listener format option, with the current format
as default, over dual-publish (which doubles the volume on the topic).
5. The events SPI and the CloudWatch listener are both in runtime/service,
and the Kafka extension depends on it. A new module would introduce a cycle
unless we extract the SPI first, so I would put the mapper next to the SPI
in runtime/service for now, and move it when the SPI is moved.
6. deliverytime is a transport metadata, I would move it to a HTTP header
instead of the shared envelope. We should also decide what "type" and
"source" look like (today it's a bare enum name and a constant).
Could you share the v1 JSON you have in mind on this thread before opening
the PR? I think it will be easier to converge on an example.
The OpenTelemetry and persistence listeners have their own representations
as well. I'm fine keeping them out of scope, but it's worth stating it
explicitly.
Regards
JB
On Sun, Oct 4, 2026 at 11:47 AM vignesh a <im...@gmail.com> wrote:
> \Hi all,
>
> Robert and EJ have both asked that we settle a shared event envelope
> before merging the webhook listener [1]. I agree with that sequencing
> and would like a quick list consensus on shape and placement before I
> open the prerequisite PR.
>
> What each path emits today (from current main / [1]):
>
> - Kafka (extensions/events/kafka/.../EventToJsonMapper):
> flat snake_case: event_type, event_id, timestamp, realm_id, principal,
> activated_roles, request_id, plus a fairly rich attribute set
> (table_identifier as a string list, namespace/catalog/view fields,
> rename source/destination, and similar).
>
> - Webhook ([1], WebhookEventListener.toJson):
> CloudEvents-shaped: specversion, id, type, source, time, plus
> extensions deliverytime, realmid, tableidentifier (single string),
> principal, activatedroles, requestid. Thinner attribute coverage than
> Kafka.
>
> - CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
> flat map: event_type, optional table_identifier (string), realm_id,
> principal, activated_roles, request_id. No event_id in the JSON body;
> timestamp is on InputLogEvent.
>
> So this is not a mechanical "extract EventToJsonMapper." Naming,
> structure, identifier encoding, and attribute breadth already diverge,
> and the shared code should not live inside the kafka extension.
>
> Proposal:
>
> 1) Land a small prerequisite that introduces a shared, versioned,
> CloudEvents-shaped mapper (core fields + a documented extension set),
> placed outside any one transport extension.
>
> 2) Rebase [1] onto that shared mapper.
>
> 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
> compatibility plan so existing consumers are not silently broken
> (version field and/or dual-publish window; exact mechanism in the
> prerequisite PR description).
>
> 4) Attribute breadth for v1: start from the webhook/CloudWatch-thin
> common set; add Kafka's richer attributes behind the same versioned
> envelope rather than blocking v1 on full Kafka parity.
>
> If that direction is wrong, the useful alternatives seem to be:
>
> - B) Prefer Kafka's flat snake_case as the shared shape instead of
> CloudEvents-shaped.
> - C) Fold the shared mapper into [1] itself (larger single PR).
>
> Unless someone prefers B or C, I will open the prerequisite PR along the
> lines of (1)-(4) early next week.
>
> Thanks,
> Vignesh
>
> [1] https://github.com/apache/polaris/pull/5084
>