You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by mattison chao <ma...@gmail.com> on 2022/01/05 15:27:33 UTC
[DISCUSSION] PIP-132: Support Pulsar system event.
Original PIP : https://github.com/apache/pulsar/issues/13628
Pasted bellow for quoting convenience.
——
## Motivation
In some case or for better observability. We can support Pulsar event notification mechanism, that can let user subscribe this system topic to receive some **"we need notice to user"** event.
## Goal
Enable users to receive Pulsar system event.
## API Changes
1. Add config ``pulsarSystemEventEnabled``
```java
pulsarSystemEventEnabled=false
```
2. Add ``SystemEventMessageBuilder``
```java
public interface SystemEventMessageBuilder {
SystemEventMessageBuilder eventType(SystemEvent event);
SystemEventMessageBuilder properties(Map<String,Object> properties);
SystemEventMessageBuilder addProperty(String key, Object value);
SystemEventMessage build();
}
```
3. Add ``SystemEventService``
```java
public interface SystemEventService {
void init();
CompletableFuture<MessageId> trigger(SystemEventMessage message);
void close();
}
```
## Implementation
This PIP just is a simple version, it's purpose is to discuss what event we need to trigger? how to trigger? and what is the event message structure.
I think we need to trigger some event as bellow:
- [ ] Client connected
- [ ] Client disconnected
- [ ] Consumer subscribe
- [ ] Consumer unsubscribe
- [ ] Producer publish
- [ ] Message delivered
- [ ] Message acked
These are far from enough, and we need to continue to make supplements here.
Please fell free to give me more advice for this PIP.
## Reject Alternatives
none.
Thanks,
Mattisonchao
Re: [DISCUSSION] PIP-132: Support Pulsar system event.
Posted by mattison chao <ma...@gmail.com>.
Hi, Dave Fisher
> These bother me too. Producer and consumers can accomplish the same use cases in their application logic.
>
> Any notification mechanism must be very lightweight. What is the implantation plan?
Agree with you.
I think we just need to record some system internal event that need human process.
> Also if this pip is for user use cases then there needs to be isolation at the tenant level.
I think this pip just for middleware team to handle the pulsar internal event that need human process.
E.g. Due to the network issue or distributed issue we can't to delete ledger.
Therefore, whether the global system topic is enough?
Thanks,
Mattisonchao
> On Jan 6, 2022, at 12:35 PM, Dave Fisher <wa...@comcast.net> wrote:
>
>
>
> Sent from my iPhone
>
>> On Jan 5, 2022, at 8:10 PM, Haiting Jiang <ji...@apache.org> wrote:
>>
>>
>>> CompletableFuture<MessageId> trigger(SystemEventMessage message);
>>
>> Maybe `record` make more sense?
>>
>>> SystemEventMessageBuilder properties(Map<String,Object> properties);
>>>
>>> SystemEventMessageBuilder addProperty(String key, Object value);
>>
>> How to handle the serialization and deserialization if the value type is Object?
>>
>>> - [ ] Producer publish
>>> - [ ] Message delivered
>>> - [ ] Message acked
>>
>> I think we don't need to record normal message producing and consuming event?
>> That would make the system topic store too much data.
>
> These bother me too. Producer and consumers can accomplish the same use cases in their application logic.
>
> Any notification mechanism must be very lightweight. What is the implantation plan?
>
> Also if this pip is for user use cases then there needs to be isolation at the tenant level.
>
> Thanks for the proposal.
>
> Regards,
> Dave
>
>>
>>
>>> On 2022/01/05 15:27:33 mattison chao wrote:
>>> Original PIP : https://github.com/apache/pulsar/issues/13628
>>>
>>> Pasted bellow for quoting convenience.
>>>
>>> ——
>>>
>>> ## Motivation
>>>
>>> In some case or for better observability. We can support Pulsar event notification mechanism, that can let user subscribe this system topic to receive some **"we need notice to user"** event.
>>>
>>> ## Goal
>>>
>>> Enable users to receive Pulsar system event.
>>>
>>> ## API Changes
>>>
>>> 1. Add config ``pulsarSystemEventEnabled``
>>>
>>> ```java
>>> pulsarSystemEventEnabled=false
>>> ```
>>> 2. Add ``SystemEventMessageBuilder``
>>> ```java
>>>
>>>
>>> public interface SystemEventMessageBuilder {
>>>
>>> SystemEventMessageBuilder eventType(SystemEvent event);
>>>
>>> SystemEventMessageBuilder properties(Map<String,Object> properties);
>>>
>>> SystemEventMessageBuilder addProperty(String key, Object value);
>>>
>>> SystemEventMessage build();
>>>
>>> }
>>>
>>> ```
>>> 3. Add ``SystemEventService``
>>>
>>> ```java
>>>
>>> public interface SystemEventService {
>>>
>>> void init();
>>>
>>> CompletableFuture<MessageId> trigger(SystemEventMessage message);
>>>
>>> void close();
>>> }
>>>
>>> ```
>>>
>>> ## Implementation
>>>
>>> This PIP just is a simple version, it's purpose is to discuss what event we need to trigger? how to trigger? and what is the event message structure.
>>>
>>> I think we need to trigger some event as bellow:
>>>
>>> - [ ] Client connected
>>> - [ ] Client disconnected
>>> - [ ] Consumer subscribe
>>> - [ ] Consumer unsubscribe
>>> - [ ] Producer publish
>>> - [ ] Message delivered
>>> - [ ] Message acked
>>>
>>> These are far from enough, and we need to continue to make supplements here.
>>> Please fell free to give me more advice for this PIP.
>>>
>>> ## Reject Alternatives
>>>
>>> none.
>>>
>>> Thanks,
>>> Mattisonchao
>>
>> Thanks,
>> Haiting Jiang
Re: [DISCUSSION] PIP-132: Support Pulsar system event.
Posted by Dave Fisher <wa...@comcast.net>.
Sent from my iPhone
> On Jan 5, 2022, at 8:10 PM, Haiting Jiang <ji...@apache.org> wrote:
>
>
>> CompletableFuture<MessageId> trigger(SystemEventMessage message);
>
> Maybe `record` make more sense?
>
>> SystemEventMessageBuilder properties(Map<String,Object> properties);
>>
>> SystemEventMessageBuilder addProperty(String key, Object value);
>
> How to handle the serialization and deserialization if the value type is Object?
>
>> - [ ] Producer publish
>> - [ ] Message delivered
>> - [ ] Message acked
>
> I think we don't need to record normal message producing and consuming event?
> That would make the system topic store too much data.
These bother me too. Producer and consumers can accomplish the same use cases in their application logic.
Any notification mechanism must be very lightweight. What is the implantation plan?
Also if this pip is for user use cases then there needs to be isolation at the tenant level.
Thanks for the proposal.
Regards,
Dave
>
>
>> On 2022/01/05 15:27:33 mattison chao wrote:
>> Original PIP : https://github.com/apache/pulsar/issues/13628
>>
>> Pasted bellow for quoting convenience.
>>
>> ——
>>
>> ## Motivation
>>
>> In some case or for better observability. We can support Pulsar event notification mechanism, that can let user subscribe this system topic to receive some **"we need notice to user"** event.
>>
>> ## Goal
>>
>> Enable users to receive Pulsar system event.
>>
>> ## API Changes
>>
>> 1. Add config ``pulsarSystemEventEnabled``
>>
>> ```java
>> pulsarSystemEventEnabled=false
>> ```
>> 2. Add ``SystemEventMessageBuilder``
>> ```java
>>
>>
>> public interface SystemEventMessageBuilder {
>>
>> SystemEventMessageBuilder eventType(SystemEvent event);
>>
>> SystemEventMessageBuilder properties(Map<String,Object> properties);
>>
>> SystemEventMessageBuilder addProperty(String key, Object value);
>>
>> SystemEventMessage build();
>>
>> }
>>
>> ```
>> 3. Add ``SystemEventService``
>>
>> ```java
>>
>> public interface SystemEventService {
>>
>> void init();
>>
>> CompletableFuture<MessageId> trigger(SystemEventMessage message);
>>
>> void close();
>> }
>>
>> ```
>>
>> ## Implementation
>>
>> This PIP just is a simple version, it's purpose is to discuss what event we need to trigger? how to trigger? and what is the event message structure.
>>
>> I think we need to trigger some event as bellow:
>>
>> - [ ] Client connected
>> - [ ] Client disconnected
>> - [ ] Consumer subscribe
>> - [ ] Consumer unsubscribe
>> - [ ] Producer publish
>> - [ ] Message delivered
>> - [ ] Message acked
>>
>> These are far from enough, and we need to continue to make supplements here.
>> Please fell free to give me more advice for this PIP.
>>
>> ## Reject Alternatives
>>
>> none.
>>
>> Thanks,
>> Mattisonchao
>
> Thanks,
> Haiting Jiang
Re: [DISCUSSION] PIP-132: Support Pulsar system event.
Posted by PengHui Li <pe...@apache.org>.
We have a pluggable broker interceptor,
> I think we need to trigger some event as bellow:
- [ ] Client connected
- [ ] Client disconnected
- [ ] Consumer subscribe
- [ ] Consumer unsubscribe
- [ ] Producer publish
- [ ] Message delivered
- [ ] Message acked
For these events, the broker interceptor can cover all of these events,
users can easily implement their own interceptor to intercept all the
events from the client.
> In some case or for better observability. We can support Pulsar event
notification mechanism, that can let user subscribe this system topic to
receive some **"we need notice to user"** event.
From the perspective of observability, for example, the load manager, we
are not easy to know why the load manager
splits the bundles or unloads the bundles. Or auto deletes the inactive
topics, we might better focus on the
observability of the broker‘s decisions.
Regards,
Penghui
On Fri, Jan 7, 2022 at 2:28 PM Michael Marshall <mm...@apache.org>
wrote:
> This is a great topic, thanks for starting the discussion Mattison.
>
> The PIP only mentions topic events right now--I think we should
> include (or plan to include) metadata related events, too.
>
> I proposed a similar feature in April 2021. There was good discussion
> then that is relevant now [0].
>
> I think a system or audit topic (more likely a collection of topics)
> would be extremely valuable. However, last year's discussion raised
> several important concerns that PIP 132 does not yet address.
>
> First, we need to ensure this feature is not a single point of failure
> for the rest of the cluster. That requires defining how we will handle
> failures when publishing audit messages. Joe Francis had several
> good notes on this topic [0]. Also, the feature must be scalable: it
> should work with 100+ brokers and 1,000,000+ topics.
>
> > Also if this pip is for user use cases then there needs to be isolation
> at the tenant level.
>
> Great point. Further, if these events are also meant for system admins
> (super users), we'll need to prevent tenant admins from hiding events
> or creating fake events (by deleting system topics/subscriptions or
> publishing to system event topics). That will require careful attention to
> authorization.
>
> The question of tenant isolation presupposes event classification.
> Namespace and topic level events obviously belong to a tenant, so
> the event topic for those events could live in a tenant namespace.
> Tenant level events like tenant creation/deletion or changes in broker
> state would likely require a "system event tenant" to host this tenant
> agnostic information.
>
> Alternatively, we could use a system event tenant for all system events.
> The
> namespaces could map to tenant names with an extra namespace for tenant
> agnostic events. We might need to use topic level policies to achieve
> proper
> tenant isolation. Personally, I think a dedicated tenant might be the best
> solution, but I need to think about it more.
>
> If I understand it correctly, PIP 45 opens the path for us to move towards
> a
> zookeeper-less metadata solution. If that solution relies on Pulsar topics
> for metadata, would we get a kind of system event topic for free? We
> should make sure we're not duplicating efforts.
>
> This is a great feature, and I'd like to make sure it is robust and well
> defined
> before we start its implementation.
>
> Thanks,
> Michael
>
> [0] https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
>
>
>
>
>
>
>
> On Thu, Jan 6, 2022 at 3:42 AM Haiting Jiang <ji...@apache.org>
> wrote:
> >
> > Hi mattison,
> >
> > > For this part, I think we need to discuss. Do we create a specific
> event class or use properties in json format?
> >
> > I think String value would be enough for most cases.
> > IMHO, these system event is mostly for administrators, so the data
> should be readable.
> >
> >
> > > Maybe the better way is to record some system internal event like node
> changed, ledger deleted etc.
> >
> > This PIP can start with a small scope of events that requires the
> attention of administrators, like ledger delete failure, zk session
> timeout, etc.
> >
> > Thanks,
> > Haiting Jiang
> >
> >
> > On 2022/01/06 05:23:52 mattison chao wrote:
> > > Hi, Haiting Jiang
> > >
> > >
> > > >> CompletableFuture<MessageId> trigger(SystemEventMessage message);
> > > >
> > > > Maybe `record` make more sense?
> > >
> > > +1
> > >
> > > >> SystemEventMessageBuilder properties(Map<String,Object>
> properties);
> > > >>
> > > >> SystemEventMessageBuilder addProperty(String key, Object value);
> > > >
> > > > How to handle the serialization and deserialization if the value
> type is Object?
> > >
> > > For this part, I think we need to discuss. Do we create a specific
> event class or use properties in json format?
> > >
> > >
> > > > I think we don't need to record normal message producing and
> consuming event?
> > > > That would make the system topic store too much data.
> > >
> > >
> > > Agree with you.
> > >
> > > Maybe the better way is to record some system internal event like node
> changed, ledger deleted etc.
> > >
> > >
> > >
> > >
> > >
> > >
> > > > On Jan 6, 2022, at 12:10 PM, Haiting Jiang <jianghaiting@apache.org
> <ma...@apache.org>> wrote:
> > > >
> > > >
> > > >> CompletableFuture<MessageId> trigger(SystemEventMessage message);
> > > >
> > > > Maybe `record` make more sense?
> > > >
> > > >> SystemEventMessageBuilder properties(Map<String,Object>
> properties);
> > > >>
> > > >> SystemEventMessageBuilder addProperty(String key, Object value);
> > > >
> > > > How to handle the serialization and deserialization if the value
> type is Object?
> > > >
> > > >> - [ ] Producer publish
> > > >> - [ ] Message delivered
> > > >> - [ ] Message acked
> > > >
> > > > I think we don't need to record normal message producing and
> consuming event?
> > > > That would make the system topic store too much data.
> > > >
> > > >
> > > > On 2022/01/05 15:27:33 mattison chao wrote:
> > > >> Original PIP : https://github.com/apache/pulsar/issues/13628 <
> https://github.com/apache/pulsar/issues/13628>
> > > >>
> > > >> Pasted bellow for quoting convenience.
> > > >>
> > > >> ——
> > > >>
> > > >> ## Motivation
> > > >>
> > > >> In some case or for better observability. We can support Pulsar
> event notification mechanism, that can let user subscribe this system topic
> to receive some **"we need notice to user"** event.
> > > >>
> > > >> ## Goal
> > > >>
> > > >> Enable users to receive Pulsar system event.
> > > >>
> > > >> ## API Changes
> > > >>
> > > >> 1. Add config ``pulsarSystemEventEnabled``
> > > >>
> > > >> ```java
> > > >> pulsarSystemEventEnabled=false
> > > >> ```
> > > >> 2. Add ``SystemEventMessageBuilder``
> > > >> ```java
> > > >>
> > > >>
> > > >> public interface SystemEventMessageBuilder {
> > > >>
> > > >> SystemEventMessageBuilder eventType(SystemEvent event);
> > > >>
> > > >> SystemEventMessageBuilder properties(Map<String,Object>
> properties);
> > > >>
> > > >> SystemEventMessageBuilder addProperty(String key, Object value);
> > > >>
> > > >> SystemEventMessage build();
> > > >>
> > > >> }
> > > >>
> > > >> ```
> > > >> 3. Add ``SystemEventService``
> > > >>
> > > >> ```java
> > > >>
> > > >> public interface SystemEventService {
> > > >>
> > > >> void init();
> > > >>
> > > >> CompletableFuture<MessageId> trigger(SystemEventMessage message);
> > > >>
> > > >> void close();
> > > >> }
> > > >>
> > > >> ```
> > > >>
> > > >> ## Implementation
> > > >>
> > > >> This PIP just is a simple version, it's purpose is to discuss what
> event we need to trigger? how to trigger? and what is the event message
> structure.
> > > >>
> > > >> I think we need to trigger some event as bellow:
> > > >>
> > > >> - [ ] Client connected
> > > >> - [ ] Client disconnected
> > > >> - [ ] Consumer subscribe
> > > >> - [ ] Consumer unsubscribe
> > > >> - [ ] Producer publish
> > > >> - [ ] Message delivered
> > > >> - [ ] Message acked
> > > >>
> > > >> These are far from enough, and we need to continue to make
> supplements here.
> > > >> Please fell free to give me more advice for this PIP.
> > > >>
> > > >> ## Reject Alternatives
> > > >>
> > > >> none.
> > > >>
> > > >> Thanks,
> > > >> Mattisonchao
> > > >
> > > > Thanks,
> > > > Haiting Jiang
> > >
> > >
>
Re: [DISCUSSION] PIP-132: Support Pulsar system event.
Posted by Michael Marshall <mm...@apache.org>.
This is a great topic, thanks for starting the discussion Mattison.
The PIP only mentions topic events right now--I think we should
include (or plan to include) metadata related events, too.
I proposed a similar feature in April 2021. There was good discussion
then that is relevant now [0].
I think a system or audit topic (more likely a collection of topics)
would be extremely valuable. However, last year's discussion raised
several important concerns that PIP 132 does not yet address.
First, we need to ensure this feature is not a single point of failure
for the rest of the cluster. That requires defining how we will handle
failures when publishing audit messages. Joe Francis had several
good notes on this topic [0]. Also, the feature must be scalable: it
should work with 100+ brokers and 1,000,000+ topics.
> Also if this pip is for user use cases then there needs to be isolation at the tenant level.
Great point. Further, if these events are also meant for system admins
(super users), we'll need to prevent tenant admins from hiding events
or creating fake events (by deleting system topics/subscriptions or
publishing to system event topics). That will require careful attention to
authorization.
The question of tenant isolation presupposes event classification.
Namespace and topic level events obviously belong to a tenant, so
the event topic for those events could live in a tenant namespace.
Tenant level events like tenant creation/deletion or changes in broker
state would likely require a "system event tenant" to host this tenant
agnostic information.
Alternatively, we could use a system event tenant for all system events. The
namespaces could map to tenant names with an extra namespace for tenant
agnostic events. We might need to use topic level policies to achieve proper
tenant isolation. Personally, I think a dedicated tenant might be the best
solution, but I need to think about it more.
If I understand it correctly, PIP 45 opens the path for us to move towards a
zookeeper-less metadata solution. If that solution relies on Pulsar topics
for metadata, would we get a kind of system event topic for free? We
should make sure we're not duplicating efforts.
This is a great feature, and I'd like to make sure it is robust and well defined
before we start its implementation.
Thanks,
Michael
[0] https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
On Thu, Jan 6, 2022 at 3:42 AM Haiting Jiang <ji...@apache.org> wrote:
>
> Hi mattison,
>
> > For this part, I think we need to discuss. Do we create a specific event class or use properties in json format?
>
> I think String value would be enough for most cases.
> IMHO, these system event is mostly for administrators, so the data should be readable.
>
>
> > Maybe the better way is to record some system internal event like node changed, ledger deleted etc.
>
> This PIP can start with a small scope of events that requires the attention of administrators, like ledger delete failure, zk session timeout, etc.
>
> Thanks,
> Haiting Jiang
>
>
> On 2022/01/06 05:23:52 mattison chao wrote:
> > Hi, Haiting Jiang
> >
> >
> > >> CompletableFuture<MessageId> trigger(SystemEventMessage message);
> > >
> > > Maybe `record` make more sense?
> >
> > +1
> >
> > >> SystemEventMessageBuilder properties(Map<String,Object> properties);
> > >>
> > >> SystemEventMessageBuilder addProperty(String key, Object value);
> > >
> > > How to handle the serialization and deserialization if the value type is Object?
> >
> > For this part, I think we need to discuss. Do we create a specific event class or use properties in json format?
> >
> >
> > > I think we don't need to record normal message producing and consuming event?
> > > That would make the system topic store too much data.
> >
> >
> > Agree with you.
> >
> > Maybe the better way is to record some system internal event like node changed, ledger deleted etc.
> >
> >
> >
> >
> >
> >
> > > On Jan 6, 2022, at 12:10 PM, Haiting Jiang <jianghaiting@apache.org <ma...@apache.org>> wrote:
> > >
> > >
> > >> CompletableFuture<MessageId> trigger(SystemEventMessage message);
> > >
> > > Maybe `record` make more sense?
> > >
> > >> SystemEventMessageBuilder properties(Map<String,Object> properties);
> > >>
> > >> SystemEventMessageBuilder addProperty(String key, Object value);
> > >
> > > How to handle the serialization and deserialization if the value type is Object?
> > >
> > >> - [ ] Producer publish
> > >> - [ ] Message delivered
> > >> - [ ] Message acked
> > >
> > > I think we don't need to record normal message producing and consuming event?
> > > That would make the system topic store too much data.
> > >
> > >
> > > On 2022/01/05 15:27:33 mattison chao wrote:
> > >> Original PIP : https://github.com/apache/pulsar/issues/13628 <https://github.com/apache/pulsar/issues/13628>
> > >>
> > >> Pasted bellow for quoting convenience.
> > >>
> > >> ——
> > >>
> > >> ## Motivation
> > >>
> > >> In some case or for better observability. We can support Pulsar event notification mechanism, that can let user subscribe this system topic to receive some **"we need notice to user"** event.
> > >>
> > >> ## Goal
> > >>
> > >> Enable users to receive Pulsar system event.
> > >>
> > >> ## API Changes
> > >>
> > >> 1. Add config ``pulsarSystemEventEnabled``
> > >>
> > >> ```java
> > >> pulsarSystemEventEnabled=false
> > >> ```
> > >> 2. Add ``SystemEventMessageBuilder``
> > >> ```java
> > >>
> > >>
> > >> public interface SystemEventMessageBuilder {
> > >>
> > >> SystemEventMessageBuilder eventType(SystemEvent event);
> > >>
> > >> SystemEventMessageBuilder properties(Map<String,Object> properties);
> > >>
> > >> SystemEventMessageBuilder addProperty(String key, Object value);
> > >>
> > >> SystemEventMessage build();
> > >>
> > >> }
> > >>
> > >> ```
> > >> 3. Add ``SystemEventService``
> > >>
> > >> ```java
> > >>
> > >> public interface SystemEventService {
> > >>
> > >> void init();
> > >>
> > >> CompletableFuture<MessageId> trigger(SystemEventMessage message);
> > >>
> > >> void close();
> > >> }
> > >>
> > >> ```
> > >>
> > >> ## Implementation
> > >>
> > >> This PIP just is a simple version, it's purpose is to discuss what event we need to trigger? how to trigger? and what is the event message structure.
> > >>
> > >> I think we need to trigger some event as bellow:
> > >>
> > >> - [ ] Client connected
> > >> - [ ] Client disconnected
> > >> - [ ] Consumer subscribe
> > >> - [ ] Consumer unsubscribe
> > >> - [ ] Producer publish
> > >> - [ ] Message delivered
> > >> - [ ] Message acked
> > >>
> > >> These are far from enough, and we need to continue to make supplements here.
> > >> Please fell free to give me more advice for this PIP.
> > >>
> > >> ## Reject Alternatives
> > >>
> > >> none.
> > >>
> > >> Thanks,
> > >> Mattisonchao
> > >
> > > Thanks,
> > > Haiting Jiang
> >
> >
Re: [DISCUSSION] PIP-132: Support Pulsar system event.
Posted by Haiting Jiang <ji...@apache.org>.
Hi mattison,
> For this part, I think we need to discuss. Do we create a specific event class or use properties in json format?
I think String value would be enough for most cases.
IMHO, these system event is mostly for administrators, so the data should be readable.
> Maybe the better way is to record some system internal event like node changed, ledger deleted etc.
This PIP can start with a small scope of events that requires the attention of administrators, like ledger delete failure, zk session timeout, etc.
Thanks,
Haiting Jiang
On 2022/01/06 05:23:52 mattison chao wrote:
> Hi, Haiting Jiang
>
>
> >> CompletableFuture<MessageId> trigger(SystemEventMessage message);
> >
> > Maybe `record` make more sense?
>
> +1
>
> >> SystemEventMessageBuilder properties(Map<String,Object> properties);
> >>
> >> SystemEventMessageBuilder addProperty(String key, Object value);
> >
> > How to handle the serialization and deserialization if the value type is Object?
>
> For this part, I think we need to discuss. Do we create a specific event class or use properties in json format?
>
>
> > I think we don't need to record normal message producing and consuming event?
> > That would make the system topic store too much data.
>
>
> Agree with you.
>
> Maybe the better way is to record some system internal event like node changed, ledger deleted etc.
>
>
>
>
>
>
> > On Jan 6, 2022, at 12:10 PM, Haiting Jiang <jianghaiting@apache.org <ma...@apache.org>> wrote:
> >
> >
> >> CompletableFuture<MessageId> trigger(SystemEventMessage message);
> >
> > Maybe `record` make more sense?
> >
> >> SystemEventMessageBuilder properties(Map<String,Object> properties);
> >>
> >> SystemEventMessageBuilder addProperty(String key, Object value);
> >
> > How to handle the serialization and deserialization if the value type is Object?
> >
> >> - [ ] Producer publish
> >> - [ ] Message delivered
> >> - [ ] Message acked
> >
> > I think we don't need to record normal message producing and consuming event?
> > That would make the system topic store too much data.
> >
> >
> > On 2022/01/05 15:27:33 mattison chao wrote:
> >> Original PIP : https://github.com/apache/pulsar/issues/13628 <https://github.com/apache/pulsar/issues/13628>
> >>
> >> Pasted bellow for quoting convenience.
> >>
> >> ——
> >>
> >> ## Motivation
> >>
> >> In some case or for better observability. We can support Pulsar event notification mechanism, that can let user subscribe this system topic to receive some **"we need notice to user"** event.
> >>
> >> ## Goal
> >>
> >> Enable users to receive Pulsar system event.
> >>
> >> ## API Changes
> >>
> >> 1. Add config ``pulsarSystemEventEnabled``
> >>
> >> ```java
> >> pulsarSystemEventEnabled=false
> >> ```
> >> 2. Add ``SystemEventMessageBuilder``
> >> ```java
> >>
> >>
> >> public interface SystemEventMessageBuilder {
> >>
> >> SystemEventMessageBuilder eventType(SystemEvent event);
> >>
> >> SystemEventMessageBuilder properties(Map<String,Object> properties);
> >>
> >> SystemEventMessageBuilder addProperty(String key, Object value);
> >>
> >> SystemEventMessage build();
> >>
> >> }
> >>
> >> ```
> >> 3. Add ``SystemEventService``
> >>
> >> ```java
> >>
> >> public interface SystemEventService {
> >>
> >> void init();
> >>
> >> CompletableFuture<MessageId> trigger(SystemEventMessage message);
> >>
> >> void close();
> >> }
> >>
> >> ```
> >>
> >> ## Implementation
> >>
> >> This PIP just is a simple version, it's purpose is to discuss what event we need to trigger? how to trigger? and what is the event message structure.
> >>
> >> I think we need to trigger some event as bellow:
> >>
> >> - [ ] Client connected
> >> - [ ] Client disconnected
> >> - [ ] Consumer subscribe
> >> - [ ] Consumer unsubscribe
> >> - [ ] Producer publish
> >> - [ ] Message delivered
> >> - [ ] Message acked
> >>
> >> These are far from enough, and we need to continue to make supplements here.
> >> Please fell free to give me more advice for this PIP.
> >>
> >> ## Reject Alternatives
> >>
> >> none.
> >>
> >> Thanks,
> >> Mattisonchao
> >
> > Thanks,
> > Haiting Jiang
>
>
Re: [DISCUSSION] PIP-132: Support Pulsar system event.
Posted by mattison chao <ma...@gmail.com>.
Hi, Haiting Jiang
>> CompletableFuture<MessageId> trigger(SystemEventMessage message);
>
> Maybe `record` make more sense?
+1
>> SystemEventMessageBuilder properties(Map<String,Object> properties);
>>
>> SystemEventMessageBuilder addProperty(String key, Object value);
>
> How to handle the serialization and deserialization if the value type is Object?
For this part, I think we need to discuss. Do we create a specific event class or use properties in json format?
> I think we don't need to record normal message producing and consuming event?
> That would make the system topic store too much data.
Agree with you.
Maybe the better way is to record some system internal event like node changed, ledger deleted etc.
> On Jan 6, 2022, at 12:10 PM, Haiting Jiang <jianghaiting@apache.org <ma...@apache.org>> wrote:
>
>
>> CompletableFuture<MessageId> trigger(SystemEventMessage message);
>
> Maybe `record` make more sense?
>
>> SystemEventMessageBuilder properties(Map<String,Object> properties);
>>
>> SystemEventMessageBuilder addProperty(String key, Object value);
>
> How to handle the serialization and deserialization if the value type is Object?
>
>> - [ ] Producer publish
>> - [ ] Message delivered
>> - [ ] Message acked
>
> I think we don't need to record normal message producing and consuming event?
> That would make the system topic store too much data.
>
>
> On 2022/01/05 15:27:33 mattison chao wrote:
>> Original PIP : https://github.com/apache/pulsar/issues/13628 <https://github.com/apache/pulsar/issues/13628>
>>
>> Pasted bellow for quoting convenience.
>>
>> ——
>>
>> ## Motivation
>>
>> In some case or for better observability. We can support Pulsar event notification mechanism, that can let user subscribe this system topic to receive some **"we need notice to user"** event.
>>
>> ## Goal
>>
>> Enable users to receive Pulsar system event.
>>
>> ## API Changes
>>
>> 1. Add config ``pulsarSystemEventEnabled``
>>
>> ```java
>> pulsarSystemEventEnabled=false
>> ```
>> 2. Add ``SystemEventMessageBuilder``
>> ```java
>>
>>
>> public interface SystemEventMessageBuilder {
>>
>> SystemEventMessageBuilder eventType(SystemEvent event);
>>
>> SystemEventMessageBuilder properties(Map<String,Object> properties);
>>
>> SystemEventMessageBuilder addProperty(String key, Object value);
>>
>> SystemEventMessage build();
>>
>> }
>>
>> ```
>> 3. Add ``SystemEventService``
>>
>> ```java
>>
>> public interface SystemEventService {
>>
>> void init();
>>
>> CompletableFuture<MessageId> trigger(SystemEventMessage message);
>>
>> void close();
>> }
>>
>> ```
>>
>> ## Implementation
>>
>> This PIP just is a simple version, it's purpose is to discuss what event we need to trigger? how to trigger? and what is the event message structure.
>>
>> I think we need to trigger some event as bellow:
>>
>> - [ ] Client connected
>> - [ ] Client disconnected
>> - [ ] Consumer subscribe
>> - [ ] Consumer unsubscribe
>> - [ ] Producer publish
>> - [ ] Message delivered
>> - [ ] Message acked
>>
>> These are far from enough, and we need to continue to make supplements here.
>> Please fell free to give me more advice for this PIP.
>>
>> ## Reject Alternatives
>>
>> none.
>>
>> Thanks,
>> Mattisonchao
>
> Thanks,
> Haiting Jiang
Re: [DISCUSSION] PIP-132: Support Pulsar system event.
Posted by Haiting Jiang <ji...@apache.org>.
> CompletableFuture<MessageId> trigger(SystemEventMessage message);
Maybe `record` make more sense?
> SystemEventMessageBuilder properties(Map<String,Object> properties);
>
> SystemEventMessageBuilder addProperty(String key, Object value);
How to handle the serialization and deserialization if the value type is Object?
> - [ ] Producer publish
> - [ ] Message delivered
> - [ ] Message acked
I think we don't need to record normal message producing and consuming event?
That would make the system topic store too much data.
On 2022/01/05 15:27:33 mattison chao wrote:
> Original PIP : https://github.com/apache/pulsar/issues/13628
>
> Pasted bellow for quoting convenience.
>
> ——
>
> ## Motivation
>
> In some case or for better observability. We can support Pulsar event notification mechanism, that can let user subscribe this system topic to receive some **"we need notice to user"** event.
>
> ## Goal
>
> Enable users to receive Pulsar system event.
>
> ## API Changes
>
> 1. Add config ``pulsarSystemEventEnabled``
>
> ```java
> pulsarSystemEventEnabled=false
> ```
> 2. Add ``SystemEventMessageBuilder``
> ```java
>
>
> public interface SystemEventMessageBuilder {
>
> SystemEventMessageBuilder eventType(SystemEvent event);
>
> SystemEventMessageBuilder properties(Map<String,Object> properties);
>
> SystemEventMessageBuilder addProperty(String key, Object value);
>
> SystemEventMessage build();
>
> }
>
> ```
> 3. Add ``SystemEventService``
>
> ```java
>
> public interface SystemEventService {
>
> void init();
>
> CompletableFuture<MessageId> trigger(SystemEventMessage message);
>
> void close();
> }
>
> ```
>
> ## Implementation
>
> This PIP just is a simple version, it's purpose is to discuss what event we need to trigger? how to trigger? and what is the event message structure.
>
> I think we need to trigger some event as bellow:
>
> - [ ] Client connected
> - [ ] Client disconnected
> - [ ] Consumer subscribe
> - [ ] Consumer unsubscribe
> - [ ] Producer publish
> - [ ] Message delivered
> - [ ] Message acked
>
> These are far from enough, and we need to continue to make supplements here.
> Please fell free to give me more advice for this PIP.
>
> ## Reject Alternatives
>
> none.
>
> Thanks,
> Mattisonchao
Thanks,
Haiting Jiang