Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover

[PengHui Li <co...@gmail.com>]

Hi all,

I would like propose change the subscription name "Key_Shared" to
"Key_Based".

This subscription type is based on the key and key based dispatching
strategies for message dispatching.
I would like to use "Key_Based" as the subscription name, may express the
characteristics of this subscription type more accurately.

"Key_Based" also used by flink data exchanges strategies, this is a better
understanding for the users.

Sijie Guo <gu...@gmail.com> 于2019年4月11日周四 下午2:22写道：

> Acked. `ordering_key` looks good to me.
>
> - Sijie
>
> On Mon, Apr 8, 2019 at 11:40 AM Jia Zhai <zh...@gmail.com> wrote:
>
>> Thanks Penghui & Sijie for the comments, updated the wiki link.
>>
>> 1. we should add the new ordering_key, it is as Penghui mentioned, There
>> may features that all enabled, but want to use key for different purposes.
>> 2. regarding consumer listener in "failover", it is originally for
>> failover/exclusive mode, which only let one consumer alive.  In
>> Key_Failover, it is more like shared mode, all alive consumer should be
>> able to consume messages.
>>
>> Best Regards.
>>
>>
>> Jia Zhai
>>
>> Beijing, China
>>
>> Mobile: +86 15810491983
>>
>>
>>
>>
>> On Fri, Apr 5, 2019 at 1:04 AM Sijie Guo <gu...@gmail.com> wrote:
>>
>> > Penguin,
>> >
>> > Thank you for your clarification! That makes sense to me then.
>> >
>> > Sijie
>> >
>> > On Thu, Apr 4, 2019 at 2:21 AM PengHui Li <co...@gmail.com>
>> wrote:
>> >
>> > > @Sijie Guo <si...@apache.org>
>> > >
>> > > Sorry, I may not have described my point in the previous email.
>> > > I try to describe it again. :)
>> > >
>> > > User can use key to achieve business logic(e.g. use user_id as message
>> > key)
>> > > At the same time, want to use another identifier as the basis for the
>> > > order.
>> > >
>> > > As the message key has many uses, avoid duplicate messages, compaction
>> > > topics.
>> > > I prefer provide a mechanism to overwrite key based sorting behavior.
>> > > Of course this is not mandatory.
>> > >
>> > > If the basis for sorting is exactly the same as the real purpose of
>> the
>> > > message key,
>> > > just use the message key. Otherwise user provide the order key to
>> > > overwrite.
>> > >
>> > > Best Regards.
>> > > Penghui
>> > >
>> > > Sijie Guo <gu...@gmail.com> 于2019年4月4日周四 下午3:36写道：
>> > >
>> > >> Jia thank you for raising this up.
>> > >>
>> > >> A couple of comments here
>> > >>
>> > >> 1. technically you don't need a consistent hashing mechanism for
>> > dividing
>> > >> hash ranges for consumers. Any hash mechanism that can map a key to a
>> > >> consumer should work here. The only advantage of consistent hashing
>> > >> mechanism is to reduce the disruptions when a consumer joins and
>> leaves.
>> > >> So
>> > >> I would prefer making the hashing mechanism pluggable.
>> > >>
>> > >> 2. regarding using key or introducing a new ordering key. I would
>> prefer
>> > >> using `key` here. having two `keys` actually makes people confused.
>> The
>> > >> `key` was used for compaction but the compaction doesn't mutate the
>> > >> original data. it just creates a new compacted ledger. in most of the
>> > >> time,
>> > >> when people use `key_failover`, they are probably accessing the
>> original
>> > >> data. Event they are trying to subscribe to compacted topic using
>> > >> `key_failover`, the ordering of messages by key will still be true.
>> so I
>> > >> wouldn't worry about this part that much.
>> > >>
>> > >> 3. in `failover` there is a consumer listener that listens on the
>> > >> partition
>> > >> assignment changes. Is there any requirements or plans on adding
>> similar
>> > >> listeners for `key_failover`?
>> > >>
>> > >> Thanks,
>> > >> Sijie
>> > >>
>> > >> On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zh...@apache.org> wrote:
>> > >>
>> > >> > Hi all,
>> > >> >
>> > >> > I would like to init a discuss of "PIP 34: Add new subscribe type
>> --
>> > >> > Key_Failover" here.
>> > >> >
>> > >> > At present, there are 3 kinds of subscription modes: exclusive,
>> > shared,
>> > >> and
>> > >> > failover. Shared mode subscription is used a lot, because consumers
>> > >> could
>> > >> > effectively parallel consume messages of same partition.  But using
>> > >> shared
>> > >> > mode will not keep the message order of same key, It would be good
>> to
>> > >> > leverage share mode, while also keeping the ordering of messages.
>> > >> >
>> > >> > This Proposal trying to introduce a new subscribe type
>> Key_Failover to
>> > >> > extend shared type. By Key_Failover, one partition could have
>> several
>> > >> > consumers to parallel consume messages, while all messages with the
>> > same
>> > >> > key will be dispatched to only one consumer.
>> > >> >
>> > >> > Here is the link contains more information of this PIP:
>> > >> >
>> > >> >
>> > >>
>> >
>> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
>> > >> >
>> > >> > Thanks.
>> > >> >
>> > >>
>> > >
>> >
>>
>