You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Jia Zhai <zh...@apache.org> on 2019/04/03 09:08:34 UTC
[DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Hi all,
I would like to init a discuss of "PIP 34: Add new subscribe type --
Key_Failover" here.
At present, there are 3 kinds of subscription modes: exclusive, shared, and
failover. Shared mode subscription is used a lot, because consumers could
effectively parallel consume messages of same partition. But using shared
mode will not keep the message order of same key, It would be good to
leverage share mode, while also keeping the ordering of messages.
This Proposal trying to introduce a new subscribe type Key_Failover to
extend shared type. By Key_Failover, one partition could have several
consumers to parallel consume messages, while all messages with the same
key will be dispatched to only one consumer.
Here is the link contains more information of this PIP:
https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
Thanks.
Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Posted by Yuva raj <uv...@gmail.com>.
On Thu, 4 Apr 2019 at 10:26, Jia Zhai <zh...@gmail.com> wrote:
> Thanks Penghui for the comments.
> Seems Message Key is mostly used in such a case, and by this design, the
> original protocol will no need too much change.
> If you are afraid of broken other feature. I would like to add an item
> “ordering_key” in "MessageMetadata" to serve this feature.
>
> I think having a different key will always help us pick and choose.
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Thu, Apr 4, 2019 at 12:51 AM PengHui Li <co...@gmail.com>
> wrote:
>
> > Hi jia,
> >
> > Is it better by a separate attribute to indicate the order? If i am
> > not mistaken the compaction topic is depends on the message key, does
> this
> > mean that it is not easy for users to use both of these features at the
> > same time?
> >
> > Jia Zhai <zh...@apache.org>于2019年4月3日 周三17:09写道：
> >
> > > Hi all,
> > >
> > > I would like to init a discuss of "PIP 34: Add new subscribe type --
> > > Key_Failover" here.
> > >
> > > At present, there are 3 kinds of subscription modes: exclusive, shared,
> > and
> > > failover. Shared mode subscription is used a lot, because consumers
> could
> > > effectively parallel consume messages of same partition. But using
> > shared
> > > mode will not keep the message order of same key, It would be good to
> > > leverage share mode, while also keeping the ordering of messages.
> > >
> > > This Proposal trying to introduce a new subscribe type Key_Failover to
> > > extend shared type. By Key_Failover, one partition could have several
> > > consumers to parallel consume messages, while all messages with the
> same
> > > key will be dispatched to only one consumer.
> > >
> > > Here is the link contains more information of this PIP:
> > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
> > >
> > > Thanks.
> > >
> >
>
--
*Thanks*
*Yuvaraj L*
Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Posted by Jia Zhai <zh...@gmail.com>.
Thanks Penghui for the comments.
Seems Message Key is mostly used in such a case, and by this design, the
original protocol will no need too much change.
If you are afraid of broken other feature. I would like to add an item
“ordering_key” in "MessageMetadata" to serve this feature.
Best Regards.
Jia Zhai
Beijing, China
Mobile: +86 15810491983
On Thu, Apr 4, 2019 at 12:51 AM PengHui Li <co...@gmail.com> wrote:
> Hi jia,
>
> Is it better by a separate attribute to indicate the order? If i am
> not mistaken the compaction topic is depends on the message key, does this
> mean that it is not easy for users to use both of these features at the
> same time?
>
> Jia Zhai <zh...@apache.org>于2019年4月3日 周三17:09写道：
>
> > Hi all,
> >
> > I would like to init a discuss of "PIP 34: Add new subscribe type --
> > Key_Failover" here.
> >
> > At present, there are 3 kinds of subscription modes: exclusive, shared,
> and
> > failover. Shared mode subscription is used a lot, because consumers could
> > effectively parallel consume messages of same partition. But using
> shared
> > mode will not keep the message order of same key, It would be good to
> > leverage share mode, while also keeping the ordering of messages.
> >
> > This Proposal trying to introduce a new subscribe type Key_Failover to
> > extend shared type. By Key_Failover, one partition could have several
> > consumers to parallel consume messages, while all messages with the same
> > key will be dispatched to only one consumer.
> >
> > Here is the link contains more information of this PIP:
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
> >
> > Thanks.
> >
>
Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Posted by PengHui Li <co...@gmail.com>.
Hi jia,
Is it better by a separate attribute to indicate the order? If i am
not mistaken the compaction topic is depends on the message key, does this
mean that it is not easy for users to use both of these features at the
same time?
Jia Zhai <zh...@apache.org>于2019年4月3日 周三17:09写道：
> Hi all,
>
> I would like to init a discuss of "PIP 34: Add new subscribe type --
> Key_Failover" here.
>
> At present, there are 3 kinds of subscription modes: exclusive, shared, and
> failover. Shared mode subscription is used a lot, because consumers could
> effectively parallel consume messages of same partition. But using shared
> mode will not keep the message order of same key, It would be good to
> leverage share mode, while also keeping the ordering of messages.
>
> This Proposal trying to introduce a new subscribe type Key_Failover to
> extend shared type. By Key_Failover, one partition could have several
> consumers to parallel consume messages, while all messages with the same
> key will be dispatched to only one consumer.
>
> Here is the link contains more information of this PIP:
>
> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
>
> Thanks.
>
Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Posted by PengHui Li <co...@gmail.com>.
Hi all,
I would like propose change the subscription name "Key_Shared" to
"Key_Based".
This subscription type is based on the key and key based dispatching
strategies for message dispatching.
I would like to use "Key_Based" as the subscription name, may express the
characteristics of this subscription type more accurately.
"Key_Based" also used by flink data exchanges strategies, this is a better
understanding for the users.
Sijie Guo <gu...@gmail.com> 于2019年4月11日周四 下午2:22写道：
> Acked. `ordering_key` looks good to me.
>
> - Sijie
>
> On Mon, Apr 8, 2019 at 11:40 AM Jia Zhai <zh...@gmail.com> wrote:
>
>> Thanks Penghui & Sijie for the comments, updated the wiki link.
>>
>> 1. we should add the new ordering_key, it is as Penghui mentioned, There
>> may features that all enabled, but want to use key for different purposes.
>> 2. regarding consumer listener in "failover", it is originally for
>> failover/exclusive mode, which only let one consumer alive. In
>> Key_Failover, it is more like shared mode, all alive consumer should be
>> able to consume messages.
>>
>> Best Regards.
>>
>>
>> Jia Zhai
>>
>> Beijing, China
>>
>> Mobile: +86 15810491983
>>
>>
>>
>>
>> On Fri, Apr 5, 2019 at 1:04 AM Sijie Guo <gu...@gmail.com> wrote:
>>
>> > Penguin,
>> >
>> > Thank you for your clarification! That makes sense to me then.
>> >
>> > Sijie
>> >
>> > On Thu, Apr 4, 2019 at 2:21 AM PengHui Li <co...@gmail.com>
>> wrote:
>> >
>> > > @Sijie Guo <si...@apache.org>
>> > >
>> > > Sorry, I may not have described my point in the previous email.
>> > > I try to describe it again. :)
>> > >
>> > > User can use key to achieve business logic(e.g. use user_id as message
>> > key)
>> > > At the same time, want to use another identifier as the basis for the
>> > > order.
>> > >
>> > > As the message key has many uses, avoid duplicate messages, compaction
>> > > topics.
>> > > I prefer provide a mechanism to overwrite key based sorting behavior.
>> > > Of course this is not mandatory.
>> > >
>> > > If the basis for sorting is exactly the same as the real purpose of
>> the
>> > > message key,
>> > > just use the message key. Otherwise user provide the order key to
>> > > overwrite.
>> > >
>> > > Best Regards.
>> > > Penghui
>> > >
>> > > Sijie Guo <gu...@gmail.com> 于2019年4月4日周四 下午3:36写道：
>> > >
>> > >> Jia thank you for raising this up.
>> > >>
>> > >> A couple of comments here
>> > >>
>> > >> 1. technically you don't need a consistent hashing mechanism for
>> > dividing
>> > >> hash ranges for consumers. Any hash mechanism that can map a key to a
>> > >> consumer should work here. The only advantage of consistent hashing
>> > >> mechanism is to reduce the disruptions when a consumer joins and
>> leaves.
>> > >> So
>> > >> I would prefer making the hashing mechanism pluggable.
>> > >>
>> > >> 2. regarding using key or introducing a new ordering key. I would
>> prefer
>> > >> using `key` here. having two `keys` actually makes people confused.
>> The
>> > >> `key` was used for compaction but the compaction doesn't mutate the
>> > >> original data. it just creates a new compacted ledger. in most of the
>> > >> time,
>> > >> when people use `key_failover`, they are probably accessing the
>> original
>> > >> data. Event they are trying to subscribe to compacted topic using
>> > >> `key_failover`, the ordering of messages by key will still be true.
>> so I
>> > >> wouldn't worry about this part that much.
>> > >>
>> > >> 3. in `failover` there is a consumer listener that listens on the
>> > >> partition
>> > >> assignment changes. Is there any requirements or plans on adding
>> similar
>> > >> listeners for `key_failover`?
>> > >>
>> > >> Thanks,
>> > >> Sijie
>> > >>
>> > >> On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zh...@apache.org> wrote:
>> > >>
>> > >> > Hi all,
>> > >> >
>> > >> > I would like to init a discuss of "PIP 34: Add new subscribe type
>> --
>> > >> > Key_Failover" here.
>> > >> >
>> > >> > At present, there are 3 kinds of subscription modes: exclusive,
>> > shared,
>> > >> and
>> > >> > failover. Shared mode subscription is used a lot, because consumers
>> > >> could
>> > >> > effectively parallel consume messages of same partition. But using
>> > >> shared
>> > >> > mode will not keep the message order of same key, It would be good
>> to
>> > >> > leverage share mode, while also keeping the ordering of messages.
>> > >> >
>> > >> > This Proposal trying to introduce a new subscribe type
>> Key_Failover to
>> > >> > extend shared type. By Key_Failover, one partition could have
>> several
>> > >> > consumers to parallel consume messages, while all messages with the
>> > same
>> > >> > key will be dispatched to only one consumer.
>> > >> >
>> > >> > Here is the link contains more information of this PIP:
>> > >> >
>> > >> >
>> > >>
>> >
>> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
>> > >> >
>> > >> > Thanks.
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Posted by Sijie Guo <gu...@gmail.com>.
Acked. `ordering_key` looks good to me.
- Sijie
On Mon, Apr 8, 2019 at 11:40 AM Jia Zhai <zh...@gmail.com> wrote:
> Thanks Penghui & Sijie for the comments, updated the wiki link.
>
> 1. we should add the new ordering_key, it is as Penghui mentioned, There
> may features that all enabled, but want to use key for different purposes.
> 2. regarding consumer listener in "failover", it is originally for
> failover/exclusive mode, which only let one consumer alive. In
> Key_Failover, it is more like shared mode, all alive consumer should be
> able to consume messages.
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Fri, Apr 5, 2019 at 1:04 AM Sijie Guo <gu...@gmail.com> wrote:
>
> > Penguin,
> >
> > Thank you for your clarification! That makes sense to me then.
> >
> > Sijie
> >
> > On Thu, Apr 4, 2019 at 2:21 AM PengHui Li <co...@gmail.com>
> wrote:
> >
> > > @Sijie Guo <si...@apache.org>
> > >
> > > Sorry, I may not have described my point in the previous email.
> > > I try to describe it again. :)
> > >
> > > User can use key to achieve business logic(e.g. use user_id as message
> > key)
> > > At the same time, want to use another identifier as the basis for the
> > > order.
> > >
> > > As the message key has many uses, avoid duplicate messages, compaction
> > > topics.
> > > I prefer provide a mechanism to overwrite key based sorting behavior.
> > > Of course this is not mandatory.
> > >
> > > If the basis for sorting is exactly the same as the real purpose of the
> > > message key,
> > > just use the message key. Otherwise user provide the order key to
> > > overwrite.
> > >
> > > Best Regards.
> > > Penghui
> > >
> > > Sijie Guo <gu...@gmail.com> 于2019年4月4日周四 下午3:36写道：
> > >
> > >> Jia thank you for raising this up.
> > >>
> > >> A couple of comments here
> > >>
> > >> 1. technically you don't need a consistent hashing mechanism for
> > dividing
> > >> hash ranges for consumers. Any hash mechanism that can map a key to a
> > >> consumer should work here. The only advantage of consistent hashing
> > >> mechanism is to reduce the disruptions when a consumer joins and
> leaves.
> > >> So
> > >> I would prefer making the hashing mechanism pluggable.
> > >>
> > >> 2. regarding using key or introducing a new ordering key. I would
> prefer
> > >> using `key` here. having two `keys` actually makes people confused.
> The
> > >> `key` was used for compaction but the compaction doesn't mutate the
> > >> original data. it just creates a new compacted ledger. in most of the
> > >> time,
> > >> when people use `key_failover`, they are probably accessing the
> original
> > >> data. Event they are trying to subscribe to compacted topic using
> > >> `key_failover`, the ordering of messages by key will still be true.
> so I
> > >> wouldn't worry about this part that much.
> > >>
> > >> 3. in `failover` there is a consumer listener that listens on the
> > >> partition
> > >> assignment changes. Is there any requirements or plans on adding
> similar
> > >> listeners for `key_failover`?
> > >>
> > >> Thanks,
> > >> Sijie
> > >>
> > >> On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zh...@apache.org> wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > I would like to init a discuss of "PIP 34: Add new subscribe type --
> > >> > Key_Failover" here.
> > >> >
> > >> > At present, there are 3 kinds of subscription modes: exclusive,
> > shared,
> > >> and
> > >> > failover. Shared mode subscription is used a lot, because consumers
> > >> could
> > >> > effectively parallel consume messages of same partition. But using
> > >> shared
> > >> > mode will not keep the message order of same key, It would be good
> to
> > >> > leverage share mode, while also keeping the ordering of messages.
> > >> >
> > >> > This Proposal trying to introduce a new subscribe type Key_Failover
> to
> > >> > extend shared type. By Key_Failover, one partition could have
> several
> > >> > consumers to parallel consume messages, while all messages with the
> > same
> > >> > key will be dispatched to only one consumer.
> > >> >
> > >> > Here is the link contains more information of this PIP:
> > >> >
> > >> >
> > >>
> >
> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
> > >> >
> > >> > Thanks.
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Posted by Jia Zhai <zh...@gmail.com>.
Thanks Penghui & Sijie for the comments, updated the wiki link.
1. we should add the new ordering_key, it is as Penghui mentioned, There
may features that all enabled, but want to use key for different purposes.
2. regarding consumer listener in "failover", it is originally for
failover/exclusive mode, which only let one consumer alive. In
Key_Failover, it is more like shared mode, all alive consumer should be
able to consume messages.
Best Regards.
Jia Zhai
Beijing, China
Mobile: +86 15810491983
On Fri, Apr 5, 2019 at 1:04 AM Sijie Guo <gu...@gmail.com> wrote:
> Penguin,
>
> Thank you for your clarification! That makes sense to me then.
>
> Sijie
>
> On Thu, Apr 4, 2019 at 2:21 AM PengHui Li <co...@gmail.com> wrote:
>
> > @Sijie Guo <si...@apache.org>
> >
> > Sorry, I may not have described my point in the previous email.
> > I try to describe it again. :)
> >
> > User can use key to achieve business logic(e.g. use user_id as message
> key)
> > At the same time, want to use another identifier as the basis for the
> > order.
> >
> > As the message key has many uses, avoid duplicate messages, compaction
> > topics.
> > I prefer provide a mechanism to overwrite key based sorting behavior.
> > Of course this is not mandatory.
> >
> > If the basis for sorting is exactly the same as the real purpose of the
> > message key,
> > just use the message key. Otherwise user provide the order key to
> > overwrite.
> >
> > Best Regards.
> > Penghui
> >
> > Sijie Guo <gu...@gmail.com> 于2019年4月4日周四 下午3:36写道：
> >
> >> Jia thank you for raising this up.
> >>
> >> A couple of comments here
> >>
> >> 1. technically you don't need a consistent hashing mechanism for
> dividing
> >> hash ranges for consumers. Any hash mechanism that can map a key to a
> >> consumer should work here. The only advantage of consistent hashing
> >> mechanism is to reduce the disruptions when a consumer joins and leaves.
> >> So
> >> I would prefer making the hashing mechanism pluggable.
> >>
> >> 2. regarding using key or introducing a new ordering key. I would prefer
> >> using `key` here. having two `keys` actually makes people confused. The
> >> `key` was used for compaction but the compaction doesn't mutate the
> >> original data. it just creates a new compacted ledger. in most of the
> >> time,
> >> when people use `key_failover`, they are probably accessing the original
> >> data. Event they are trying to subscribe to compacted topic using
> >> `key_failover`, the ordering of messages by key will still be true. so I
> >> wouldn't worry about this part that much.
> >>
> >> 3. in `failover` there is a consumer listener that listens on the
> >> partition
> >> assignment changes. Is there any requirements or plans on adding similar
> >> listeners for `key_failover`?
> >>
> >> Thanks,
> >> Sijie
> >>
> >> On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zh...@apache.org> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I would like to init a discuss of "PIP 34: Add new subscribe type --
> >> > Key_Failover" here.
> >> >
> >> > At present, there are 3 kinds of subscription modes: exclusive,
> shared,
> >> and
> >> > failover. Shared mode subscription is used a lot, because consumers
> >> could
> >> > effectively parallel consume messages of same partition. But using
> >> shared
> >> > mode will not keep the message order of same key, It would be good to
> >> > leverage share mode, while also keeping the ordering of messages.
> >> >
> >> > This Proposal trying to introduce a new subscribe type Key_Failover to
> >> > extend shared type. By Key_Failover, one partition could have several
> >> > consumers to parallel consume messages, while all messages with the
> same
> >> > key will be dispatched to only one consumer.
> >> >
> >> > Here is the link contains more information of this PIP:
> >> >
> >> >
> >>
> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
> >> >
> >> > Thanks.
> >> >
> >>
> >
>
Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Posted by Sijie Guo <gu...@gmail.com>.
Penguin,
Thank you for your clarification! That makes sense to me then.
Sijie
On Thu, Apr 4, 2019 at 2:21 AM PengHui Li <co...@gmail.com> wrote:
> @Sijie Guo <si...@apache.org>
>
> Sorry, I may not have described my point in the previous email.
> I try to describe it again. :)
>
> User can use key to achieve business logic(e.g. use user_id as message key)
> At the same time, want to use another identifier as the basis for the
> order.
>
> As the message key has many uses, avoid duplicate messages, compaction
> topics.
> I prefer provide a mechanism to overwrite key based sorting behavior.
> Of course this is not mandatory.
>
> If the basis for sorting is exactly the same as the real purpose of the
> message key,
> just use the message key. Otherwise user provide the order key to
> overwrite.
>
> Best Regards.
> Penghui
>
> Sijie Guo <gu...@gmail.com> 于2019年4月4日周四 下午3:36写道：
>
>> Jia thank you for raising this up.
>>
>> A couple of comments here
>>
>> 1. technically you don't need a consistent hashing mechanism for dividing
>> hash ranges for consumers. Any hash mechanism that can map a key to a
>> consumer should work here. The only advantage of consistent hashing
>> mechanism is to reduce the disruptions when a consumer joins and leaves.
>> So
>> I would prefer making the hashing mechanism pluggable.
>>
>> 2. regarding using key or introducing a new ordering key. I would prefer
>> using `key` here. having two `keys` actually makes people confused. The
>> `key` was used for compaction but the compaction doesn't mutate the
>> original data. it just creates a new compacted ledger. in most of the
>> time,
>> when people use `key_failover`, they are probably accessing the original
>> data. Event they are trying to subscribe to compacted topic using
>> `key_failover`, the ordering of messages by key will still be true. so I
>> wouldn't worry about this part that much.
>>
>> 3. in `failover` there is a consumer listener that listens on the
>> partition
>> assignment changes. Is there any requirements or plans on adding similar
>> listeners for `key_failover`?
>>
>> Thanks,
>> Sijie
>>
>> On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zh...@apache.org> wrote:
>>
>> > Hi all,
>> >
>> > I would like to init a discuss of "PIP 34: Add new subscribe type --
>> > Key_Failover" here.
>> >
>> > At present, there are 3 kinds of subscription modes: exclusive, shared,
>> and
>> > failover. Shared mode subscription is used a lot, because consumers
>> could
>> > effectively parallel consume messages of same partition. But using
>> shared
>> > mode will not keep the message order of same key, It would be good to
>> > leverage share mode, while also keeping the ordering of messages.
>> >
>> > This Proposal trying to introduce a new subscribe type Key_Failover to
>> > extend shared type. By Key_Failover, one partition could have several
>> > consumers to parallel consume messages, while all messages with the same
>> > key will be dispatched to only one consumer.
>> >
>> > Here is the link contains more information of this PIP:
>> >
>> >
>> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
>> >
>> > Thanks.
>> >
>>
>
Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Posted by PengHui Li <co...@gmail.com>.
@Sijie Guo <si...@apache.org>
Sorry, I may not have described my point in the previous email.
I try to describe it again. :)
User can use key to achieve business logic(e.g. use user_id as message key)
At the same time, want to use another identifier as the basis for the
order.
As the message key has many uses, avoid duplicate messages, compaction
topics.
I prefer provide a mechanism to overwrite key based sorting behavior.
Of course this is not mandatory.
If the basis for sorting is exactly the same as the real purpose of the
message key,
just use the message key. Otherwise user provide the order key to overwrite.
Best Regards.
Penghui
Sijie Guo <gu...@gmail.com> 于2019年4月4日周四 下午3:36写道：
> Jia thank you for raising this up.
>
> A couple of comments here
>
> 1. technically you don't need a consistent hashing mechanism for dividing
> hash ranges for consumers. Any hash mechanism that can map a key to a
> consumer should work here. The only advantage of consistent hashing
> mechanism is to reduce the disruptions when a consumer joins and leaves. So
> I would prefer making the hashing mechanism pluggable.
>
> 2. regarding using key or introducing a new ordering key. I would prefer
> using `key` here. having two `keys` actually makes people confused. The
> `key` was used for compaction but the compaction doesn't mutate the
> original data. it just creates a new compacted ledger. in most of the time,
> when people use `key_failover`, they are probably accessing the original
> data. Event they are trying to subscribe to compacted topic using
> `key_failover`, the ordering of messages by key will still be true. so I
> wouldn't worry about this part that much.
>
> 3. in `failover` there is a consumer listener that listens on the partition
> assignment changes. Is there any requirements or plans on adding similar
> listeners for `key_failover`?
>
> Thanks,
> Sijie
>
> On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zh...@apache.org> wrote:
>
> > Hi all,
> >
> > I would like to init a discuss of "PIP 34: Add new subscribe type --
> > Key_Failover" here.
> >
> > At present, there are 3 kinds of subscription modes: exclusive, shared,
> and
> > failover. Shared mode subscription is used a lot, because consumers could
> > effectively parallel consume messages of same partition. But using
> shared
> > mode will not keep the message order of same key, It would be good to
> > leverage share mode, while also keeping the ordering of messages.
> >
> > This Proposal trying to introduce a new subscribe type Key_Failover to
> > extend shared type. By Key_Failover, one partition could have several
> > consumers to parallel consume messages, while all messages with the same
> > key will be dispatched to only one consumer.
> >
> > Here is the link contains more information of this PIP:
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
> >
> > Thanks.
> >
>
Re: [DISCUSS] PIP 34: Add new subscribe type -- Key_Failover
Posted by Sijie Guo <gu...@gmail.com>.
Jia thank you for raising this up.
A couple of comments here
1. technically you don't need a consistent hashing mechanism for dividing
hash ranges for consumers. Any hash mechanism that can map a key to a
consumer should work here. The only advantage of consistent hashing
mechanism is to reduce the disruptions when a consumer joins and leaves. So
I would prefer making the hashing mechanism pluggable.
2. regarding using key or introducing a new ordering key. I would prefer
using `key` here. having two `keys` actually makes people confused. The
`key` was used for compaction but the compaction doesn't mutate the
original data. it just creates a new compacted ledger. in most of the time,
when people use `key_failover`, they are probably accessing the original
data. Event they are trying to subscribe to compacted topic using
`key_failover`, the ordering of messages by key will still be true. so I
wouldn't worry about this part that much.
3. in `failover` there is a consumer listener that listens on the partition
assignment changes. Is there any requirements or plans on adding similar
listeners for `key_failover`?
Thanks,
Sijie
On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zh...@apache.org> wrote:
> Hi all,
>
> I would like to init a discuss of "PIP 34: Add new subscribe type --
> Key_Failover" here.
>
> At present, there are 3 kinds of subscription modes: exclusive, shared, and
> failover. Shared mode subscription is used a lot, because consumers could
> effectively parallel consume messages of same partition. But using shared
> mode will not keep the message order of same key, It would be good to
> leverage share mode, while also keeping the ordering of messages.
>
> This Proposal trying to introduce a new subscribe type Key_Failover to
> extend shared type. By Key_Failover, one partition could have several
> consumers to parallel consume messages, while all messages with the same
> key will be dispatched to only one consumer.
>
> Here is the link contains more information of this PIP:
>
> https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover
>
> Thanks.
>