You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "hsy541@gmail.com" <hs...@gmail.com> on 2015/11/18 18:02:50 UTC
Q about PartitionAssignor
Hey guys,
I saw the PartitionAssignor is not in public doc API and the package name
is internals.
Does it mean this API is not stable and could be changed even in minor
release?
And in the assign method signature, the key for the "subscription" map is
memberId, what is memberId, can I manually set the id to identify member?
I want to do some sticky assignment.
Thanks!
Best,
Siyuan
Re: Q about PartitionAssignor
Posted by Guozhang Wang <wa...@gmail.com>.
It is used to carry data metadata that leader wants to propagate to other
members while doing the rebalance. For example, in Kafka Stream userData
contains the mapping of stream tasks to partition groups; in Kafka
Connector different connectors can also use this field to fill in
app-specific assignments.
In fact with userData, you have another way of sticky partitioning: set the
full assignment in the userData so all members can remember the whole
mapping and hence you do not need another persistent storage for leader
failover.
Guozhang
On Wed, Nov 18, 2015 at 3:25 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Thanks Guozhang, what is userData for in the Subscription?
>
> On Wed, Nov 18, 2015 at 12:05 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Currently the whole KafkaConsumer interface is tagged as "
> > @InterfaceStability.Unstable", meaning that the API may change in the
> > future. We have been very careful to make any dramatic public API changes
> > but still cannot guarantee this will not happen.
> >
> > Member-Id is assigned by the server-side coordinator upon accepting the
> > consumer to join the specified group, hence it cannot be manually set.
> But
> > the memberId will not change as long as the consumer is still part of the
> > members of the group, so you want to do some sticky assignment you can
> just
> > remember the memberId -> partitions map on the consumer side in some
> > persistent storage so that even when the leader who does the assignment
> has
> > failed over other new leaders can still access the past assignment
> history.
> >
> > Guozhang
> >
> >
> >
> > On Wed, Nov 18, 2015 at 9:02 AM, hsy541@gmail.com <hs...@gmail.com>
> > wrote:
> >
> > > Hey guys,
> > >
> > > I saw the PartitionAssignor is not in public doc API and the package
> name
> > > is internals.
> > >
> > > Does it mean this API is not stable and could be changed even in minor
> > > release?
> > >
> > > And in the assign method signature, the key for the "subscription" map
> is
> > > memberId, what is memberId, can I manually set the id to identify
> member?
> > > I want to do some sticky assignment.
> > >
> > >
> > > Thanks!
> > >
> > > Best,
> > > Siyuan
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Q about PartitionAssignor
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Thanks Guozhang, what is userData for in the Subscription?
On Wed, Nov 18, 2015 at 12:05 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Currently the whole KafkaConsumer interface is tagged as "
> @InterfaceStability.Unstable", meaning that the API may change in the
> future. We have been very careful to make any dramatic public API changes
> but still cannot guarantee this will not happen.
>
> Member-Id is assigned by the server-side coordinator upon accepting the
> consumer to join the specified group, hence it cannot be manually set. But
> the memberId will not change as long as the consumer is still part of the
> members of the group, so you want to do some sticky assignment you can just
> remember the memberId -> partitions map on the consumer side in some
> persistent storage so that even when the leader who does the assignment has
> failed over other new leaders can still access the past assignment history.
>
> Guozhang
>
>
>
> On Wed, Nov 18, 2015 at 9:02 AM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
> > Hey guys,
> >
> > I saw the PartitionAssignor is not in public doc API and the package name
> > is internals.
> >
> > Does it mean this API is not stable and could be changed even in minor
> > release?
> >
> > And in the assign method signature, the key for the "subscription" map is
> > memberId, what is memberId, can I manually set the id to identify member?
> > I want to do some sticky assignment.
> >
> >
> > Thanks!
> >
> > Best,
> > Siyuan
> >
>
>
>
> --
> -- Guozhang
>
Re: Q about PartitionAssignor
Posted by Guozhang Wang <wa...@gmail.com>.
Currently the whole KafkaConsumer interface is tagged as "
@InterfaceStability.Unstable", meaning that the API may change in the
future. We have been very careful to make any dramatic public API changes
but still cannot guarantee this will not happen.
Member-Id is assigned by the server-side coordinator upon accepting the
consumer to join the specified group, hence it cannot be manually set. But
the memberId will not change as long as the consumer is still part of the
members of the group, so you want to do some sticky assignment you can just
remember the memberId -> partitions map on the consumer side in some
persistent storage so that even when the leader who does the assignment has
failed over other new leaders can still access the past assignment history.
Guozhang
On Wed, Nov 18, 2015 at 9:02 AM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Hey guys,
>
> I saw the PartitionAssignor is not in public doc API and the package name
> is internals.
>
> Does it mean this API is not stable and could be changed even in minor
> release?
>
> And in the assign method signature, the key for the "subscription" map is
> memberId, what is memberId, can I manually set the id to identify member?
> I want to do some sticky assignment.
>
>
> Thanks!
>
> Best,
> Siyuan
>
--
-- Guozhang