You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by craig w <co...@gmail.com> on 2016/02/22 16:19:22 UTC
Does pausing a topic allow other consumers to consume it?
Using kafka 0.9.0.0, if there are multiple consumers in a group and one
consumer pauses the topic+partition it's consuming, does that allow/cause
other consumers in the group to pick it up?
Re: Does pausing a topic allow other consumers to consume it?
Posted by craig w <co...@gmail.com>.
This appears to be a coding accident, I accidentally had started two
consumers for each instance of the app I was testing. Removing that seems
to have fixed what I was seeing.
On Wed, Feb 24, 2016 at 12:54 PM, Jason Gustafson <ja...@confluent.io>
wrote:
> It shouldn't be possible to have conflicting assignments. It kind of sounds
> like your consumers aren't all staying in the group. Maybe you could share
> a code snippet with what you're trying to do?
>
> -Jason
>
> On Wed, Feb 24, 2016 at 3:47 AM, craig w <co...@gmail.com> wrote:
>
> > I implemented the ConsumerRebalanceListener just to log the
> TopicPartitions
> > revoked and assigned. In a test, I had 5 consumers subscribed to one
> topic
> > (which had 15 partitions) all with the same group id. Upon startup they
> all
> > were assigned 3 different partitions. I pushed in 15 messages (one per
> > partition) and each consumer received 3 messages. All 15 messages were
> > processed once.
> >
> > I then pushed 90 messages (to ensure each consumer would be busy for a
> > bit), then I started a 6th consumer. I watched the logs of the 5 original
> > consumers and saw their TopicPartitions were being revoked/assigned a few
> > times. When the dust settled I saw duplicates subscriptions, at least
> > according to the ConsumerRebalanceListener.
> >
> > One consumer was assigned all 15 partitions. While other consumers had
> > anywhere from 2 - 4 partitions assigned. This means, multiple consumers
> in
> > the same group are consuming the same partition. Am I doing something
> wrong
> > or perhaps misunderstanding what's going on with the
> > ConsumberRebalanceListener? I didn't believe it was possible for
> consumers
> > in the same group subscribed to the same topic to be consuming the same
> > partition.
> >
> > Thanks
> >
> > On Mon, Feb 22, 2016 at 2:12 PM, craig w <co...@gmail.com> wrote:
> >
> > > I think I'll need to do that.
> > >
> > > My current setup is where I have 200 partitions for a topic with 200
> > > consumers, all part of the same group. Each consumer does a "poll".
> Upon
> > > receiving messages it will pause the topic/paritions it received, while
> > the
> > > messages are processed, as each message is processed the offset is
> > > committed. When all messages are processed, the topic/partition is
> > resumed
> > > and poll is called again. However, during this it's possible that the
> > > consumers get restarted (as part of an upgrade, etc) or a consumer dies
> > and
> > > a new one starts up.
> > >
> > > On Mon, Feb 22, 2016 at 2:05 PM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > >> That is correct. The typical way to handle this (if you need to) is to
> > >> repause the partitions in the ConsumerRebalanceListener
> > >> onPartitionsAssigned() hook.
> > >>
> > >> -Jason
> > >>
> > >> On Mon, Feb 22, 2016 at 11:03 AM, craig w <co...@gmail.com>
> wrote:
> > >>
> > >> > so if I had a bunch of consumers coming online while some consumers
> > had
> > >> > partitions paused, those consumers that were paused would become
> > >> unpaused?
> > >> >
> > >> > On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson <
> jason@confluent.io>
> > >> > wrote:
> > >> >
> > >> > > Nope. Pausing a partition just stops the consumer from sending any
> > >> more
> > >> > > fetches for it. It will not trigger a partition reassignment. One
> > >> thing
> > >> > to
> > >> > > be wary of, however, is that the partition will automatically be
> > >> unpaused
> > >> > > after the next normal rebalance.
> > >> > >
> > >> > > -Jason
> > >> > >
> > >> > > On Mon, Feb 22, 2016 at 7:19 AM, craig w <co...@gmail.com>
> > wrote:
> > >> > >
> > >> > > > Using kafka 0.9.0.0, if there are multiple consumers in a group
> > and
> > >> one
> > >> > > > consumer pauses the topic+partition it's consuming, does that
> > >> > allow/cause
> > >> > > > other consumers in the group to pick it up?
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > https://github.com/mindscratch
> > >> > https://www.google.com/+CraigWickesser
> > >> > https://twitter.com/mind_scratch
> > >> > https://twitter.com/craig_links
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > https://github.com/mindscratch
> > > https://www.google.com/+CraigWickesser
> > > https://twitter.com/mind_scratch
> > > https://twitter.com/craig_links
> > >
> > >
> >
> >
> > --
> >
> > https://github.com/mindscratch
> > https://www.google.com/+CraigWickesser
> > https://twitter.com/mind_scratch
> > https://twitter.com/craig_links
> >
>
--
https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links
Re: Does pausing a topic allow other consumers to consume it?
Posted by Jason Gustafson <ja...@confluent.io>.
It shouldn't be possible to have conflicting assignments. It kind of sounds
like your consumers aren't all staying in the group. Maybe you could share
a code snippet with what you're trying to do?
-Jason
On Wed, Feb 24, 2016 at 3:47 AM, craig w <co...@gmail.com> wrote:
> I implemented the ConsumerRebalanceListener just to log the TopicPartitions
> revoked and assigned. In a test, I had 5 consumers subscribed to one topic
> (which had 15 partitions) all with the same group id. Upon startup they all
> were assigned 3 different partitions. I pushed in 15 messages (one per
> partition) and each consumer received 3 messages. All 15 messages were
> processed once.
>
> I then pushed 90 messages (to ensure each consumer would be busy for a
> bit), then I started a 6th consumer. I watched the logs of the 5 original
> consumers and saw their TopicPartitions were being revoked/assigned a few
> times. When the dust settled I saw duplicates subscriptions, at least
> according to the ConsumerRebalanceListener.
>
> One consumer was assigned all 15 partitions. While other consumers had
> anywhere from 2 - 4 partitions assigned. This means, multiple consumers in
> the same group are consuming the same partition. Am I doing something wrong
> or perhaps misunderstanding what's going on with the
> ConsumberRebalanceListener? I didn't believe it was possible for consumers
> in the same group subscribed to the same topic to be consuming the same
> partition.
>
> Thanks
>
> On Mon, Feb 22, 2016 at 2:12 PM, craig w <co...@gmail.com> wrote:
>
> > I think I'll need to do that.
> >
> > My current setup is where I have 200 partitions for a topic with 200
> > consumers, all part of the same group. Each consumer does a "poll". Upon
> > receiving messages it will pause the topic/paritions it received, while
> the
> > messages are processed, as each message is processed the offset is
> > committed. When all messages are processed, the topic/partition is
> resumed
> > and poll is called again. However, during this it's possible that the
> > consumers get restarted (as part of an upgrade, etc) or a consumer dies
> and
> > a new one starts up.
> >
> > On Mon, Feb 22, 2016 at 2:05 PM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> >> That is correct. The typical way to handle this (if you need to) is to
> >> repause the partitions in the ConsumerRebalanceListener
> >> onPartitionsAssigned() hook.
> >>
> >> -Jason
> >>
> >> On Mon, Feb 22, 2016 at 11:03 AM, craig w <co...@gmail.com> wrote:
> >>
> >> > so if I had a bunch of consumers coming online while some consumers
> had
> >> > partitions paused, those consumers that were paused would become
> >> unpaused?
> >> >
> >> > On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson <ja...@confluent.io>
> >> > wrote:
> >> >
> >> > > Nope. Pausing a partition just stops the consumer from sending any
> >> more
> >> > > fetches for it. It will not trigger a partition reassignment. One
> >> thing
> >> > to
> >> > > be wary of, however, is that the partition will automatically be
> >> unpaused
> >> > > after the next normal rebalance.
> >> > >
> >> > > -Jason
> >> > >
> >> > > On Mon, Feb 22, 2016 at 7:19 AM, craig w <co...@gmail.com>
> wrote:
> >> > >
> >> > > > Using kafka 0.9.0.0, if there are multiple consumers in a group
> and
> >> one
> >> > > > consumer pauses the topic+partition it's consuming, does that
> >> > allow/cause
> >> > > > other consumers in the group to pick it up?
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > https://github.com/mindscratch
> >> > https://www.google.com/+CraigWickesser
> >> > https://twitter.com/mind_scratch
> >> > https://twitter.com/craig_links
> >> >
> >>
> >
> >
> >
> > --
> >
> > https://github.com/mindscratch
> > https://www.google.com/+CraigWickesser
> > https://twitter.com/mind_scratch
> > https://twitter.com/craig_links
> >
> >
>
>
> --
>
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links
>
Re: Does pausing a topic allow other consumers to consume it?
Posted by craig w <co...@gmail.com>.
I implemented the ConsumerRebalanceListener just to log the TopicPartitions
revoked and assigned. In a test, I had 5 consumers subscribed to one topic
(which had 15 partitions) all with the same group id. Upon startup they all
were assigned 3 different partitions. I pushed in 15 messages (one per
partition) and each consumer received 3 messages. All 15 messages were
processed once.
I then pushed 90 messages (to ensure each consumer would be busy for a
bit), then I started a 6th consumer. I watched the logs of the 5 original
consumers and saw their TopicPartitions were being revoked/assigned a few
times. When the dust settled I saw duplicates subscriptions, at least
according to the ConsumerRebalanceListener.
One consumer was assigned all 15 partitions. While other consumers had
anywhere from 2 - 4 partitions assigned. This means, multiple consumers in
the same group are consuming the same partition. Am I doing something wrong
or perhaps misunderstanding what's going on with the
ConsumberRebalanceListener? I didn't believe it was possible for consumers
in the same group subscribed to the same topic to be consuming the same
partition.
Thanks
On Mon, Feb 22, 2016 at 2:12 PM, craig w <co...@gmail.com> wrote:
> I think I'll need to do that.
>
> My current setup is where I have 200 partitions for a topic with 200
> consumers, all part of the same group. Each consumer does a "poll". Upon
> receiving messages it will pause the topic/paritions it received, while the
> messages are processed, as each message is processed the offset is
> committed. When all messages are processed, the topic/partition is resumed
> and poll is called again. However, during this it's possible that the
> consumers get restarted (as part of an upgrade, etc) or a consumer dies and
> a new one starts up.
>
> On Mon, Feb 22, 2016 at 2:05 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
>> That is correct. The typical way to handle this (if you need to) is to
>> repause the partitions in the ConsumerRebalanceListener
>> onPartitionsAssigned() hook.
>>
>> -Jason
>>
>> On Mon, Feb 22, 2016 at 11:03 AM, craig w <co...@gmail.com> wrote:
>>
>> > so if I had a bunch of consumers coming online while some consumers had
>> > partitions paused, those consumers that were paused would become
>> unpaused?
>> >
>> > On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson <ja...@confluent.io>
>> > wrote:
>> >
>> > > Nope. Pausing a partition just stops the consumer from sending any
>> more
>> > > fetches for it. It will not trigger a partition reassignment. One
>> thing
>> > to
>> > > be wary of, however, is that the partition will automatically be
>> unpaused
>> > > after the next normal rebalance.
>> > >
>> > > -Jason
>> > >
>> > > On Mon, Feb 22, 2016 at 7:19 AM, craig w <co...@gmail.com> wrote:
>> > >
>> > > > Using kafka 0.9.0.0, if there are multiple consumers in a group and
>> one
>> > > > consumer pauses the topic+partition it's consuming, does that
>> > allow/cause
>> > > > other consumers in the group to pick it up?
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > https://github.com/mindscratch
>> > https://www.google.com/+CraigWickesser
>> > https://twitter.com/mind_scratch
>> > https://twitter.com/craig_links
>> >
>>
>
>
>
> --
>
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links
>
>
--
https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links
Re: Does pausing a topic allow other consumers to consume it?
Posted by craig w <co...@gmail.com>.
I think I'll need to do that.
My current setup is where I have 200 partitions for a topic with 200
consumers, all part of the same group. Each consumer does a "poll". Upon
receiving messages it will pause the topic/paritions it received, while the
messages are processed, as each message is processed the offset is
committed. When all messages are processed, the topic/partition is resumed
and poll is called again. However, during this it's possible that the
consumers get restarted (as part of an upgrade, etc) or a consumer dies and
a new one starts up.
On Mon, Feb 22, 2016 at 2:05 PM, Jason Gustafson <ja...@confluent.io> wrote:
> That is correct. The typical way to handle this (if you need to) is to
> repause the partitions in the ConsumerRebalanceListener
> onPartitionsAssigned() hook.
>
> -Jason
>
> On Mon, Feb 22, 2016 at 11:03 AM, craig w <co...@gmail.com> wrote:
>
> > so if I had a bunch of consumers coming online while some consumers had
> > partitions paused, those consumers that were paused would become
> unpaused?
> >
> > On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> > > Nope. Pausing a partition just stops the consumer from sending any more
> > > fetches for it. It will not trigger a partition reassignment. One thing
> > to
> > > be wary of, however, is that the partition will automatically be
> unpaused
> > > after the next normal rebalance.
> > >
> > > -Jason
> > >
> > > On Mon, Feb 22, 2016 at 7:19 AM, craig w <co...@gmail.com> wrote:
> > >
> > > > Using kafka 0.9.0.0, if there are multiple consumers in a group and
> one
> > > > consumer pauses the topic+partition it's consuming, does that
> > allow/cause
> > > > other consumers in the group to pick it up?
> > > >
> > >
> >
> >
> >
> > --
> >
> > https://github.com/mindscratch
> > https://www.google.com/+CraigWickesser
> > https://twitter.com/mind_scratch
> > https://twitter.com/craig_links
> >
>
--
https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links
Re: Does pausing a topic allow other consumers to consume it?
Posted by Jason Gustafson <ja...@confluent.io>.
That is correct. The typical way to handle this (if you need to) is to
repause the partitions in the ConsumerRebalanceListener
onPartitionsAssigned() hook.
-Jason
On Mon, Feb 22, 2016 at 11:03 AM, craig w <co...@gmail.com> wrote:
> so if I had a bunch of consumers coming online while some consumers had
> partitions paused, those consumers that were paused would become unpaused?
>
> On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Nope. Pausing a partition just stops the consumer from sending any more
> > fetches for it. It will not trigger a partition reassignment. One thing
> to
> > be wary of, however, is that the partition will automatically be unpaused
> > after the next normal rebalance.
> >
> > -Jason
> >
> > On Mon, Feb 22, 2016 at 7:19 AM, craig w <co...@gmail.com> wrote:
> >
> > > Using kafka 0.9.0.0, if there are multiple consumers in a group and one
> > > consumer pauses the topic+partition it's consuming, does that
> allow/cause
> > > other consumers in the group to pick it up?
> > >
> >
>
>
>
> --
>
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links
>
Re: Does pausing a topic allow other consumers to consume it?
Posted by craig w <co...@gmail.com>.
so if I had a bunch of consumers coming online while some consumers had
partitions paused, those consumers that were paused would become unpaused?
On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Nope. Pausing a partition just stops the consumer from sending any more
> fetches for it. It will not trigger a partition reassignment. One thing to
> be wary of, however, is that the partition will automatically be unpaused
> after the next normal rebalance.
>
> -Jason
>
> On Mon, Feb 22, 2016 at 7:19 AM, craig w <co...@gmail.com> wrote:
>
> > Using kafka 0.9.0.0, if there are multiple consumers in a group and one
> > consumer pauses the topic+partition it's consuming, does that allow/cause
> > other consumers in the group to pick it up?
> >
>
--
https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links
Re: Does pausing a topic allow other consumers to consume it?
Posted by Jason Gustafson <ja...@confluent.io>.
Nope. Pausing a partition just stops the consumer from sending any more
fetches for it. It will not trigger a partition reassignment. One thing to
be wary of, however, is that the partition will automatically be unpaused
after the next normal rebalance.
-Jason
On Mon, Feb 22, 2016 at 7:19 AM, craig w <co...@gmail.com> wrote:
> Using kafka 0.9.0.0, if there are multiple consumers in a group and one
> consumer pauses the topic+partition it's consuming, does that allow/cause
> other consumers in the group to pick it up?
>