You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Dan S <da...@gmail.com> on 2023/03/23 18:10:54 UTC
Should the mock consumer call the consumer rebalance listener on rebalance?
Hello all,
It seems to me based on reading the code, that the consumer rebalance
listener that is passed into the mock consumer when subscribing to a topic
is not actually called when a rebalance is simulated. My understanding is
that the consumer rebalance listener is called from the consumer
coordinator, which is called by kafka consumer. The mock consumer doesn't
seem to use the consumer coordinator or use any other mechanism to call the
consumer rebalance listener. Is my understanding correct? Would it make
sense to trigger the consumer rebalance listener when rebalance is called?
I would be willing to try to make the patch if the behavior is indeed
currently incorrect/incomplete.
Thanks,
Daniel
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Posted by Dan S <da...@gmail.com>.
I have also created the pull request for the behaviour change.
https://github.com/apache/kafka/pull/13455
Thanks again,
Dan
On Thu, Mar 23, 2023 at 11:40 PM Dan S <da...@gmail.com> wrote:
> I have created the jira, the PR will follow within the next few days.
> https://issues.apache.org/jira/browse/KAFKA-14841
> Thanks again
>
> On Thu, Mar 23, 2023 at 7:33 PM Philip Nee <ph...@gmail.com> wrote:
>
>> Awesome!
>>
>> On Thu, Mar 23, 2023 at 12:31 PM Dan S <da...@gmail.com> wrote:
>>
>> > I'll pick it up, thanks!
>> >
>> > On Thu, Mar 23, 2023, 19:27 Philip Nee <ph...@gmail.com> wrote:
>> >
>> > > Hey Dan,
>> > >
>> > > Your analysis looks right. I do see TODO item there to implement the
>> > > rebalance callback. Would you like to create a jira issue and work on
>> > that?
>> > >
>> > > Thanks,
>> > > P
>> > >
>> > > On Thu, Mar 23, 2023 at 12:11 PM Dan S <da...@gmail.com> wrote:
>> > >
>> > > > Hi Philip,
>> > > >
>> > > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our
>> case
>> > > > we're calling the variant of subscribe that takes a custom
>> > > > ConsumerRebalanceListener(which among other things logs when it's
>> > > called),
>> > > > and we're then calling rebalance (to simulate a rebalance) and
>> removing
>> > > all
>> > > > partitions from the consumer, polling a few times, and then adding
>> them
>> > > > back. We're noticing our custom listener is never called, which was
>> > > > unexpected, but based on the code analysis in the original email
>> seems
>> > to
>> > > > be the current implementation. The question is whether this is
>> indeed
>> > > > desired behavior.
>> > > >
>> > > > On Thu, Mar 23, 2023, 18:17 Philip Nee <ph...@gmail.com> wrote:
>> > > >
>> > > > > Hey Dan,
>> > > > >
>> > > > > Thanks for looking into this. Are you talking about
>> MockConsumer? If
>> > > you
>> > > > > invoke subscribe(Collection<String> topics), it actually
>> registers a
>> > > Noop
>> > > > > callback. Perhaps this is what you are seeing?
>> > > > >
>> > > > > P
>> > > > >
>> > > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S <da...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > > Hello all,
>> > > > > >
>> > > > > > It seems to me based on reading the code, that the consumer
>> > rebalance
>> > > > > > listener that is passed into the mock consumer when subscribing
>> to
>> > a
>> > > > > topic
>> > > > > > is not actually called when a rebalance is simulated. My
>> > > understanding
>> > > > is
>> > > > > > that the consumer rebalance listener is called from the consumer
>> > > > > > coordinator, which is called by kafka consumer. The mock
>> consumer
>> > > > doesn't
>> > > > > > seem to use the consumer coordinator or use any other mechanism
>> to
>> > > call
>> > > > > the
>> > > > > > consumer rebalance listener. Is my understanding correct? Would
>> it
>> > > make
>> > > > > > sense to trigger the consumer rebalance listener when rebalance
>> is
>> > > > > called?
>> > > > > >
>> > > > > > I would be willing to try to make the patch if the behavior is
>> > indeed
>> > > > > > currently incorrect/incomplete.
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Daniel
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Posted by Dan S <da...@gmail.com>.
I have created the jira, the PR will follow within the next few days.
https://issues.apache.org/jira/browse/KAFKA-14841
Thanks again
On Thu, Mar 23, 2023 at 7:33 PM Philip Nee <ph...@gmail.com> wrote:
> Awesome!
>
> On Thu, Mar 23, 2023 at 12:31 PM Dan S <da...@gmail.com> wrote:
>
> > I'll pick it up, thanks!
> >
> > On Thu, Mar 23, 2023, 19:27 Philip Nee <ph...@gmail.com> wrote:
> >
> > > Hey Dan,
> > >
> > > Your analysis looks right. I do see TODO item there to implement the
> > > rebalance callback. Would you like to create a jira issue and work on
> > that?
> > >
> > > Thanks,
> > > P
> > >
> > > On Thu, Mar 23, 2023 at 12:11 PM Dan S <da...@gmail.com> wrote:
> > >
> > > > Hi Philip,
> > > >
> > > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our
> case
> > > > we're calling the variant of subscribe that takes a custom
> > > > ConsumerRebalanceListener(which among other things logs when it's
> > > called),
> > > > and we're then calling rebalance (to simulate a rebalance) and
> removing
> > > all
> > > > partitions from the consumer, polling a few times, and then adding
> them
> > > > back. We're noticing our custom listener is never called, which was
> > > > unexpected, but based on the code analysis in the original email
> seems
> > to
> > > > be the current implementation. The question is whether this is indeed
> > > > desired behavior.
> > > >
> > > > On Thu, Mar 23, 2023, 18:17 Philip Nee <ph...@gmail.com> wrote:
> > > >
> > > > > Hey Dan,
> > > > >
> > > > > Thanks for looking into this. Are you talking about MockConsumer?
> If
> > > you
> > > > > invoke subscribe(Collection<String> topics), it actually registers
> a
> > > Noop
> > > > > callback. Perhaps this is what you are seeing?
> > > > >
> > > > > P
> > > > >
> > > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S <da...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > It seems to me based on reading the code, that the consumer
> > rebalance
> > > > > > listener that is passed into the mock consumer when subscribing
> to
> > a
> > > > > topic
> > > > > > is not actually called when a rebalance is simulated. My
> > > understanding
> > > > is
> > > > > > that the consumer rebalance listener is called from the consumer
> > > > > > coordinator, which is called by kafka consumer. The mock consumer
> > > > doesn't
> > > > > > seem to use the consumer coordinator or use any other mechanism
> to
> > > call
> > > > > the
> > > > > > consumer rebalance listener. Is my understanding correct? Would
> it
> > > make
> > > > > > sense to trigger the consumer rebalance listener when rebalance
> is
> > > > > called?
> > > > > >
> > > > > > I would be willing to try to make the patch if the behavior is
> > indeed
> > > > > > currently incorrect/incomplete.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Daniel
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Posted by Philip Nee <ph...@gmail.com>.
Awesome!
On Thu, Mar 23, 2023 at 12:31 PM Dan S <da...@gmail.com> wrote:
> I'll pick it up, thanks!
>
> On Thu, Mar 23, 2023, 19:27 Philip Nee <ph...@gmail.com> wrote:
>
> > Hey Dan,
> >
> > Your analysis looks right. I do see TODO item there to implement the
> > rebalance callback. Would you like to create a jira issue and work on
> that?
> >
> > Thanks,
> > P
> >
> > On Thu, Mar 23, 2023 at 12:11 PM Dan S <da...@gmail.com> wrote:
> >
> > > Hi Philip,
> > >
> > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our case
> > > we're calling the variant of subscribe that takes a custom
> > > ConsumerRebalanceListener(which among other things logs when it's
> > called),
> > > and we're then calling rebalance (to simulate a rebalance) and removing
> > all
> > > partitions from the consumer, polling a few times, and then adding them
> > > back. We're noticing our custom listener is never called, which was
> > > unexpected, but based on the code analysis in the original email seems
> to
> > > be the current implementation. The question is whether this is indeed
> > > desired behavior.
> > >
> > > On Thu, Mar 23, 2023, 18:17 Philip Nee <ph...@gmail.com> wrote:
> > >
> > > > Hey Dan,
> > > >
> > > > Thanks for looking into this. Are you talking about MockConsumer? If
> > you
> > > > invoke subscribe(Collection<String> topics), it actually registers a
> > Noop
> > > > callback. Perhaps this is what you are seeing?
> > > >
> > > > P
> > > >
> > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S <da...@gmail.com>
> wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > It seems to me based on reading the code, that the consumer
> rebalance
> > > > > listener that is passed into the mock consumer when subscribing to
> a
> > > > topic
> > > > > is not actually called when a rebalance is simulated. My
> > understanding
> > > is
> > > > > that the consumer rebalance listener is called from the consumer
> > > > > coordinator, which is called by kafka consumer. The mock consumer
> > > doesn't
> > > > > seem to use the consumer coordinator or use any other mechanism to
> > call
> > > > the
> > > > > consumer rebalance listener. Is my understanding correct? Would it
> > make
> > > > > sense to trigger the consumer rebalance listener when rebalance is
> > > > called?
> > > > >
> > > > > I would be willing to try to make the patch if the behavior is
> indeed
> > > > > currently incorrect/incomplete.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Daniel
> > > > >
> > > >
> > >
> >
>
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Posted by Dan S <da...@gmail.com>.
I'll pick it up, thanks!
On Thu, Mar 23, 2023, 19:27 Philip Nee <ph...@gmail.com> wrote:
> Hey Dan,
>
> Your analysis looks right. I do see TODO item there to implement the
> rebalance callback. Would you like to create a jira issue and work on that?
>
> Thanks,
> P
>
> On Thu, Mar 23, 2023 at 12:11 PM Dan S <da...@gmail.com> wrote:
>
> > Hi Philip,
> >
> > Thanks for the quick reply. Yes, it's the MockConsumer, but in our case
> > we're calling the variant of subscribe that takes a custom
> > ConsumerRebalanceListener(which among other things logs when it's
> called),
> > and we're then calling rebalance (to simulate a rebalance) and removing
> all
> > partitions from the consumer, polling a few times, and then adding them
> > back. We're noticing our custom listener is never called, which was
> > unexpected, but based on the code analysis in the original email seems to
> > be the current implementation. The question is whether this is indeed
> > desired behavior.
> >
> > On Thu, Mar 23, 2023, 18:17 Philip Nee <ph...@gmail.com> wrote:
> >
> > > Hey Dan,
> > >
> > > Thanks for looking into this. Are you talking about MockConsumer? If
> you
> > > invoke subscribe(Collection<String> topics), it actually registers a
> Noop
> > > callback. Perhaps this is what you are seeing?
> > >
> > > P
> > >
> > > On Thu, Mar 23, 2023 at 11:11 AM Dan S <da...@gmail.com> wrote:
> > >
> > > > Hello all,
> > > >
> > > > It seems to me based on reading the code, that the consumer rebalance
> > > > listener that is passed into the mock consumer when subscribing to a
> > > topic
> > > > is not actually called when a rebalance is simulated. My
> understanding
> > is
> > > > that the consumer rebalance listener is called from the consumer
> > > > coordinator, which is called by kafka consumer. The mock consumer
> > doesn't
> > > > seem to use the consumer coordinator or use any other mechanism to
> call
> > > the
> > > > consumer rebalance listener. Is my understanding correct? Would it
> make
> > > > sense to trigger the consumer rebalance listener when rebalance is
> > > called?
> > > >
> > > > I would be willing to try to make the patch if the behavior is indeed
> > > > currently incorrect/incomplete.
> > > >
> > > > Thanks,
> > > >
> > > > Daniel
> > > >
> > >
> >
>
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Posted by Philip Nee <ph...@gmail.com>.
Hey Dan,
Your analysis looks right. I do see TODO item there to implement the
rebalance callback. Would you like to create a jira issue and work on that?
Thanks,
P
On Thu, Mar 23, 2023 at 12:11 PM Dan S <da...@gmail.com> wrote:
> Hi Philip,
>
> Thanks for the quick reply. Yes, it's the MockConsumer, but in our case
> we're calling the variant of subscribe that takes a custom
> ConsumerRebalanceListener(which among other things logs when it's called),
> and we're then calling rebalance (to simulate a rebalance) and removing all
> partitions from the consumer, polling a few times, and then adding them
> back. We're noticing our custom listener is never called, which was
> unexpected, but based on the code analysis in the original email seems to
> be the current implementation. The question is whether this is indeed
> desired behavior.
>
> On Thu, Mar 23, 2023, 18:17 Philip Nee <ph...@gmail.com> wrote:
>
> > Hey Dan,
> >
> > Thanks for looking into this. Are you talking about MockConsumer? If you
> > invoke subscribe(Collection<String> topics), it actually registers a Noop
> > callback. Perhaps this is what you are seeing?
> >
> > P
> >
> > On Thu, Mar 23, 2023 at 11:11 AM Dan S <da...@gmail.com> wrote:
> >
> > > Hello all,
> > >
> > > It seems to me based on reading the code, that the consumer rebalance
> > > listener that is passed into the mock consumer when subscribing to a
> > topic
> > > is not actually called when a rebalance is simulated. My understanding
> is
> > > that the consumer rebalance listener is called from the consumer
> > > coordinator, which is called by kafka consumer. The mock consumer
> doesn't
> > > seem to use the consumer coordinator or use any other mechanism to call
> > the
> > > consumer rebalance listener. Is my understanding correct? Would it make
> > > sense to trigger the consumer rebalance listener when rebalance is
> > called?
> > >
> > > I would be willing to try to make the patch if the behavior is indeed
> > > currently incorrect/incomplete.
> > >
> > > Thanks,
> > >
> > > Daniel
> > >
> >
>
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Posted by Dan S <da...@gmail.com>.
Hi Philip,
Thanks for the quick reply. Yes, it's the MockConsumer, but in our case
we're calling the variant of subscribe that takes a custom
ConsumerRebalanceListener(which among other things logs when it's called),
and we're then calling rebalance (to simulate a rebalance) and removing all
partitions from the consumer, polling a few times, and then adding them
back. We're noticing our custom listener is never called, which was
unexpected, but based on the code analysis in the original email seems to
be the current implementation. The question is whether this is indeed
desired behavior.
On Thu, Mar 23, 2023, 18:17 Philip Nee <ph...@gmail.com> wrote:
> Hey Dan,
>
> Thanks for looking into this. Are you talking about MockConsumer? If you
> invoke subscribe(Collection<String> topics), it actually registers a Noop
> callback. Perhaps this is what you are seeing?
>
> P
>
> On Thu, Mar 23, 2023 at 11:11 AM Dan S <da...@gmail.com> wrote:
>
> > Hello all,
> >
> > It seems to me based on reading the code, that the consumer rebalance
> > listener that is passed into the mock consumer when subscribing to a
> topic
> > is not actually called when a rebalance is simulated. My understanding is
> > that the consumer rebalance listener is called from the consumer
> > coordinator, which is called by kafka consumer. The mock consumer doesn't
> > seem to use the consumer coordinator or use any other mechanism to call
> the
> > consumer rebalance listener. Is my understanding correct? Would it make
> > sense to trigger the consumer rebalance listener when rebalance is
> called?
> >
> > I would be willing to try to make the patch if the behavior is indeed
> > currently incorrect/incomplete.
> >
> > Thanks,
> >
> > Daniel
> >
>
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Posted by Philip Nee <ph...@gmail.com>.
Hey Dan,
Thanks for looking into this. Are you talking about MockConsumer? If you
invoke subscribe(Collection<String> topics), it actually registers a Noop
callback. Perhaps this is what you are seeing?
P
On Thu, Mar 23, 2023 at 11:11 AM Dan S <da...@gmail.com> wrote:
> Hello all,
>
> It seems to me based on reading the code, that the consumer rebalance
> listener that is passed into the mock consumer when subscribing to a topic
> is not actually called when a rebalance is simulated. My understanding is
> that the consumer rebalance listener is called from the consumer
> coordinator, which is called by kafka consumer. The mock consumer doesn't
> seem to use the consumer coordinator or use any other mechanism to call the
> consumer rebalance listener. Is my understanding correct? Would it make
> sense to trigger the consumer rebalance listener when rebalance is called?
>
> I would be willing to try to make the patch if the behavior is indeed
> currently incorrect/incomplete.
>
> Thanks,
>
> Daniel
>