New consumer - partitions auto assigned only on poll

[Stevo Slavić <ss...@gmail.com>]

Hello Apache Kafka community,

In the new consumer I encountered unexpected behavior. After constructing
KafakConsumer instance with configured consumer rebalance callback handler,
and subscribing to a topic with "consumer.subscribe(topic)", retrieving
subscriptions would return empty set and callback handler would not get
called (no partitions ever assigned or revoked), no matter how long
instance was up.

Then I found by inspecting KafkaConsumer code that partition assignment
will only be triggered on first poll, pollOnce has:

// ensure we have partitions assigned if we expect to
if (subscriptions.partitionsAutoAssigned())
    coordinator.ensurePartitionAssignment();

Would it make sense to include this fragment in KafkaConsumer.subscriptions
accessor as well?

Kind regards,
Stevo Slavic.

Re: New consumer - partitions auto assigned only on poll

[Stevo Slavić <ss...@gmail.com>]

Hello Jason,

Thanks for feedback. I've created ticket for this
https://issues.apache.org/jira/browse/KAFKA-2359

Kind regards,
Stevo Slavic.

On Wed, Jul 22, 2015 at 6:18 PM, Jason Gustafson <ja...@confluent.io> wrote:

> Hey Stevo,
>
> That's a good point. I think the javadoc is pretty clear that this could
> return no partitions when the consumer has no active assignment, but it may
> be a little unintuitive to have to call poll() after subscribing before you
> can get the assigned partitions. I can't think of a strong reason not to go
> ahead with the assignment in subscriptions() other than to keep it
> non-blocking. Perhaps you can open a ticket and we can get feedback from
> some other devs?
>
> Thanks,
> Jason
>
> On Wed, Jul 22, 2015 at 2:09 AM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > Hello Apache Kafka community,
> >
> > In the new consumer I encountered unexpected behavior. After constructing
> > KafakConsumer instance with configured consumer rebalance callback
> handler,
> > and subscribing to a topic with "consumer.subscribe(topic)", retrieving
> > subscriptions would return empty set and callback handler would not get
> > called (no partitions ever assigned or revoked), no matter how long
> > instance was up.
> >
> > Then I found by inspecting KafkaConsumer code that partition assignment
> > will only be triggered on first poll, pollOnce has:
> >
> > // ensure we have partitions assigned if we expect to
> > if (subscriptions.partitionsAutoAssigned())
> >     coordinator.ensurePartitionAssignment();
> >
> > Would it make sense to include this fragment in
> KafkaConsumer.subscriptions
> > accessor as well?
> >
> > Kind regards,
> > Stevo Slavic.
> >
>

Re: New consumer - partitions auto assigned only on poll

[Jason Gustafson <ja...@confluent.io>]

Hey Stevo,

That's a good point. I think the javadoc is pretty clear that this could
return no partitions when the consumer has no active assignment, but it may
be a little unintuitive to have to call poll() after subscribing before you
can get the assigned partitions. I can't think of a strong reason not to go
ahead with the assignment in subscriptions() other than to keep it
non-blocking. Perhaps you can open a ticket and we can get feedback from
some other devs?

Thanks,
Jason

On Wed, Jul 22, 2015 at 2:09 AM, Stevo Slavić <ss...@gmail.com> wrote:

> Hello Apache Kafka community,
>
> In the new consumer I encountered unexpected behavior. After constructing
> KafakConsumer instance with configured consumer rebalance callback handler,
> and subscribing to a topic with "consumer.subscribe(topic)", retrieving
> subscriptions would return empty set and callback handler would not get
> called (no partitions ever assigned or revoked), no matter how long
> instance was up.
>
> Then I found by inspecting KafkaConsumer code that partition assignment
> will only be triggered on first poll, pollOnce has:
>
> // ensure we have partitions assigned if we expect to
> if (subscriptions.partitionsAutoAssigned())
>     coordinator.ensurePartitionAssignment();
>
> Would it make sense to include this fragment in KafkaConsumer.subscriptions
> accessor as well?
>
> Kind regards,
> Stevo Slavic.
>