You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Venkatesan Subramanian <vs...@lendingclub.com> on 2016/02/19 19:43:17 UTC
Partitions not assigned on topic auto creation
I am running into a curious problem with auto creation of topics and would appreciate any insight into what could be causing it.
Some context first: We are using Confluent 2.0 platform and I have an integration test where the following happens:
1. A worker thread is spun up and it creates a kafka consumer that subscribes to two topics: Topic A and Topic B. Both of these topics do not exist at this time.
2. The kafka consumer polls the two topics periodically.
3. A kafka producer sends an event into Topic A (and hence creates the topic through auto topic creation)
4. The kafka consumer created previously is assigned a partition from Topic A (no assignment for Topic B since it does not exist yet).
5. worker polls and receives the event from Topic A partition. It processes the event and produces an event into Topic B (and hence the topic B is created at this time)
6. I am expected to see a partition from Topic B getting assigned to the consumer next time a poll happens but nothing happens.
Note that the same integration test works fine if I pre-create the two topics before running the test.
________________________________
DISCLAIMER: The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this document and any attachments.
Re: Partitions not assigned on topic auto creation
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Venkat,
The logs will help, but I'll probably need to see some code to understand
what is going on (consumer usage can be a little subtle sometimes). You're
welcome to open a ticket if you like or post here.
Thanks,
Jason
On Mon, Feb 22, 2016 at 5:09 PM, Venkatesan Subramanian <
vsubramanian@lendingclub.com> wrote:
> Jason:
>
> Thanks for your reply.
>
> I did turn on logging for kafka as well when I ran the test. I do not see
> any partition assignment for the topic happening.
> If you don’t see anything obviously wrong in what I am doing, I can file a
> ticket with Confluent and attach the logs to it.
>
> Regards
> Venkat
>
>
>
> On 2/19/16, 11:22 AM, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> >Hi Venkatesan,
> >
> >Autocreation of topics happens when the broker receives a topic metadata
> >request. That should mean that both topics get created when the consumer
> >does the initial poll() since that is the first time that topic metadata
> >would be fetched (fetching topic metadata allows the consumer and producer
> >to determine the number of partitions in the topic and the current
> >partition leaders). Are you sure that partitions from both topics have not
> >been assigned at that time?
> >
> >Thanks,
> >Jason
> >
> >On Fri, Feb 19, 2016 at 10:43 AM, Venkatesan Subramanian <
> >vsubramanian@lendingclub.com> wrote:
> >
> >> I am running into a curious problem with auto creation of topics and
> would
> >> appreciate any insight into what could be causing it.
> >>
> >> Some context first: We are using Confluent 2.0 platform and I have an
> >> integration test where the following happens:
> >>
> >>
> >> 1. A worker thread is spun up and it creates a kafka consumer that
> >> subscribes to two topics: Topic A and Topic B. Both of these topics do
> not
> >> exist at this time.
> >> 2. The kafka consumer polls the two topics periodically.
> >> 3. A kafka producer sends an event into Topic A (and hence creates
> the
> >> topic through auto topic creation)
> >> 4. The kafka consumer created previously is assigned a partition from
> >> Topic A (no assignment for Topic B since it does not exist yet).
> >> 5. worker polls and receives the event from Topic A partition. It
> >> processes the event and produces an event into Topic B (and hence the
> topic
> >> B is created at this time)
> >> 6. I am expected to see a partition from Topic B getting assigned to
> >> the consumer next time a poll happens but nothing happens.
> >>
> >> Note that the same integration test works fine if I pre-create the two
> >> topics before running the test.
> >>
> >> ________________________________
> >> DISCLAIMER: The information transmitted is intended only for the person
> or
> >> entity to which it is addressed and may contain confidential and/or
> >> privileged material. Any review, re-transmission, dissemination or other
> >> use of, or taking of any action in reliance upon this information by
> >> persons or entities other than the intended recipient is prohibited. If
> you
> >> received this in error, please contact the sender and destroy any
> copies of
> >> this document and any attachments.
> >>
>
Re: Partitions not assigned on topic auto creation
Posted by Venkatesan Subramanian <vs...@lendingclub.com>.
Jason:
Thanks for your reply.
I did turn on logging for kafka as well when I ran the test. I do not see any partition assignment for the topic happening.
If you don’t see anything obviously wrong in what I am doing, I can file a ticket with Confluent and attach the logs to it.
Regards
Venkat
On 2/19/16, 11:22 AM, "Jason Gustafson" <ja...@confluent.io> wrote:
>Hi Venkatesan,
>
>Autocreation of topics happens when the broker receives a topic metadata
>request. That should mean that both topics get created when the consumer
>does the initial poll() since that is the first time that topic metadata
>would be fetched (fetching topic metadata allows the consumer and producer
>to determine the number of partitions in the topic and the current
>partition leaders). Are you sure that partitions from both topics have not
>been assigned at that time?
>
>Thanks,
>Jason
>
>On Fri, Feb 19, 2016 at 10:43 AM, Venkatesan Subramanian <
>vsubramanian@lendingclub.com> wrote:
>
>> I am running into a curious problem with auto creation of topics and would
>> appreciate any insight into what could be causing it.
>>
>> Some context first: We are using Confluent 2.0 platform and I have an
>> integration test where the following happens:
>>
>>
>> 1. A worker thread is spun up and it creates a kafka consumer that
>> subscribes to two topics: Topic A and Topic B. Both of these topics do not
>> exist at this time.
>> 2. The kafka consumer polls the two topics periodically.
>> 3. A kafka producer sends an event into Topic A (and hence creates the
>> topic through auto topic creation)
>> 4. The kafka consumer created previously is assigned a partition from
>> Topic A (no assignment for Topic B since it does not exist yet).
>> 5. worker polls and receives the event from Topic A partition. It
>> processes the event and produces an event into Topic B (and hence the topic
>> B is created at this time)
>> 6. I am expected to see a partition from Topic B getting assigned to
>> the consumer next time a poll happens but nothing happens.
>>
>> Note that the same integration test works fine if I pre-create the two
>> topics before running the test.
>>
>> ________________________________
>> DISCLAIMER: The information transmitted is intended only for the person or
>> entity to which it is addressed and may contain confidential and/or
>> privileged material. Any review, re-transmission, dissemination or other
>> use of, or taking of any action in reliance upon this information by
>> persons or entities other than the intended recipient is prohibited. If you
>> received this in error, please contact the sender and destroy any copies of
>> this document and any attachments.
>>
Re: Partitions not assigned on topic auto creation
Posted by Jason Gustafson <ja...@confluent.io>.
Hi Venkatesan,
Autocreation of topics happens when the broker receives a topic metadata
request. That should mean that both topics get created when the consumer
does the initial poll() since that is the first time that topic metadata
would be fetched (fetching topic metadata allows the consumer and producer
to determine the number of partitions in the topic and the current
partition leaders). Are you sure that partitions from both topics have not
been assigned at that time?
Thanks,
Jason
On Fri, Feb 19, 2016 at 10:43 AM, Venkatesan Subramanian <
vsubramanian@lendingclub.com> wrote:
> I am running into a curious problem with auto creation of topics and would
> appreciate any insight into what could be causing it.
>
> Some context first: We are using Confluent 2.0 platform and I have an
> integration test where the following happens:
>
>
> 1. A worker thread is spun up and it creates a kafka consumer that
> subscribes to two topics: Topic A and Topic B. Both of these topics do not
> exist at this time.
> 2. The kafka consumer polls the two topics periodically.
> 3. A kafka producer sends an event into Topic A (and hence creates the
> topic through auto topic creation)
> 4. The kafka consumer created previously is assigned a partition from
> Topic A (no assignment for Topic B since it does not exist yet).
> 5. worker polls and receives the event from Topic A partition. It
> processes the event and produces an event into Topic B (and hence the topic
> B is created at this time)
> 6. I am expected to see a partition from Topic B getting assigned to
> the consumer next time a poll happens but nothing happens.
>
> Note that the same integration test works fine if I pre-create the two
> topics before running the test.
>
> ________________________________
> DISCLAIMER: The information transmitted is intended only for the person or
> entity to which it is addressed and may contain confidential and/or
> privileged material. Any review, re-transmission, dissemination or other
> use of, or taking of any action in reliance upon this information by
> persons or entities other than the intended recipient is prohibited. If you
> received this in error, please contact the sender and destroy any copies of
> this document and any attachments.
>