You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by sahil aggarwal <sa...@gmail.com> on 2017/12/21 14:05:57 UTC
Consumer group describe issue
Hi,
Facing issue where *kafka-consumer-groups.sh --describe * get stuck if one
of the partition is unavailable i.e no leader. Going through some code
found that it does following to get log end offset:
* Create consumer
* For each partition
* assign partition
* seek to end
* get position
Issue is KafkaConsumer.posiiton() use Fetcher.retrieveOffsetsByTimes()
internally which is called with timeout Long.MAX_VALUE and it gets stuck in
loop there.
Any pointers?
*Version*: 0.10.0.1
Thanks,
Sahil
Re: Consumer group describe issue
Posted by Bill Bejeck <bi...@confluent.io>.
Can someone add Sahil to the contributor list?
Thanks,
Bill
On Thu, Feb 22, 2018 at 3:39 AM, sahil aggarwal <sa...@gmail.com>
wrote:
> Bill,
>
> I have created the jira: https://issues.apache.org/jira/browse/KAFKA-6581
> but i am not able to assign it to myself.
>
> Can you please help?
>
>
> Thanks,
> Sahil
>
> On 28 December 2017 at 11:50, sahil aggarwal <sa...@gmail.com>
> wrote:
>
> > @TedYu
> > From 0.10.0 <https://github.com/apache/kafka/tree/0.10.0>
> >
> > @Bill
> > Thanks for the pointer. Will follow the steps mentioned in the doc.
> >
> > On 28 December 2017 at 07:39, Ted Yu <yu...@gmail.com> wrote:
> >
> >> Which branch was the patch generated from ?
> >> When I tried to apply the patch:
> >>
> >> 6 out of 7 hunks FAILED -- saving rejects to file
> >> core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala.rej
> >>
> >> FYI
> >>
> >> On Mon, Dec 25, 2017 at 9:45 PM, sahil aggarwal <sa...@gmail.com>
> >> wrote:
> >>
> >> > Attached the patch. If someone can review it will be very helpfull.
> >> >
> >> >
> >> >
> >> > Thanks,
> >> > sahil
> >> >
> >> > On 23 December 2017 at 13:11, sahil aggarwal <sa...@gmail.com>
> >> > wrote:
> >> >
> >> >> my bad its consumer.endOffsets in ConsumerGroupCommand.scala.
> >> >>
> >> >> https://github.com/apache/kafka/blob/ef97ed7ee5cb883a30245ea
> >> >> 4f77ef99d4db15373/core/src/main/scala/kafka/admin/Consume
> >> >> rGroupCommand.scala#L467
> >> >>
> >> >> On 23 December 2017 at 13:07, Ted Yu <yu...@gmail.com> wrote:
> >> >>
> >> >>> Sahil:
> >> >>> I did a quick search in 0.11.0 branch and trunk for getEndOffsets
> but
> >> >>> didn't find any occurrence.
> >> >>>
> >> >>> Mind giving us the location (and class) where getEndOffsets is
> called
> >> ?
> >> >>>
> >> >>> Thanks
> >> >>>
> >> >>> On Fri, Dec 22, 2017 at 11:29 PM, sahil aggarwal <
> >> sahil.agg15@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>> > Fixed it by some code change in ConsumerGroupCommand.scala.
> >> Possible to
> >> >>> > push it upstream for 0.10.* ?
> >> >>> >
> >> >>> > It seems to be fixed in 0.11.* where it uses getEndOffsets() which
> >> has
> >> >>> > requestTimeoutMs instead of Long.MAX_VALUE.
> >> >>> >
> >> >>> > On 23 December 2017 at 02:46, Matthias J. Sax <
> >> matthias@confluent.io>
> >> >>> > wrote:
> >> >>> >
> >> >>> > > Your observation is correct. KafkaConsumer.position() is a
> >> blocking
> >> >>> > > call. It's a know issue that there is no configurable timeout
> >> value.
> >> >>> > >
> >> >>> > > I am not aware of any workaround.
> >> >>> > >
> >> >>> > >
> >> >>> > > -Matthias
> >> >>> > >
> >> >>> > > On 12/21/17 6:05 AM, sahil aggarwal wrote:
> >> >>> > > > Hi,
> >> >>> > > >
> >> >>> > > > Facing issue where *kafka-consumer-groups.sh --describe * get
> >> >>> stuck if
> >> >>> > > one
> >> >>> > > > of the partition is unavailable i.e no leader. Going through
> >> some
> >> >>> code
> >> >>> > > > found that it does following to get log end offset:
> >> >>> > > >
> >> >>> > > > * Create consumer
> >> >>> > > > * For each partition
> >> >>> > > > * assign partition
> >> >>> > > > * seek to end
> >> >>> > > > * get position
> >> >>> > > >
> >> >>> > > > Issue is KafkaConsumer.posiiton() use
> >> >>> Fetcher.retrieveOffsetsByTimes()
> >> >>> > > > internally which is called with timeout Long.MAX_VALUE and it
> >> gets
> >> >>> > stuck
> >> >>> > > in
> >> >>> > > > loop there.
> >> >>> > > >
> >> >>> > > >
> >> >>> > > > Any pointers?
> >> >>> > > >
> >> >>> > > >
> >> >>> > > > *Version*: 0.10.0.1
> >> >>> > > >
> >> >>> > > >
> >> >>> > > > Thanks,
> >> >>> > > > Sahil
> >> >>> > > >
> >> >>> > >
> >> >>> > >
> >> >>> >
> >> >>>
> >> >>
> >> >>
> >> >
> >>
> >
> >
>
Re: Consumer group describe issue
Posted by sahil aggarwal <sa...@gmail.com>.
Bill,
I have created the jira: https://issues.apache.org/jira/browse/KAFKA-6581
but i am not able to assign it to myself.
Can you please help?
Thanks,
Sahil
On 28 December 2017 at 11:50, sahil aggarwal <sa...@gmail.com> wrote:
> @TedYu
> From 0.10.0 <https://github.com/apache/kafka/tree/0.10.0>
>
> @Bill
> Thanks for the pointer. Will follow the steps mentioned in the doc.
>
> On 28 December 2017 at 07:39, Ted Yu <yu...@gmail.com> wrote:
>
>> Which branch was the patch generated from ?
>> When I tried to apply the patch:
>>
>> 6 out of 7 hunks FAILED -- saving rejects to file
>> core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala.rej
>>
>> FYI
>>
>> On Mon, Dec 25, 2017 at 9:45 PM, sahil aggarwal <sa...@gmail.com>
>> wrote:
>>
>> > Attached the patch. If someone can review it will be very helpfull.
>> >
>> >
>> >
>> > Thanks,
>> > sahil
>> >
>> > On 23 December 2017 at 13:11, sahil aggarwal <sa...@gmail.com>
>> > wrote:
>> >
>> >> my bad its consumer.endOffsets in ConsumerGroupCommand.scala.
>> >>
>> >> https://github.com/apache/kafka/blob/ef97ed7ee5cb883a30245ea
>> >> 4f77ef99d4db15373/core/src/main/scala/kafka/admin/Consume
>> >> rGroupCommand.scala#L467
>> >>
>> >> On 23 December 2017 at 13:07, Ted Yu <yu...@gmail.com> wrote:
>> >>
>> >>> Sahil:
>> >>> I did a quick search in 0.11.0 branch and trunk for getEndOffsets but
>> >>> didn't find any occurrence.
>> >>>
>> >>> Mind giving us the location (and class) where getEndOffsets is called
>> ?
>> >>>
>> >>> Thanks
>> >>>
>> >>> On Fri, Dec 22, 2017 at 11:29 PM, sahil aggarwal <
>> sahil.agg15@gmail.com>
>> >>> wrote:
>> >>>
>> >>> > Fixed it by some code change in ConsumerGroupCommand.scala.
>> Possible to
>> >>> > push it upstream for 0.10.* ?
>> >>> >
>> >>> > It seems to be fixed in 0.11.* where it uses getEndOffsets() which
>> has
>> >>> > requestTimeoutMs instead of Long.MAX_VALUE.
>> >>> >
>> >>> > On 23 December 2017 at 02:46, Matthias J. Sax <
>> matthias@confluent.io>
>> >>> > wrote:
>> >>> >
>> >>> > > Your observation is correct. KafkaConsumer.position() is a
>> blocking
>> >>> > > call. It's a know issue that there is no configurable timeout
>> value.
>> >>> > >
>> >>> > > I am not aware of any workaround.
>> >>> > >
>> >>> > >
>> >>> > > -Matthias
>> >>> > >
>> >>> > > On 12/21/17 6:05 AM, sahil aggarwal wrote:
>> >>> > > > Hi,
>> >>> > > >
>> >>> > > > Facing issue where *kafka-consumer-groups.sh --describe * get
>> >>> stuck if
>> >>> > > one
>> >>> > > > of the partition is unavailable i.e no leader. Going through
>> some
>> >>> code
>> >>> > > > found that it does following to get log end offset:
>> >>> > > >
>> >>> > > > * Create consumer
>> >>> > > > * For each partition
>> >>> > > > * assign partition
>> >>> > > > * seek to end
>> >>> > > > * get position
>> >>> > > >
>> >>> > > > Issue is KafkaConsumer.posiiton() use
>> >>> Fetcher.retrieveOffsetsByTimes()
>> >>> > > > internally which is called with timeout Long.MAX_VALUE and it
>> gets
>> >>> > stuck
>> >>> > > in
>> >>> > > > loop there.
>> >>> > > >
>> >>> > > >
>> >>> > > > Any pointers?
>> >>> > > >
>> >>> > > >
>> >>> > > > *Version*: 0.10.0.1
>> >>> > > >
>> >>> > > >
>> >>> > > > Thanks,
>> >>> > > > Sahil
>> >>> > > >
>> >>> > >
>> >>> > >
>> >>> >
>> >>>
>> >>
>> >>
>> >
>>
>
>
Re: Consumer group describe issue
Posted by sahil aggarwal <sa...@gmail.com>.
@TedYu
From 0.10.0 <https://github.com/apache/kafka/tree/0.10.0>
@Bill
Thanks for the pointer. Will follow the steps mentioned in the doc.
On 28 December 2017 at 07:39, Ted Yu <yu...@gmail.com> wrote:
> Which branch was the patch generated from ?
> When I tried to apply the patch:
>
> 6 out of 7 hunks FAILED -- saving rejects to file
> core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala.rej
>
> FYI
>
> On Mon, Dec 25, 2017 at 9:45 PM, sahil aggarwal <sa...@gmail.com>
> wrote:
>
> > Attached the patch. If someone can review it will be very helpfull.
> >
> >
> >
> > Thanks,
> > sahil
> >
> > On 23 December 2017 at 13:11, sahil aggarwal <sa...@gmail.com>
> > wrote:
> >
> >> my bad its consumer.endOffsets in ConsumerGroupCommand.scala.
> >>
> >> https://github.com/apache/kafka/blob/ef97ed7ee5cb883a30245ea
> >> 4f77ef99d4db15373/core/src/main/scala/kafka/admin/Consume
> >> rGroupCommand.scala#L467
> >>
> >> On 23 December 2017 at 13:07, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> Sahil:
> >>> I did a quick search in 0.11.0 branch and trunk for getEndOffsets but
> >>> didn't find any occurrence.
> >>>
> >>> Mind giving us the location (and class) where getEndOffsets is called ?
> >>>
> >>> Thanks
> >>>
> >>> On Fri, Dec 22, 2017 at 11:29 PM, sahil aggarwal <
> sahil.agg15@gmail.com>
> >>> wrote:
> >>>
> >>> > Fixed it by some code change in ConsumerGroupCommand.scala. Possible
> to
> >>> > push it upstream for 0.10.* ?
> >>> >
> >>> > It seems to be fixed in 0.11.* where it uses getEndOffsets() which
> has
> >>> > requestTimeoutMs instead of Long.MAX_VALUE.
> >>> >
> >>> > On 23 December 2017 at 02:46, Matthias J. Sax <matthias@confluent.io
> >
> >>> > wrote:
> >>> >
> >>> > > Your observation is correct. KafkaConsumer.position() is a blocking
> >>> > > call. It's a know issue that there is no configurable timeout
> value.
> >>> > >
> >>> > > I am not aware of any workaround.
> >>> > >
> >>> > >
> >>> > > -Matthias
> >>> > >
> >>> > > On 12/21/17 6:05 AM, sahil aggarwal wrote:
> >>> > > > Hi,
> >>> > > >
> >>> > > > Facing issue where *kafka-consumer-groups.sh --describe * get
> >>> stuck if
> >>> > > one
> >>> > > > of the partition is unavailable i.e no leader. Going through some
> >>> code
> >>> > > > found that it does following to get log end offset:
> >>> > > >
> >>> > > > * Create consumer
> >>> > > > * For each partition
> >>> > > > * assign partition
> >>> > > > * seek to end
> >>> > > > * get position
> >>> > > >
> >>> > > > Issue is KafkaConsumer.posiiton() use
> >>> Fetcher.retrieveOffsetsByTimes()
> >>> > > > internally which is called with timeout Long.MAX_VALUE and it
> gets
> >>> > stuck
> >>> > > in
> >>> > > > loop there.
> >>> > > >
> >>> > > >
> >>> > > > Any pointers?
> >>> > > >
> >>> > > >
> >>> > > > *Version*: 0.10.0.1
> >>> > > >
> >>> > > >
> >>> > > > Thanks,
> >>> > > > Sahil
> >>> > > >
> >>> > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
Re: Consumer group describe issue
Posted by Ted Yu <yu...@gmail.com>.
Which branch was the patch generated from ?
When I tried to apply the patch:
6 out of 7 hunks FAILED -- saving rejects to file
core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala.rej
FYI
On Mon, Dec 25, 2017 at 9:45 PM, sahil aggarwal <sa...@gmail.com>
wrote:
> Attached the patch. If someone can review it will be very helpfull.
>
>
>
> Thanks,
> sahil
>
> On 23 December 2017 at 13:11, sahil aggarwal <sa...@gmail.com>
> wrote:
>
>> my bad its consumer.endOffsets in ConsumerGroupCommand.scala.
>>
>> https://github.com/apache/kafka/blob/ef97ed7ee5cb883a30245ea
>> 4f77ef99d4db15373/core/src/main/scala/kafka/admin/Consume
>> rGroupCommand.scala#L467
>>
>> On 23 December 2017 at 13:07, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Sahil:
>>> I did a quick search in 0.11.0 branch and trunk for getEndOffsets but
>>> didn't find any occurrence.
>>>
>>> Mind giving us the location (and class) where getEndOffsets is called ?
>>>
>>> Thanks
>>>
>>> On Fri, Dec 22, 2017 at 11:29 PM, sahil aggarwal <sa...@gmail.com>
>>> wrote:
>>>
>>> > Fixed it by some code change in ConsumerGroupCommand.scala. Possible to
>>> > push it upstream for 0.10.* ?
>>> >
>>> > It seems to be fixed in 0.11.* where it uses getEndOffsets() which has
>>> > requestTimeoutMs instead of Long.MAX_VALUE.
>>> >
>>> > On 23 December 2017 at 02:46, Matthias J. Sax <ma...@confluent.io>
>>> > wrote:
>>> >
>>> > > Your observation is correct. KafkaConsumer.position() is a blocking
>>> > > call. It's a know issue that there is no configurable timeout value.
>>> > >
>>> > > I am not aware of any workaround.
>>> > >
>>> > >
>>> > > -Matthias
>>> > >
>>> > > On 12/21/17 6:05 AM, sahil aggarwal wrote:
>>> > > > Hi,
>>> > > >
>>> > > > Facing issue where *kafka-consumer-groups.sh --describe * get
>>> stuck if
>>> > > one
>>> > > > of the partition is unavailable i.e no leader. Going through some
>>> code
>>> > > > found that it does following to get log end offset:
>>> > > >
>>> > > > * Create consumer
>>> > > > * For each partition
>>> > > > * assign partition
>>> > > > * seek to end
>>> > > > * get position
>>> > > >
>>> > > > Issue is KafkaConsumer.posiiton() use
>>> Fetcher.retrieveOffsetsByTimes()
>>> > > > internally which is called with timeout Long.MAX_VALUE and it gets
>>> > stuck
>>> > > in
>>> > > > loop there.
>>> > > >
>>> > > >
>>> > > > Any pointers?
>>> > > >
>>> > > >
>>> > > > *Version*: 0.10.0.1
>>> > > >
>>> > > >
>>> > > > Thanks,
>>> > > > Sahil
>>> > > >
>>> > >
>>> > >
>>> >
>>>
>>
>>
>
Re: Consumer group describe issue
Posted by Bill Bejeck <bi...@confluent.io>.
Sahil,
Thanks for the patch.
In order to have your patch reviewed, please look over the contribution
guidelines outlined here https://kafka.apache.org/contributing
Thanks,
Bill
On Tue, Dec 26, 2017 at 12:45 AM, sahil aggarwal <sa...@gmail.com>
wrote:
> Attached the patch. If someone can review it will be very helpfull.
>
>
>
> Thanks,
> sahil
>
> On 23 December 2017 at 13:11, sahil aggarwal <sa...@gmail.com>
> wrote:
>
>> my bad its consumer.endOffsets in ConsumerGroupCommand.scala.
>>
>> https://github.com/apache/kafka/blob/ef97ed7ee5cb883a30245ea
>> 4f77ef99d4db15373/core/src/main/scala/kafka/admin/Consume
>> rGroupCommand.scala#L467
>>
>> On 23 December 2017 at 13:07, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Sahil:
>>> I did a quick search in 0.11.0 branch and trunk for getEndOffsets but
>>> didn't find any occurrence.
>>>
>>> Mind giving us the location (and class) where getEndOffsets is called ?
>>>
>>> Thanks
>>>
>>> On Fri, Dec 22, 2017 at 11:29 PM, sahil aggarwal <sa...@gmail.com>
>>> wrote:
>>>
>>> > Fixed it by some code change in ConsumerGroupCommand.scala. Possible to
>>> > push it upstream for 0.10.* ?
>>> >
>>> > It seems to be fixed in 0.11.* where it uses getEndOffsets() which has
>>> > requestTimeoutMs instead of Long.MAX_VALUE.
>>> >
>>> > On 23 December 2017 at 02:46, Matthias J. Sax <ma...@confluent.io>
>>> > wrote:
>>> >
>>> > > Your observation is correct. KafkaConsumer.position() is a blocking
>>> > > call. It's a know issue that there is no configurable timeout value.
>>> > >
>>> > > I am not aware of any workaround.
>>> > >
>>> > >
>>> > > -Matthias
>>> > >
>>> > > On 12/21/17 6:05 AM, sahil aggarwal wrote:
>>> > > > Hi,
>>> > > >
>>> > > > Facing issue where *kafka-consumer-groups.sh --describe * get
>>> stuck if
>>> > > one
>>> > > > of the partition is unavailable i.e no leader. Going through some
>>> code
>>> > > > found that it does following to get log end offset:
>>> > > >
>>> > > > * Create consumer
>>> > > > * For each partition
>>> > > > * assign partition
>>> > > > * seek to end
>>> > > > * get position
>>> > > >
>>> > > > Issue is KafkaConsumer.posiiton() use
>>> Fetcher.retrieveOffsetsByTimes()
>>> > > > internally which is called with timeout Long.MAX_VALUE and it gets
>>> > stuck
>>> > > in
>>> > > > loop there.
>>> > > >
>>> > > >
>>> > > > Any pointers?
>>> > > >
>>> > > >
>>> > > > *Version*: 0.10.0.1
>>> > > >
>>> > > >
>>> > > > Thanks,
>>> > > > Sahil
>>> > > >
>>> > >
>>> > >
>>> >
>>>
>>
>>
>
Re: Consumer group describe issue
Posted by sahil aggarwal <sa...@gmail.com>.
Attached the patch. If someone can review it will be very helpfull.
Thanks,
sahil
On 23 December 2017 at 13:11, sahil aggarwal <sa...@gmail.com> wrote:
> my bad its consumer.endOffsets in ConsumerGroupCommand.scala.
>
> https://github.com/apache/kafka/blob/ef97ed7ee5cb883a30245ea4f77ef9
> 9d4db15373/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L467
>
> On 23 December 2017 at 13:07, Ted Yu <yu...@gmail.com> wrote:
>
>> Sahil:
>> I did a quick search in 0.11.0 branch and trunk for getEndOffsets but
>> didn't find any occurrence.
>>
>> Mind giving us the location (and class) where getEndOffsets is called ?
>>
>> Thanks
>>
>> On Fri, Dec 22, 2017 at 11:29 PM, sahil aggarwal <sa...@gmail.com>
>> wrote:
>>
>> > Fixed it by some code change in ConsumerGroupCommand.scala. Possible to
>> > push it upstream for 0.10.* ?
>> >
>> > It seems to be fixed in 0.11.* where it uses getEndOffsets() which has
>> > requestTimeoutMs instead of Long.MAX_VALUE.
>> >
>> > On 23 December 2017 at 02:46, Matthias J. Sax <ma...@confluent.io>
>> > wrote:
>> >
>> > > Your observation is correct. KafkaConsumer.position() is a blocking
>> > > call. It's a know issue that there is no configurable timeout value.
>> > >
>> > > I am not aware of any workaround.
>> > >
>> > >
>> > > -Matthias
>> > >
>> > > On 12/21/17 6:05 AM, sahil aggarwal wrote:
>> > > > Hi,
>> > > >
>> > > > Facing issue where *kafka-consumer-groups.sh --describe * get stuck
>> if
>> > > one
>> > > > of the partition is unavailable i.e no leader. Going through some
>> code
>> > > > found that it does following to get log end offset:
>> > > >
>> > > > * Create consumer
>> > > > * For each partition
>> > > > * assign partition
>> > > > * seek to end
>> > > > * get position
>> > > >
>> > > > Issue is KafkaConsumer.posiiton() use Fetcher.retrieveOffsetsByTimes
>> ()
>> > > > internally which is called with timeout Long.MAX_VALUE and it gets
>> > stuck
>> > > in
>> > > > loop there.
>> > > >
>> > > >
>> > > > Any pointers?
>> > > >
>> > > >
>> > > > *Version*: 0.10.0.1
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Sahil
>> > > >
>> > >
>> > >
>> >
>>
>
>
Re: Consumer group describe issue
Posted by sahil aggarwal <sa...@gmail.com>.
my bad its consumer.endOffsets in ConsumerGroupCommand.scala.
https://github.com/apache/kafka/blob/ef97ed7ee5cb883a30245ea4f77ef99d4db15373/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L467
On 23 December 2017 at 13:07, Ted Yu <yu...@gmail.com> wrote:
> Sahil:
> I did a quick search in 0.11.0 branch and trunk for getEndOffsets but
> didn't find any occurrence.
>
> Mind giving us the location (and class) where getEndOffsets is called ?
>
> Thanks
>
> On Fri, Dec 22, 2017 at 11:29 PM, sahil aggarwal <sa...@gmail.com>
> wrote:
>
> > Fixed it by some code change in ConsumerGroupCommand.scala. Possible to
> > push it upstream for 0.10.* ?
> >
> > It seems to be fixed in 0.11.* where it uses getEndOffsets() which has
> > requestTimeoutMs instead of Long.MAX_VALUE.
> >
> > On 23 December 2017 at 02:46, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> > > Your observation is correct. KafkaConsumer.position() is a blocking
> > > call. It's a know issue that there is no configurable timeout value.
> > >
> > > I am not aware of any workaround.
> > >
> > >
> > > -Matthias
> > >
> > > On 12/21/17 6:05 AM, sahil aggarwal wrote:
> > > > Hi,
> > > >
> > > > Facing issue where *kafka-consumer-groups.sh --describe * get stuck
> if
> > > one
> > > > of the partition is unavailable i.e no leader. Going through some
> code
> > > > found that it does following to get log end offset:
> > > >
> > > > * Create consumer
> > > > * For each partition
> > > > * assign partition
> > > > * seek to end
> > > > * get position
> > > >
> > > > Issue is KafkaConsumer.posiiton() use Fetcher.
> retrieveOffsetsByTimes()
> > > > internally which is called with timeout Long.MAX_VALUE and it gets
> > stuck
> > > in
> > > > loop there.
> > > >
> > > >
> > > > Any pointers?
> > > >
> > > >
> > > > *Version*: 0.10.0.1
> > > >
> > > >
> > > > Thanks,
> > > > Sahil
> > > >
> > >
> > >
> >
>
Re: Consumer group describe issue
Posted by Ted Yu <yu...@gmail.com>.
Sahil:
I did a quick search in 0.11.0 branch and trunk for getEndOffsets but
didn't find any occurrence.
Mind giving us the location (and class) where getEndOffsets is called ?
Thanks
On Fri, Dec 22, 2017 at 11:29 PM, sahil aggarwal <sa...@gmail.com>
wrote:
> Fixed it by some code change in ConsumerGroupCommand.scala. Possible to
> push it upstream for 0.10.* ?
>
> It seems to be fixed in 0.11.* where it uses getEndOffsets() which has
> requestTimeoutMs instead of Long.MAX_VALUE.
>
> On 23 December 2017 at 02:46, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > Your observation is correct. KafkaConsumer.position() is a blocking
> > call. It's a know issue that there is no configurable timeout value.
> >
> > I am not aware of any workaround.
> >
> >
> > -Matthias
> >
> > On 12/21/17 6:05 AM, sahil aggarwal wrote:
> > > Hi,
> > >
> > > Facing issue where *kafka-consumer-groups.sh --describe * get stuck if
> > one
> > > of the partition is unavailable i.e no leader. Going through some code
> > > found that it does following to get log end offset:
> > >
> > > * Create consumer
> > > * For each partition
> > > * assign partition
> > > * seek to end
> > > * get position
> > >
> > > Issue is KafkaConsumer.posiiton() use Fetcher.retrieveOffsetsByTimes()
> > > internally which is called with timeout Long.MAX_VALUE and it gets
> stuck
> > in
> > > loop there.
> > >
> > >
> > > Any pointers?
> > >
> > >
> > > *Version*: 0.10.0.1
> > >
> > >
> > > Thanks,
> > > Sahil
> > >
> >
> >
>
Re: Consumer group describe issue
Posted by sahil aggarwal <sa...@gmail.com>.
Fixed it by some code change in ConsumerGroupCommand.scala. Possible to
push it upstream for 0.10.* ?
It seems to be fixed in 0.11.* where it uses getEndOffsets() which has
requestTimeoutMs instead of Long.MAX_VALUE.
On 23 December 2017 at 02:46, Matthias J. Sax <ma...@confluent.io> wrote:
> Your observation is correct. KafkaConsumer.position() is a blocking
> call. It's a know issue that there is no configurable timeout value.
>
> I am not aware of any workaround.
>
>
> -Matthias
>
> On 12/21/17 6:05 AM, sahil aggarwal wrote:
> > Hi,
> >
> > Facing issue where *kafka-consumer-groups.sh --describe * get stuck if
> one
> > of the partition is unavailable i.e no leader. Going through some code
> > found that it does following to get log end offset:
> >
> > * Create consumer
> > * For each partition
> > * assign partition
> > * seek to end
> > * get position
> >
> > Issue is KafkaConsumer.posiiton() use Fetcher.retrieveOffsetsByTimes()
> > internally which is called with timeout Long.MAX_VALUE and it gets stuck
> in
> > loop there.
> >
> >
> > Any pointers?
> >
> >
> > *Version*: 0.10.0.1
> >
> >
> > Thanks,
> > Sahil
> >
>
>
Re: Consumer group describe issue
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Your observation is correct. KafkaConsumer.position() is a blocking
call. It's a know issue that there is no configurable timeout value.
I am not aware of any workaround.
-Matthias
On 12/21/17 6:05 AM, sahil aggarwal wrote:
> Hi,
>
> Facing issue where *kafka-consumer-groups.sh --describe * get stuck if one
> of the partition is unavailable i.e no leader. Going through some code
> found that it does following to get log end offset:
>
> * Create consumer
> * For each partition
> * assign partition
> * seek to end
> * get position
>
> Issue is KafkaConsumer.posiiton() use Fetcher.retrieveOffsetsByTimes()
> internally which is called with timeout Long.MAX_VALUE and it gets stuck in
> loop there.
>
>
> Any pointers?
>
>
> *Version*: 0.10.0.1
>
>
> Thanks,
> Sahil
>