You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by hacker win7 <ha...@gmail.com> on 2018/11/01 07:28:25 UTC
consumer fetch multiple topic partitions committed offset
Hi,
After reviewing the KafkaConsumer source about API of *committed():*
I found that old consumer support committed(mutipleTopicPartitions) to
return multiple committed offset, while in new consumer, there is only
committed(singleTopicPartition) and return only one committed offset.
It is a little weird for me that why new consumer only support fetch single
topic partition committed offset. I search some KIPs but didn't find the
reason about this. Anyway, How to fetch multiple topic partitions committed
offset in new consumer?
Re: consumer fetch multiple topic partitions committed offset
Posted by "Matthias J. Sax" <ma...@confluent.io>.
I am not sure why the API is this way. I agree, that it seems reasonable
to do a single request for multiple topic partitions.
I cc'ed Jason. He might know more.
-Matthias
On 11/2/18 3:31 AM, hacker win7 wrote:
> Thanks for your reply, I want to know why new consumer did not add the committed(TopicPartitions)? Just modify the source in committed(TopicPartition, Duration) from ‘coordinator.fetchCommittedOffsets(Collections.single)’ to ‘coordinator.fetchCommittedOffsets(TopicPartitions)’.
> Besides, call multiple times with committed() will send multiple request to coordinator, maybe block in the middle request, why not accumulate the multiple TopicPartitions and send one fetch offset request to coordinator?
>
>> On Nov 2, 2018, at 04:20, Matthias J. Sax <ma...@confluent.io> wrote:
>>
>> You need to call `committed()` multiple times.
>>
>> -Matthias
>>
>> On 11/1/18 12:28 AM, hacker win7 wrote:
>>> Hi,
>>>
>>> After reviewing the KafkaConsumer source about API of *committed():*
>>> I found that old consumer support committed(mutipleTopicPartitions) to
>>> return multiple committed offset, while in new consumer, there is only
>>> committed(singleTopicPartition) and return only one committed offset.
>>>
>>> It is a little weird for me that why new consumer only support fetch single
>>> topic partition committed offset. I search some KIPs but didn't find the
>>> reason about this. Anyway, How to fetch multiple topic partitions committed
>>> offset in new consumer?
>>>
>>
>
Re: consumer fetch multiple topic partitions committed offset
Posted by hacker win7 <ha...@gmail.com>.
Thanks for your reply, I want to know why new consumer did not add the committed(TopicPartitions)? Just modify the source in committed(TopicPartition, Duration) from ‘coordinator.fetchCommittedOffsets(Collections.single)’ to ‘coordinator.fetchCommittedOffsets(TopicPartitions)’.
Besides, call multiple times with committed() will send multiple request to coordinator, maybe block in the middle request, why not accumulate the multiple TopicPartitions and send one fetch offset request to coordinator?
> On Nov 2, 2018, at 04:20, Matthias J. Sax <ma...@confluent.io> wrote:
>
> You need to call `committed()` multiple times.
>
> -Matthias
>
> On 11/1/18 12:28 AM, hacker win7 wrote:
>> Hi,
>>
>> After reviewing the KafkaConsumer source about API of *committed():*
>> I found that old consumer support committed(mutipleTopicPartitions) to
>> return multiple committed offset, while in new consumer, there is only
>> committed(singleTopicPartition) and return only one committed offset.
>>
>> It is a little weird for me that why new consumer only support fetch single
>> topic partition committed offset. I search some KIPs but didn't find the
>> reason about this. Anyway, How to fetch multiple topic partitions committed
>> offset in new consumer?
>>
>
Re: consumer fetch multiple topic partitions committed offset
Posted by "Matthias J. Sax" <ma...@confluent.io>.
You need to call `committed()` multiple times.
-Matthias
On 11/1/18 12:28 AM, hacker win7 wrote:
> Hi,
>
> After reviewing the KafkaConsumer source about API of *committed():*
> I found that old consumer support committed(mutipleTopicPartitions) to
> return multiple committed offset, while in new consumer, there is only
> committed(singleTopicPartition) and return only one committed offset.
>
> It is a little weird for me that why new consumer only support fetch single
> topic partition committed offset. I search some KIPs but didn't find the
> reason about this. Anyway, How to fetch multiple topic partitions committed
> offset in new consumer?
>