You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Evelina <st...@gmail.com> on 2013/12/16 16:33:58 UTC

Kafka 0.8 High-Level Consumer: fetcher threads

Hi folks!

I'm trying High-Level Consumer and wondering why it uses only 1 fetcher 
per broker?
Why can't it be scaled on partition basis?

I even see the parameter '/numFetchers/' in /AbstractFetcherManager/ but 
it explicitly set  to 1 in /ConsumerFetcherManager/.

--
Very Best Regards,
Evelina

Re: Kafka 0.8 High-Level Consumer: fetcher threads

Posted by Evelina <st...@gmail.com>.

Hi Jun,

Thank you for the quick feedback!
here is the issue https://issues.apache.org/jira/browse/KAFKA-1184

--
Best Regards,
Evelina

On 12/16/2013 08:18 PM, Jun Rao wrote:
> It's possible to support multi-fetchers per broker in the consumer. Could
> you file a jira to track that? However, typically the bottleneck is not in
> the fetcher, but in decompression, decoding and application logic during
> the consumption.
>
> Thanks,
>
> Jun
>
>
> On Mon, Dec 16, 2013 at 7:33 AM, Evelina <st...@gmail.com> wrote:
>
>> Hi folks!
>>
>> I'm trying High-Level Consumer and wondering why it uses only 1 fetcher
>> per broker?
>> Why can't it be scaled on partition basis?
>>
>> I even see the parameter '/numFetchers/' in /AbstractFetcherManager/ but
>> it explicitly set  to 1 in /ConsumerFetcherManager/.
>>
>> --
>> Very Best Regards,
>> Evelina
>>
>>
>>

Re: Kafka 0.8 High-Level Consumer: fetcher threads

Posted by Jun Rao <ju...@gmail.com>.

It's possible to support multi-fetchers per broker in the consumer. Could
you file a jira to track that? However, typically the bottleneck is not in
the fetcher, but in decompression, decoding and application logic during
the consumption.

Thanks,

Jun

On Mon, Dec 16, 2013 at 7:33 AM, Evelina <st...@gmail.com> wrote:

> Hi folks!
>
> I'm trying High-Level Consumer and wondering why it uses only 1 fetcher
> per broker?
> Why can't it be scaled on partition basis?
>
> I even see the parameter '/numFetchers/' in /AbstractFetcherManager/ but
> it explicitly set  to 1 in /ConsumerFetcherManager/.
>
> --
> Very Best Regards,
> Evelina
>
>
>