You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Laxmi Narayan <ni...@gmail.com> on 2017/03/31 05:51:59 UTC
how to listen each partition with separate threads ?
Hi ,
I was thinking to listen each partition with separate thread in Kafka.
But i get error saying :
*org.apache.kafka.clients.consumer.KafkaConsumer@383ad023KafkaConsumer is
not safe for multi-threaded accessSubscription to topics, partitions and
pattern are mutually exclusive.*
Is there any way to achieve this ?
**
Keep learning keep moving .....
Re: how to listen each partition with separate threads ?
Posted by "Matthias J. Sax" <ma...@confluent.io>.
That sounds a lot -- even if brokers can handle quite some load. How
many brokers do you got?
Main question: why do you need a consumer per partition? If you
processing super expensive such that one consumer cannot handle multiple
partitions?
-Matthias
On 3/31/17 5:02 AM, Laxmi Narayan wrote:
> Hi,
>
> Is there any performance downside of creating so many consumers ?
>
> I mean literally I am gonna create atleast 7k connections in that case , I
> have nearly 7k partitions with a given topic.
>
>
> Keep learning keep moving .....
>
> On Fri, Mar 31, 2017 at 12:48 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> You need to create a KafkaConsumer per thread.
>>
>> -Matthias
>>
>> On 3/30/17 10:51 PM, Laxmi Narayan wrote:
>>> Hi ,
>>>
>>> I was thinking to listen each partition with separate thread in Kafka.
>>> But i get error saying :
>>>
>>>
>>>
>>>
>>> *org.apache.kafka.clients.consumer.KafkaConsumer@383ad023KafkaConsumer
>> is
>>> not safe for multi-threaded accessSubscription to topics, partitions and
>>> pattern are mutually exclusive.*
>>>
>>> Is there any way to achieve this ?
>>>
>>>
>>> **
>>> Keep learning keep moving .....
>>>
>>
>>
>
Re: how to listen each partition with separate threads ?
Posted by Laxmi Narayan <ni...@gmail.com>.
Hi,
Is there any performance downside of creating so many consumers ?
I mean literally I am gonna create atleast 7k connections in that case , I
have nearly 7k partitions with a given topic.
Keep learning keep moving .....
On Fri, Mar 31, 2017 at 12:48 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> You need to create a KafkaConsumer per thread.
>
> -Matthias
>
> On 3/30/17 10:51 PM, Laxmi Narayan wrote:
> > Hi ,
> >
> > I was thinking to listen each partition with separate thread in Kafka.
> > But i get error saying :
> >
> >
> >
> >
> > *org.apache.kafka.clients.consumer.KafkaConsumer@383ad023KafkaConsumer
> is
> > not safe for multi-threaded accessSubscription to topics, partitions and
> > pattern are mutually exclusive.*
> >
> > Is there any way to achieve this ?
> >
> >
> > **
> > Keep learning keep moving .....
> >
>
>
Re: how to listen each partition with separate threads ?
Posted by "Matthias J. Sax" <ma...@confluent.io>.
You need to create a KafkaConsumer per thread.
-Matthias
On 3/30/17 10:51 PM, Laxmi Narayan wrote:
> Hi ,
>
> I was thinking to listen each partition with separate thread in Kafka.
> But i get error saying :
>
>
>
>
> *org.apache.kafka.clients.consumer.KafkaConsumer@383ad023KafkaConsumer is
> not safe for multi-threaded accessSubscription to topics, partitions and
> pattern are mutually exclusive.*
>
> Is there any way to achieve this ?
>
>
> **
> Keep learning keep moving .....
>