You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ming Li <lm...@gmail.com> on 2013/05/13 12:15:47 UTC
Is there a limitation on the number of simultaneous consumer
connections to the same topic
Hi,
Does Kafka have a limitation on the simultaneous connections (created with
Consumer.createJavaConsumerConnector) for the same topic within the same
group?
My scenario is I need to consume a topic from different process (not
thread), so I need to create lots of high level consumers.
Best Regards,
Li Ming
Re: Is there a limitation on the number of simultaneous consumer
connections to the same topic
Posted by Jun Rao <ju...@gmail.com>.
The only limit is #4 in http://kafka.apache.org/faq.html
Thanks,
Jun
On Mon, May 13, 2013 at 3:15 AM, Ming Li <lm...@gmail.com> wrote:
> Hi,
>
> Does Kafka have a limitation on the simultaneous connections (created with
> Consumer.createJavaConsumerConnector) for the same topic within the same
> group?
>
> My scenario is I need to consume a topic from different process (not
> thread), so I need to create lots of high level consumers.
>
>
> Best Regards,
> Li Ming
>
Re: Is there a limitation on the number of simultaneous consumer
connections to the same topic
Posted by Ming Li <lm...@gmail.com>.
Got it and thanks for your explanation~~
Best Regards,
Li Ming
On Mon, May 13, 2013 at 9:10 PM, Chris Curtin <cu...@gmail.com>wrote:
> Yes. However be aware that starting and stopping processes will cause a
> rebalance of the consumers, so your code may find itself receiving events
> from a different partition suddenly (so don't assume the partition you are
> reading isn't going to change!) Also as things are starting up you may
> find a process receives many partitions at first, but as the other
> processes are started the partitions get reassigned.
>
> Finally, running more processes than partitions will mean those other
> processes are idle.
>
>
>
>
> On Mon, May 13, 2013 at 8:49 AM, Ming Li <lm...@gmail.com> wrote:
>
> > Hi Andrea,
> >
> > Thanks for your reply~~ you mean, it is no difference between
> > having N threads share the same ConsumerConnector created by
> > Consumer.createJavaConsumerConnector,
> > and
> > having N consumer process which has its own ConsumerConnector in every
> one
> > of them?
> >
> > Best Regards,
> > Li Ming
> >
> >
> >
> > On Mon, May 13, 2013 at 6:43 PM, Andrea Gazzarini <
> > andrea.gazzarini@gmail.com> wrote:
> >
> > > It shouldn't.
> > > Creating several listener / consumer processes belonging to the same
> > group
> > > means you are working with a point-to-point message channel so incoming
> > > messages will be delivered only to one consumer.
> > >
> > > Maybe I'm wrong but I believe in that scenario there's no difference
> > (from
> > > broker perspective) between threads and processes.
> > >
> > > Regards,
> > > Andrea
> > >
> > >
> > > On 05/13/2013 12:15 PM, Ming Li wrote:
> > >
> > >> Hi,
> > >>
> > >> Does Kafka have a limitation on the simultaneous connections (created
> > with
> > >> Consumer.**createJavaConsumerConnector) for the same topic within the
> > >> same
> > >> group?
> > >>
> > >> My scenario is I need to consume a topic from different process (not
> > >> thread), so I need to create lots of high level consumers.
> > >>
> > >>
> > >> Best Regards,
> > >> Li Ming
> > >>
> > >>
> > >
> >
>
Re: Is there a limitation on the number of simultaneous consumer
connections to the same topic
Posted by Chris Curtin <cu...@gmail.com>.
Yes. However be aware that starting and stopping processes will cause a
rebalance of the consumers, so your code may find itself receiving events
from a different partition suddenly (so don't assume the partition you are
reading isn't going to change!) Also as things are starting up you may
find a process receives many partitions at first, but as the other
processes are started the partitions get reassigned.
Finally, running more processes than partitions will mean those other
processes are idle.
On Mon, May 13, 2013 at 8:49 AM, Ming Li <lm...@gmail.com> wrote:
> Hi Andrea,
>
> Thanks for your reply~~ you mean, it is no difference between
> having N threads share the same ConsumerConnector created by
> Consumer.createJavaConsumerConnector,
> and
> having N consumer process which has its own ConsumerConnector in every one
> of them?
>
> Best Regards,
> Li Ming
>
>
>
> On Mon, May 13, 2013 at 6:43 PM, Andrea Gazzarini <
> andrea.gazzarini@gmail.com> wrote:
>
> > It shouldn't.
> > Creating several listener / consumer processes belonging to the same
> group
> > means you are working with a point-to-point message channel so incoming
> > messages will be delivered only to one consumer.
> >
> > Maybe I'm wrong but I believe in that scenario there's no difference
> (from
> > broker perspective) between threads and processes.
> >
> > Regards,
> > Andrea
> >
> >
> > On 05/13/2013 12:15 PM, Ming Li wrote:
> >
> >> Hi,
> >>
> >> Does Kafka have a limitation on the simultaneous connections (created
> with
> >> Consumer.**createJavaConsumerConnector) for the same topic within the
> >> same
> >> group?
> >>
> >> My scenario is I need to consume a topic from different process (not
> >> thread), so I need to create lots of high level consumers.
> >>
> >>
> >> Best Regards,
> >> Li Ming
> >>
> >>
> >
>
Re: Is there a limitation on the number of simultaneous consumer
connections to the same topic
Posted by Ming Li <lm...@gmail.com>.
Hi Andrea,
Thanks for your reply~~ you mean, it is no difference between
having N threads share the same ConsumerConnector created by
Consumer.createJavaConsumerConnector,
and
having N consumer process which has its own ConsumerConnector in every one
of them?
Best Regards,
Li Ming
On Mon, May 13, 2013 at 6:43 PM, Andrea Gazzarini <
andrea.gazzarini@gmail.com> wrote:
> It shouldn't.
> Creating several listener / consumer processes belonging to the same group
> means you are working with a point-to-point message channel so incoming
> messages will be delivered only to one consumer.
>
> Maybe I'm wrong but I believe in that scenario there's no difference (from
> broker perspective) between threads and processes.
>
> Regards,
> Andrea
>
>
> On 05/13/2013 12:15 PM, Ming Li wrote:
>
>> Hi,
>>
>> Does Kafka have a limitation on the simultaneous connections (created with
>> Consumer.**createJavaConsumerConnector) for the same topic within the
>> same
>> group?
>>
>> My scenario is I need to consume a topic from different process (not
>> thread), so I need to create lots of high level consumers.
>>
>>
>> Best Regards,
>> Li Ming
>>
>>
>
Re: Is there a limitation on the number of simultaneous consumer
connections to the same topic
Posted by Andrea Gazzarini <an...@gmail.com>.
It shouldn't.
Creating several listener / consumer processes belonging to the same
group means you are working with a point-to-point message channel so
incoming messages will be delivered only to one consumer.
Maybe I'm wrong but I believe in that scenario there's no difference
(from broker perspective) between threads and processes.
Regards,
Andrea
On 05/13/2013 12:15 PM, Ming Li wrote:
> Hi,
>
> Does Kafka have a limitation on the simultaneous connections (created with
> Consumer.createJavaConsumerConnector) for the same topic within the same
> group?
>
> My scenario is I need to consume a topic from different process (not
> thread), so I need to create lots of high level consumers.
>
>
> Best Regards,
> Li Ming
>