You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vaibhav Puranik <vp...@gmail.com> on 2012/10/11 17:20:03 UTC
Number of consumers limited to number of partitions
Hi all,
We have a topic with three partitions - one on each broker (Kafka 0.7.1).
Using ConsumerConnector I noticed that I can consume messages from three
consumers at most. When I start fourth one, one of the older one stops.
Is this as designed? If I want more parallelism, do I need to increase
number of partitions?
Regards,
Vaibhav
GumGum
Re: Number of consumers limited to number of partitions
Posted by Vaibhav Puranik <vp...@gmail.com>.
Thanks Neha and Jay.
It makes sense.
Regards,
Vaibhav
On Thu, Oct 11, 2012 at 8:41 AM, Jay Kreps <ja...@gmail.com> wrote:
> Yes, the reason for this is that if two consumers consumed the same
> partition internally we would have to keep track of per-message consumption
> (which is expensive) and the ordering guarantee would be broken (since the
> actual processing in the two consumer processes might happen out of order).
> Since you can always have more partitions this should not be a limitation,
> just a parameter to be aware of.
>
> -Jay
>
> On Thu, Oct 11, 2012 at 8:20 AM, Vaibhav Puranik <vp...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > We have a topic with three partitions - one on each broker (Kafka 0.7.1).
> > Using ConsumerConnector I noticed that I can consume messages from three
> > consumers at most. When I start fourth one, one of the older one stops.
> >
> > Is this as designed? If I want more parallelism, do I need to increase
> > number of partitions?
> >
> > Regards,
> > Vaibhav
> > GumGum
> >
>
Re: Number of consumers limited to number of partitions
Posted by Jay Kreps <ja...@gmail.com>.
Yes, the reason for this is that if two consumers consumed the same
partition internally we would have to keep track of per-message consumption
(which is expensive) and the ordering guarantee would be broken (since the
actual processing in the two consumer processes might happen out of order).
Since you can always have more partitions this should not be a limitation,
just a parameter to be aware of.
-Jay
On Thu, Oct 11, 2012 at 8:20 AM, Vaibhav Puranik <vp...@gmail.com> wrote:
> Hi all,
>
> We have a topic with three partitions - one on each broker (Kafka 0.7.1).
> Using ConsumerConnector I noticed that I can consume messages from three
> consumers at most. When I start fourth one, one of the older one stops.
>
> Is this as designed? If I want more parallelism, do I need to increase
> number of partitions?
>
> Regards,
> Vaibhav
> GumGum
>
Re: Number of consumers limited to number of partitions
Posted by Neha Narkhede <ne...@gmail.com>.
Hi Vaibhav,
Yes, that is the behavior by design. The parallelism on the consumers
is limited by the number of partitions for the topics that they are
subscribed to.
Thanks,
Neha
On Thu, Oct 11, 2012 at 8:20 AM, Vaibhav Puranik <vp...@gmail.com> wrote:
> Hi all,
>
> We have a topic with three partitions - one on each broker (Kafka 0.7.1).
> Using ConsumerConnector I noticed that I can consume messages from three
> consumers at most. When I start fourth one, one of the older one stops.
>
> Is this as designed? If I want more parallelism, do I need to increase
> number of partitions?
>
> Regards,
> Vaibhav
> GumGum