You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jeff Widman <je...@jeffwidman.com> on 2017/11/13 23:24:49 UTC
What are reasonable limits for max number of consumer groups per
partition and per broker?
We're considering an architecture that would result in 5K-10K consumer
groups consuming from a single topic that has one partition.
What are the reasonable limits for the max number of consumer groups per
partition and per broker?
Can a single broker be the group coordinator for 1K+ consumer groups?
--
*Jeff Widman*
jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
<><
Re: What are reasonable limits for max number of consumer groups per
partition and per broker?
Posted by Avi Levi <av...@indeni.com>.
I think you will find this article
https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
useful
On Tue, Nov 14, 2017 at 6:49 PM, Viktor Somogyi <vi...@gmail.com>
wrote:
> Hi Jeff,
>
> I think it's also worth considering that 1K consumer (implying that you
> have at least 1 consumer per group) for 1 partition means 1K TCP connection
> which means that they have to share the available bandwidth.
> Why do you have so many consumer groups? Do you basically want to
> multicast?
>
> Viktor
>
>
> On Tue, Nov 14, 2017 at 4:18 PM, Arunkumar <pm_arunkumar@yahoo.com.
> invalid>
> wrote:
>
> > Hi Jeff
> > Number of partition depends on number of consumers on that particular
> > consumer group. So you may have to create your partitions based on that.
> >
> > ThanksArunkumar Pichaimuthu, PMP
> >
> > On Monday, November 13, 2017, 5:25:35 PM CST, Jeff Widman <
> > jeff@jeffwidman.com> wrote:
> >
> > We're considering an architecture that would result in 5K-10K consumer
> > groups consuming from a single topic that has one partition.
> >
> > What are the reasonable limits for the max number of consumer groups per
> > partition and per broker?
> >
> > Can a single broker be the group coordinator for 1K+ consumer groups?
> >
> > --
> >
> > *Jeff Widman*
> > jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
> > <><
> >
> >
>
Re: What are reasonable limits for max number of consumer groups per
partition and per broker?
Posted by Viktor Somogyi <vi...@gmail.com>.
Hi Jeff,
I think it's also worth considering that 1K consumer (implying that you
have at least 1 consumer per group) for 1 partition means 1K TCP connection
which means that they have to share the available bandwidth.
Why do you have so many consumer groups? Do you basically want to multicast?
Viktor
On Tue, Nov 14, 2017 at 4:18 PM, Arunkumar <pm...@yahoo.com.invalid>
wrote:
> Hi Jeff
> Number of partition depends on number of consumers on that particular
> consumer group. So you may have to create your partitions based on that.
>
> ThanksArunkumar Pichaimuthu, PMP
>
> On Monday, November 13, 2017, 5:25:35 PM CST, Jeff Widman <
> jeff@jeffwidman.com> wrote:
>
> We're considering an architecture that would result in 5K-10K consumer
> groups consuming from a single topic that has one partition.
>
> What are the reasonable limits for the max number of consumer groups per
> partition and per broker?
>
> Can a single broker be the group coordinator for 1K+ consumer groups?
>
> --
>
> *Jeff Widman*
> jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
> <><
>
>
Re: What are reasonable limits for max number of consumer groups
per partition and per broker?
Posted by Arunkumar <pm...@yahoo.com.INVALID>.
Hi Jeff
Number of partition depends on number of consumers on that particular consumer group. So you may have to create your partitions based on that.
ThanksArunkumar Pichaimuthu, PMP
On Monday, November 13, 2017, 5:25:35 PM CST, Jeff Widman <je...@jeffwidman.com> wrote:
We're considering an architecture that would result in 5K-10K consumer
groups consuming from a single topic that has one partition.
What are the reasonable limits for the max number of consumer groups per
partition and per broker?
Can a single broker be the group coordinator for 1K+ consumer groups?
--
*Jeff Widman*
jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
<><