You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Suyog Rao <su...@gmail.com> on 2013/03/27 07:52:04 UTC
Limit on number of Kafka Topic
Hello,
Wanted to check if there is any known limit on the # of topics in a Kafka
cluster? I wanted to design a system which has say 5k topics and
multi-threaded consumers reading messages from these topics. Does anyone
have experience with such large topic size? I see in Kafka's page a test
for throughput w.r.t. to topic size, but the max topic size is 50.
Thanks
Re: Limit on number of Kafka Topic
Posted by Matthew Rathbone <ma...@foursquare.com>.
We've seen big performace degradation when we tested 1024 topics, so we've
opted to go for a much smaller topic count (< 100).
On the read side, I think performance is largely driven by the ability of
the operating system to effectively cache access to #partitions*topic
files. Clearly if you divide your available memory by 1024 topics you'll
have less in the file cache per topic than you would with only 3 topics
(for example).
On Wed, Mar 27, 2013 at 11:14 AM, Jun Rao <ju...@gmail.com> wrote:
> With more topics, you may hit one of those limits: (1) # dirs allowed in a
> FS; (2) open file handlers (we keep all log segments open in the broker);
> (3) ZK nodes.
>
> Thanks,
>
> Jun
>
> On Wed, Mar 27, 2013 at 8:36 AM, Jason Huang <ja...@icare.com>
> wrote:
>
> > Just curious - I don't see any limit in the # of topics that you can
> > have in a Kafka cluster.
> >
> > So in principle, you could have as many topics as you want, so long as
> > your hardware can keep up, right?
> >
> > Jason
> >
> > On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <ju...@gmail.com> wrote:
> > > At LinkedIn, our largest cluster has more than 2K topics. 5K topics
> > should
> > > be fine.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <su...@gmail.com>
> wrote:
> > >
> > >> Hello,
> > >>
> > >> Wanted to check if there is any known limit on the # of topics in a
> > Kafka
> > >> cluster? I wanted to design a system which has say 5k topics and
> > >> multi-threaded consumers reading messages from these topics. Does
> anyone
> > >> have experience with such large topic size? I see in Kafka's page a
> test
> > >> for throughput w.r.t. to topic size, but the max topic size is 50.
> > >>
> > >> Thanks
> > >>
> >
>
--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
matthew@foursquare.com | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>
Re: Limit on number of Kafka Topic
Posted by Jun Rao <ju...@gmail.com>.
With more topics, you may hit one of those limits: (1) # dirs allowed in a
FS; (2) open file handlers (we keep all log segments open in the broker);
(3) ZK nodes.
Thanks,
Jun
On Wed, Mar 27, 2013 at 8:36 AM, Jason Huang <ja...@icare.com> wrote:
> Just curious - I don't see any limit in the # of topics that you can
> have in a Kafka cluster.
>
> So in principle, you could have as many topics as you want, so long as
> your hardware can keep up, right?
>
> Jason
>
> On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <ju...@gmail.com> wrote:
> > At LinkedIn, our largest cluster has more than 2K topics. 5K topics
> should
> > be fine.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <su...@gmail.com> wrote:
> >
> >> Hello,
> >>
> >> Wanted to check if there is any known limit on the # of topics in a
> Kafka
> >> cluster? I wanted to design a system which has say 5k topics and
> >> multi-threaded consumers reading messages from these topics. Does anyone
> >> have experience with such large topic size? I see in Kafka's page a test
> >> for throughput w.r.t. to topic size, but the max topic size is 50.
> >>
> >> Thanks
> >>
>
Re: Limit on number of Kafka Topic
Posted by Jason Huang <ja...@icare.com>.
Just curious - I don't see any limit in the # of topics that you can
have in a Kafka cluster.
So in principle, you could have as many topics as you want, so long as
your hardware can keep up, right?
Jason
On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <ju...@gmail.com> wrote:
> At LinkedIn, our largest cluster has more than 2K topics. 5K topics should
> be fine.
>
> Thanks,
>
> Jun
>
> On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <su...@gmail.com> wrote:
>
>> Hello,
>>
>> Wanted to check if there is any known limit on the # of topics in a Kafka
>> cluster? I wanted to design a system which has say 5k topics and
>> multi-threaded consumers reading messages from these topics. Does anyone
>> have experience with such large topic size? I see in Kafka's page a test
>> for throughput w.r.t. to topic size, but the max topic size is 50.
>>
>> Thanks
>>
Re: Limit on number of Kafka Topic
Posted by Jun Rao <ju...@gmail.com>.
At LinkedIn, our largest cluster has more than 2K topics. 5K topics should
be fine.
Thanks,
Jun
On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <su...@gmail.com> wrote:
> Hello,
>
> Wanted to check if there is any known limit on the # of topics in a Kafka
> cluster? I wanted to design a system which has say 5k topics and
> multi-threaded consumers reading messages from these topics. Does anyone
> have experience with such large topic size? I see in Kafka's page a test
> for throughput w.r.t. to topic size, but the max topic size is 50.
>
> Thanks
>