You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by kant kodali <ka...@gmail.com> on 2017/05/16 09:06:31 UTC
is anyone able to create 1M or 10M or 100M or 1B partitions in a topic?
Hi All,
1. I was wondering if anyone has seen or heard or able to create 1M or 10M
or 100M or 1B partitions in a topic? I understand lot of this depends on
filesystem limitations (we are using ext4) and the OS limitations but I
just would like to know what is the scale one had seen in production?
2. Is it advisable?
Thanks!
Re: is anyone able to create 1M or 10M or 100M or 1B partitions in a topic?
Posted by Ofir Manor <of...@equalum.io>.
Each partition is a directory, whose name is topicName-partitionNumber
In each directory, there are likely many files, based on how you configure
the Kafka log rotation and retention (check log.roll.* parameters, for
starter), plus some internal files (indexing etc)
You can easily try it out yourself - create a topic with some partitions,
send some data and observe the log dir(s) (based on log.dir or log.dirs in
the broker config)
Ofir Manor
Co-Founder & CTO | Equalum
Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io
On Tue, May 16, 2017 at 12:44 PM, kant kodali <ka...@gmail.com> wrote:
> Got it! but do you mean directories or files? I thought partition = file
> and topic = directory
>
> If there are 10000 partitions in a topic that means there are 10000 files
> in one directory?
>
> On Tue, May 16, 2017 at 2:29 AM, Sameer Kumar <sa...@gmail.com>
> wrote:
>
> > I don't think this would be the right approach. from broker side, this
> > would mean creating 1M/10M/100M/1B directories, this would be too much
> for
> > the file system itself.
> >
> > For most cases, even some thousand partitions per node should be
> > sufficient.
> >
> > For more details, please refer to
> > https://www.confluent.io/blog/how-to-choose-the-number-of-
> > topicspartitions-in-a-kafka-cluster/
> >
> > -Sameer.
> >
> > On Tue, May 16, 2017 at 2:40 PM, kant kodali <ka...@gmail.com> wrote:
> >
> > > Forgot to mention: The question in this thread is for one node which
> has
> > 8
> > > CPU's 16GB RAM & 500GB hard disk space.
> > >
> > > On Tue, May 16, 2017 at 2:06 AM, kant kodali <ka...@gmail.com>
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > 1. I was wondering if anyone has seen or heard or able to create 1M
> or
> > > 10M
> > > > or 100M or 1B partitions in a topic? I understand lot of this depends
> > on
> > > > filesystem limitations (we are using ext4) and the OS limitations
> but I
> > > > just would like to know what is the scale one had seen in production?
> > > > 2. Is it advisable?
> > > >
> > > > Thanks!
> > > >
> > >
> >
>
Re: is anyone able to create 1M or 10M or 100M or 1B partitions in a topic?
Posted by kant kodali <ka...@gmail.com>.
Got it! but do you mean directories or files? I thought partition = file
and topic = directory
If there are 10000 partitions in a topic that means there are 10000 files
in one directory?
On Tue, May 16, 2017 at 2:29 AM, Sameer Kumar <sa...@gmail.com>
wrote:
> I don't think this would be the right approach. from broker side, this
> would mean creating 1M/10M/100M/1B directories, this would be too much for
> the file system itself.
>
> For most cases, even some thousand partitions per node should be
> sufficient.
>
> For more details, please refer to
> https://www.confluent.io/blog/how-to-choose-the-number-of-
> topicspartitions-in-a-kafka-cluster/
>
> -Sameer.
>
> On Tue, May 16, 2017 at 2:40 PM, kant kodali <ka...@gmail.com> wrote:
>
> > Forgot to mention: The question in this thread is for one node which has
> 8
> > CPU's 16GB RAM & 500GB hard disk space.
> >
> > On Tue, May 16, 2017 at 2:06 AM, kant kodali <ka...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > 1. I was wondering if anyone has seen or heard or able to create 1M or
> > 10M
> > > or 100M or 1B partitions in a topic? I understand lot of this depends
> on
> > > filesystem limitations (we are using ext4) and the OS limitations but I
> > > just would like to know what is the scale one had seen in production?
> > > 2. Is it advisable?
> > >
> > > Thanks!
> > >
> >
>
Re: is anyone able to create 1M or 10M or 100M or 1B partitions in a topic?
Posted by Sameer Kumar <sa...@gmail.com>.
I don't think this would be the right approach. from broker side, this
would mean creating 1M/10M/100M/1B directories, this would be too much for
the file system itself.
For most cases, even some thousand partitions per node should be
sufficient.
For more details, please refer to
https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
-Sameer.
On Tue, May 16, 2017 at 2:40 PM, kant kodali <ka...@gmail.com> wrote:
> Forgot to mention: The question in this thread is for one node which has 8
> CPU's 16GB RAM & 500GB hard disk space.
>
> On Tue, May 16, 2017 at 2:06 AM, kant kodali <ka...@gmail.com> wrote:
>
> > Hi All,
> >
> > 1. I was wondering if anyone has seen or heard or able to create 1M or
> 10M
> > or 100M or 1B partitions in a topic? I understand lot of this depends on
> > filesystem limitations (we are using ext4) and the OS limitations but I
> > just would like to know what is the scale one had seen in production?
> > 2. Is it advisable?
> >
> > Thanks!
> >
>
Re: is anyone able to create 1M or 10M or 100M or 1B partitions in a topic?
Posted by kant kodali <ka...@gmail.com>.
Forgot to mention: The question in this thread is for one node which has 8
CPU's 16GB RAM & 500GB hard disk space.
On Tue, May 16, 2017 at 2:06 AM, kant kodali <ka...@gmail.com> wrote:
> Hi All,
>
> 1. I was wondering if anyone has seen or heard or able to create 1M or 10M
> or 100M or 1B partitions in a topic? I understand lot of this depends on
> filesystem limitations (we are using ext4) and the OS limitations but I
> just would like to know what is the scale one had seen in production?
> 2. Is it advisable?
>
> Thanks!
>