You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jungyeol Lee <ju...@skelterlabs.com> on 2019/10/26 07:10:20 UTC
On the number of partitions.
I'm running a kafka cluster running only one broker with GCP n1-standard-2
instance.
I configured the number of partitions to 1000, but it is suffering from
running a broker on startup. It seems that it needs much more time for
topic creation. So, what I did is just set the smaller number of partitions
which is now 32. Everything works okay for now. Is this normal behavior or
performance? Or, are there any guidelines on setting the number of
partitions?
Best,
--
Re: On the number of partitions.
Posted by Jungyeol Lee <ju...@skelterlabs.com>.
Thanks for all the advice. I just thought that 1000 partitions are not that
big, but it turns out it is not :)
Best,
On Sun, Oct 27, 2019 at 6:37 AM Todd Palino <tp...@gmail.com> wrote:
> Going to pile on here, and also say take that blog post with a grain of
> salt :)
>
> Look at your data size, and your desired degree of parallelism for
> consumers, and guide to that. Yes, 1000 partitions takes a while to create.
> But unless your data size is something like 1 TB per day or more, you
> probably don't need that.
>
> -Todd
>
>
> On Sat, Oct 26, 2019, 1:19 PM Edward Capriolo <ed...@gmail.com>
> wrote:
>
> > On Saturday, October 26, 2019, M. Manna <ma...@gmail.com> wrote:
> >
> > > You should also check out Becket Qin’s presentation on producer
> > performance
> > > tuning on YouTube. Both these items should give you all positives and
> > > negatives of having many/less portions.
> > >
> > > Thanks,
> > >
> > > On Sat, 26 Oct 2019 at 09:19, Manasvi Gupta <fa...@gmail.com>
> wrote:
> > >
> > > >
> > > > https://www.confluent.io/blog/how-choose-number-topics-
> > > partitions-kafka-cluster
> > > >
> > > > This might be useful for you.
> > > >
> > > > On Sat 26 Oct, 2019, 12:40 PM Jungyeol Lee, <
> jungyeol@skelterlabs.com>
> > > > wrote:
> > > >
> > > > > I'm running a kafka cluster running only one broker with GCP
> > > > n1-standard-2
> > > > > instance.
> > > > > I configured the number of partitions to 1000, but it is suffering
> > from
> > > > > running a broker on startup. It seems that it needs much more time
> > for
> > > > > topic creation. So, what I did is just set the smaller number of
> > > > partitions
> > > > > which is now 32. Everything works okay for now. Is this normal
> > behavior
> > > > or
> > > > > performance? Or, are there any guidelines on setting the number of
> > > > > partitions?
> > > > >
> > > > > Best,
> > > > > --
> > > > >
> > > >
> > >
> >
> > To many partitions is too many files and folders and file systems and
> > longer failovers.
> >
> >
> > --
> > Sorry this was sent from mobile. Will do less grammar and spell check
> than
> > usual.
> >
>
Re: On the number of partitions.
Posted by Todd Palino <tp...@gmail.com>.
Going to pile on here, and also say take that blog post with a grain of
salt :)
Look at your data size, and your desired degree of parallelism for
consumers, and guide to that. Yes, 1000 partitions takes a while to create.
But unless your data size is something like 1 TB per day or more, you
probably don't need that.
-Todd
On Sat, Oct 26, 2019, 1:19 PM Edward Capriolo <ed...@gmail.com> wrote:
> On Saturday, October 26, 2019, M. Manna <ma...@gmail.com> wrote:
>
> > You should also check out Becket Qin’s presentation on producer
> performance
> > tuning on YouTube. Both these items should give you all positives and
> > negatives of having many/less portions.
> >
> > Thanks,
> >
> > On Sat, 26 Oct 2019 at 09:19, Manasvi Gupta <fa...@gmail.com> wrote:
> >
> > >
> > > https://www.confluent.io/blog/how-choose-number-topics-
> > partitions-kafka-cluster
> > >
> > > This might be useful for you.
> > >
> > > On Sat 26 Oct, 2019, 12:40 PM Jungyeol Lee, <ju...@skelterlabs.com>
> > > wrote:
> > >
> > > > I'm running a kafka cluster running only one broker with GCP
> > > n1-standard-2
> > > > instance.
> > > > I configured the number of partitions to 1000, but it is suffering
> from
> > > > running a broker on startup. It seems that it needs much more time
> for
> > > > topic creation. So, what I did is just set the smaller number of
> > > partitions
> > > > which is now 32. Everything works okay for now. Is this normal
> behavior
> > > or
> > > > performance? Or, are there any guidelines on setting the number of
> > > > partitions?
> > > >
> > > > Best,
> > > > --
> > > >
> > >
> >
>
> To many partitions is too many files and folders and file systems and
> longer failovers.
>
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>
Re: On the number of partitions.
Posted by Edward Capriolo <ed...@gmail.com>.
On Saturday, October 26, 2019, M. Manna <ma...@gmail.com> wrote:
> You should also check out Becket Qin’s presentation on producer performance
> tuning on YouTube. Both these items should give you all positives and
> negatives of having many/less portions.
>
> Thanks,
>
> On Sat, 26 Oct 2019 at 09:19, Manasvi Gupta <fa...@gmail.com> wrote:
>
> >
> > https://www.confluent.io/blog/how-choose-number-topics-
> partitions-kafka-cluster
> >
> > This might be useful for you.
> >
> > On Sat 26 Oct, 2019, 12:40 PM Jungyeol Lee, <ju...@skelterlabs.com>
> > wrote:
> >
> > > I'm running a kafka cluster running only one broker with GCP
> > n1-standard-2
> > > instance.
> > > I configured the number of partitions to 1000, but it is suffering from
> > > running a broker on startup. It seems that it needs much more time for
> > > topic creation. So, what I did is just set the smaller number of
> > partitions
> > > which is now 32. Everything works okay for now. Is this normal behavior
> > or
> > > performance? Or, are there any guidelines on setting the number of
> > > partitions?
> > >
> > > Best,
> > > --
> > >
> >
>
To many partitions is too many files and folders and file systems and
longer failovers.
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: On the number of partitions.
Posted by "M. Manna" <ma...@gmail.com>.
You should also check out Becket Qin’s presentation on producer performance
tuning on YouTube. Both these items should give you all positives and
negatives of having many/less portions.
Thanks,
On Sat, 26 Oct 2019 at 09:19, Manasvi Gupta <fa...@gmail.com> wrote:
>
> https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster
>
> This might be useful for you.
>
> On Sat 26 Oct, 2019, 12:40 PM Jungyeol Lee, <ju...@skelterlabs.com>
> wrote:
>
> > I'm running a kafka cluster running only one broker with GCP
> n1-standard-2
> > instance.
> > I configured the number of partitions to 1000, but it is suffering from
> > running a broker on startup. It seems that it needs much more time for
> > topic creation. So, what I did is just set the smaller number of
> partitions
> > which is now 32. Everything works okay for now. Is this normal behavior
> or
> > performance? Or, are there any guidelines on setting the number of
> > partitions?
> >
> > Best,
> > --
> >
>
Re: On the number of partitions.
Posted by Manasvi Gupta <fa...@gmail.com>.
https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster
This might be useful for you.
On Sat 26 Oct, 2019, 12:40 PM Jungyeol Lee, <ju...@skelterlabs.com>
wrote:
> I'm running a kafka cluster running only one broker with GCP n1-standard-2
> instance.
> I configured the number of partitions to 1000, but it is suffering from
> running a broker on startup. It seems that it needs much more time for
> topic creation. So, what I did is just set the smaller number of partitions
> which is now 32. Everything works okay for now. Is this normal behavior or
> performance? Or, are there any guidelines on setting the number of
> partitions?
>
> Best,
> --
>