You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jun Rao <ju...@confluent.io> on 2018/11/02 18:50:27 UTC
Apache Kafka blog on more partitions support
Hi, Everyone,
The follow is the preview of a blog on Kafka supporting more partitions.
https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt
Please let me know if you have any comments by Tuesday.
Thanks,
Jun
Re: Apache Kafka blog on more partitions support
Posted by Jun Rao <ju...@confluent.io>.
Hi, Harsha,
The recommendation is the upper bound for partitions. One can certainly
have more than 50 brokers in a cluster if there are fewer partitions per
broker. I will clarify this in the blog.
Thanks,
Jun
On Fri, Nov 2, 2018 at 12:07 PM, <ha...@fastmail.fm> wrote:
> Thanks Jun for sharing the post.
> Minor Nit: Date says December 16, 2019.
>
> Did this test measured the replication affects on the overall cluster
> health and performance?
> It looks like we are suggesting with 200k partitions and 4k per broker max
> size of a cluster should be around 50 brokers?
>
> Thanks,
> Harsha
> On Nov 2, 2018, 11:50 AM -0700, Jun Rao <ju...@confluent.io>, wrote:
> > Hi, Everyone,
> >
> > The follow is the preview of a blog on Kafka supporting more partitions.
> >
> > https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt
> >
> > Please let me know if you have any comments by Tuesday.
> >
> > Thanks,
> >
> > Jun
>
Re: Apache Kafka blog on more partitions support
Posted by Jun Rao <ju...@confluent.io>.
The blog is now published to
https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
Thanks,
Jun
On Fri, Nov 2, 2018 at 2:36 PM, Jun Rao <ju...@confluent.io> wrote:
> Hi, Mayuresh,
>
> Most of the controlled shutdown time is in leader election. The controller
> currently doesn't wait for LeaderAndIsrRequest to be sent out before
> responding to the controlled shutdown request.
>
> Thanks,
>
> Jun
>
> On Fri, Nov 2, 2018 at 1:58 PM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com> wrote:
>
>> Thanks Jun for sharing this. Looks nice !
>>
>> Do we intend to shed light on how much time is required, on an average,
>> for
>> new Leader election. Also would it be good to add "if the controller waits
>> for the LeaderAndIsrResponses before sending shutDown_OK to the shutting
>> down broker".
>>
>> Thanks,
>>
>> Mayuresh
>>
>> On Fri, Nov 2, 2018 at 12:07 PM <ha...@fastmail.fm> wrote:
>>
>> > Thanks Jun for sharing the post.
>> > Minor Nit: Date says December 16, 2019.
>> >
>> > Did this test measured the replication affects on the overall cluster
>> > health and performance?
>> > It looks like we are suggesting with 200k partitions and 4k per broker
>> max
>> > size of a cluster should be around 50 brokers?
>> >
>> > Thanks,
>> > Harsha
>> > On Nov 2, 2018, 11:50 AM -0700, Jun Rao <ju...@confluent.io>, wrote:
>> > > Hi, Everyone,
>> > >
>> > > The follow is the preview of a blog on Kafka supporting more
>> partitions.
>> > >
>> > > https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt
>> > >
>> > > Please let me know if you have any comments by Tuesday.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> >
>>
>>
>> --
>> -Regards,
>> Mayuresh R. Gharat
>> (862) 250-7125
>>
>
>
Re: Apache Kafka blog on more partitions support
Posted by Jun Rao <ju...@confluent.io>.
Hi, Mayuresh,
Most of the controlled shutdown time is in leader election. The controller
currently doesn't wait for LeaderAndIsrRequest to be sent out before
responding to the controlled shutdown request.
Thanks,
Jun
On Fri, Nov 2, 2018 at 1:58 PM, Mayuresh Gharat <gh...@gmail.com>
wrote:
> Thanks Jun for sharing this. Looks nice !
>
> Do we intend to shed light on how much time is required, on an average, for
> new Leader election. Also would it be good to add "if the controller waits
> for the LeaderAndIsrResponses before sending shutDown_OK to the shutting
> down broker".
>
> Thanks,
>
> Mayuresh
>
> On Fri, Nov 2, 2018 at 12:07 PM <ha...@fastmail.fm> wrote:
>
> > Thanks Jun for sharing the post.
> > Minor Nit: Date says December 16, 2019.
> >
> > Did this test measured the replication affects on the overall cluster
> > health and performance?
> > It looks like we are suggesting with 200k partitions and 4k per broker
> max
> > size of a cluster should be around 50 brokers?
> >
> > Thanks,
> > Harsha
> > On Nov 2, 2018, 11:50 AM -0700, Jun Rao <ju...@confluent.io>, wrote:
> > > Hi, Everyone,
> > >
> > > The follow is the preview of a blog on Kafka supporting more
> partitions.
> > >
> > > https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt
> > >
> > > Please let me know if you have any comments by Tuesday.
> > >
> > > Thanks,
> > >
> > > Jun
> >
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Apache Kafka blog on more partitions support
Posted by Mayuresh Gharat <gh...@gmail.com>.
Thanks Jun for sharing this. Looks nice !
Do we intend to shed light on how much time is required, on an average, for
new Leader election. Also would it be good to add "if the controller waits
for the LeaderAndIsrResponses before sending shutDown_OK to the shutting
down broker".
Thanks,
Mayuresh
On Fri, Nov 2, 2018 at 12:07 PM <ha...@fastmail.fm> wrote:
> Thanks Jun for sharing the post.
> Minor Nit: Date says December 16, 2019.
>
> Did this test measured the replication affects on the overall cluster
> health and performance?
> It looks like we are suggesting with 200k partitions and 4k per broker max
> size of a cluster should be around 50 brokers?
>
> Thanks,
> Harsha
> On Nov 2, 2018, 11:50 AM -0700, Jun Rao <ju...@confluent.io>, wrote:
> > Hi, Everyone,
> >
> > The follow is the preview of a blog on Kafka supporting more partitions.
> >
> > https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt
> >
> > Please let me know if you have any comments by Tuesday.
> >
> > Thanks,
> >
> > Jun
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Apache Kafka blog on more partitions support
Posted by ha...@fastmail.fm.
Thanks Jun for sharing the post.
Minor Nit: Date says December 16, 2019.
Did this test measured the replication affects on the overall cluster health and performance?
It looks like we are suggesting with 200k partitions and 4k per broker max size of a cluster should be around 50 brokers?
Thanks,
Harsha
On Nov 2, 2018, 11:50 AM -0700, Jun Rao <ju...@confluent.io>, wrote:
> Hi, Everyone,
>
> The follow is the preview of a blog on Kafka supporting more partitions.
>
> https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt
>
> Please let me know if you have any comments by Tuesday.
>
> Thanks,
>
> Jun