You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Dana Powers <da...@gmail.com> on 2016/06/19 18:42:33 UTC
Re: [DISCUSS] KIP-53 Add custom policies for reconnect attempts to NetworkdClient
I took a stab at implementing a simplified exponential + randomized
backoff policy here: https://github.com/apache/kafka/pull/1523
Rather than changing public interfaces / using plugins, it just adds a
new client configuration "reconnect.backoff.max" that can be used to
enable increasing backoff when node failures repeat. If this
configuration is not set higher than reconnect.backoff.ms then the
current constant backoff policy is retained. The default is to
continue w/ current 50ms constant backoff.
Thoughts? Would a change like this require a KIP?
-Dana
On Mon, May 2, 2016 at 10:29 AM, Guozhang Wang <wa...@gmail.com> wrote:
> For the specific problem of connection storm, randomized with normal
> distribution at specified mean as "reconnect.backoff.ms" has been proved
> pretty well. The most recent usage of it in my mind is RAFT, and it turns
> out pretty effective in eliminating leader-election storms.
>
>
> Guozhang
>
> On Fri, Apr 29, 2016 at 8:57 PM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
>> I'll agree w/ Jay and point out that the implementations of
>> ReconnectionPolicy provided built-in with that driver are Constant,
>> Exponential, and Counting. Constant and exponential can be combined with
>> the right set of policy config parameters. I'm curious if there's a real
>> need for something else, or if you're just looking for something
>> exponential instead of non-constant? I think a fixed exponential backoff
>> policy that defaults parameters to the current constant backoff policy
>> would probably satisfy our needs.
>>
>> On Mon, Apr 4, 2016 at 1:25 PM, Jay Kreps <ja...@confluent.io> wrote:
>>
>> > If I understand the problem we are fixing is a connection storm where
>> when
>> > a new broker comes online it is overwhelmed with connections.
>> >
>> > In general we try hard to avoid plugins where possible. Maybe instead of
>> > adding another plugin interface we could just directly solve this problem
>> > by doing some randomization in the backoff to space out the
>> reconnections?
>> > This seems like it would be good for anyone with a large client
>> > environment?
>> >
>> > -Jay
>> >
>> > On Mon, Apr 4, 2016 at 12:54 PM, Florian Hussonnois <
>> fhussonnois@gmail.com
>> > >
>> > wrote:
>> >
>> > > Hi Kafka Team,
>> > >
>> > > I have made a new Kafka Improvement Proposal.
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-53+-+Add+custom+policies+for+reconnect+attempts+to+NetworkdClient
>> > >
>> > > This is my first proposal so I don't know if I have given enough
>> > > information.
>> > > Also I have already proposed an implementation :
>> > > https://github.com/apache/kafka/pull/1179
>> > >
>> > > Thanks
>> > >
>> > > --
>> > > Florian HUSSONNOIS
>> > >
>> >
>>
>>
>>
>> --
>> Thanks,
>> Ewen
>>
>
>
>
> --
> -- Guozhang
Re: [DISCUSS] KIP-53 Add custom policies for reconnect attempts to NetworkdClient
Posted by Dana Powers <da...@gmail.com>.
Thanks for the confirmation, Ismael. I will write something up for further
discussion.
-Dana
On Jul 5, 2016 4:43 PM, "Ismael Juma" <is...@juma.me.uk> wrote:
> Hi Dana,
>
> Thanks for the PR. Technically, a (simple) KIP is required when proposing
> new configs.
>
> Ismael
>
> On Sun, Jun 19, 2016 at 7:42 PM, Dana Powers <da...@gmail.com>
> wrote:
>
> > I took a stab at implementing a simplified exponential + randomized
> > backoff policy here: https://github.com/apache/kafka/pull/1523
> >
> > Rather than changing public interfaces / using plugins, it just adds a
> > new client configuration "reconnect.backoff.max" that can be used to
> > enable increasing backoff when node failures repeat. If this
> > configuration is not set higher than reconnect.backoff.ms then the
> > current constant backoff policy is retained. The default is to
> > continue w/ current 50ms constant backoff.
> >
> > Thoughts? Would a change like this require a KIP?
> >
> > -Dana
> >
> >
> > On Mon, May 2, 2016 at 10:29 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
> > > For the specific problem of connection storm, randomized with normal
> > > distribution at specified mean as "reconnect.backoff.ms" has been
> proved
> > > pretty well. The most recent usage of it in my mind is RAFT, and it
> turns
> > > out pretty effective in eliminating leader-election storms.
> > >
> > >
> > > Guozhang
> > >
> > > On Fri, Apr 29, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> > ewen@confluent.io>
> > > wrote:
> > >
> > >> I'll agree w/ Jay and point out that the implementations of
> > >> ReconnectionPolicy provided built-in with that driver are Constant,
> > >> Exponential, and Counting. Constant and exponential can be combined
> with
> > >> the right set of policy config parameters. I'm curious if there's a
> real
> > >> need for something else, or if you're just looking for something
> > >> exponential instead of non-constant? I think a fixed exponential
> backoff
> > >> policy that defaults parameters to the current constant backoff policy
> > >> would probably satisfy our needs.
> > >>
> > >> On Mon, Apr 4, 2016 at 1:25 PM, Jay Kreps <ja...@confluent.io> wrote:
> > >>
> > >> > If I understand the problem we are fixing is a connection storm
> where
> > >> when
> > >> > a new broker comes online it is overwhelmed with connections.
> > >> >
> > >> > In general we try hard to avoid plugins where possible. Maybe
> instead
> > of
> > >> > adding another plugin interface we could just directly solve this
> > problem
> > >> > by doing some randomization in the backoff to space out the
> > >> reconnections?
> > >> > This seems like it would be good for anyone with a large client
> > >> > environment?
> > >> >
> > >> > -Jay
> > >> >
> > >> > On Mon, Apr 4, 2016 at 12:54 PM, Florian Hussonnois <
> > >> fhussonnois@gmail.com
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Hi Kafka Team,
> > >> > >
> > >> > > I have made a new Kafka Improvement Proposal.
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-53+-+Add+custom+policies+for+reconnect+attempts+to+NetworkdClient
> > >> > >
> > >> > > This is my first proposal so I don't know if I have given enough
> > >> > > information.
> > >> > > Also I have already proposed an implementation :
> > >> > > https://github.com/apache/kafka/pull/1179
> > >> > >
> > >> > > Thanks
> > >> > >
> > >> > > --
> > >> > > Florian HUSSONNOIS
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Ewen
> > >>
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
>
Re: [DISCUSS] KIP-53 Add custom policies for reconnect attempts to NetworkdClient
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Dana,
Thanks for the PR. Technically, a (simple) KIP is required when proposing
new configs.
Ismael
On Sun, Jun 19, 2016 at 7:42 PM, Dana Powers <da...@gmail.com> wrote:
> I took a stab at implementing a simplified exponential + randomized
> backoff policy here: https://github.com/apache/kafka/pull/1523
>
> Rather than changing public interfaces / using plugins, it just adds a
> new client configuration "reconnect.backoff.max" that can be used to
> enable increasing backoff when node failures repeat. If this
> configuration is not set higher than reconnect.backoff.ms then the
> current constant backoff policy is retained. The default is to
> continue w/ current 50ms constant backoff.
>
> Thoughts? Would a change like this require a KIP?
>
> -Dana
>
>
> On Mon, May 2, 2016 at 10:29 AM, Guozhang Wang <wa...@gmail.com> wrote:
> > For the specific problem of connection storm, randomized with normal
> > distribution at specified mean as "reconnect.backoff.ms" has been proved
> > pretty well. The most recent usage of it in my mind is RAFT, and it turns
> > out pretty effective in eliminating leader-election storms.
> >
> >
> > Guozhang
> >
> > On Fri, Apr 29, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> ewen@confluent.io>
> > wrote:
> >
> >> I'll agree w/ Jay and point out that the implementations of
> >> ReconnectionPolicy provided built-in with that driver are Constant,
> >> Exponential, and Counting. Constant and exponential can be combined with
> >> the right set of policy config parameters. I'm curious if there's a real
> >> need for something else, or if you're just looking for something
> >> exponential instead of non-constant? I think a fixed exponential backoff
> >> policy that defaults parameters to the current constant backoff policy
> >> would probably satisfy our needs.
> >>
> >> On Mon, Apr 4, 2016 at 1:25 PM, Jay Kreps <ja...@confluent.io> wrote:
> >>
> >> > If I understand the problem we are fixing is a connection storm where
> >> when
> >> > a new broker comes online it is overwhelmed with connections.
> >> >
> >> > In general we try hard to avoid plugins where possible. Maybe instead
> of
> >> > adding another plugin interface we could just directly solve this
> problem
> >> > by doing some randomization in the backoff to space out the
> >> reconnections?
> >> > This seems like it would be good for anyone with a large client
> >> > environment?
> >> >
> >> > -Jay
> >> >
> >> > On Mon, Apr 4, 2016 at 12:54 PM, Florian Hussonnois <
> >> fhussonnois@gmail.com
> >> > >
> >> > wrote:
> >> >
> >> > > Hi Kafka Team,
> >> > >
> >> > > I have made a new Kafka Improvement Proposal.
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-53+-+Add+custom+policies+for+reconnect+attempts+to+NetworkdClient
> >> > >
> >> > > This is my first proposal so I don't know if I have given enough
> >> > > information.
> >> > > Also I have already proposed an implementation :
> >> > > https://github.com/apache/kafka/pull/1179
> >> > >
> >> > > Thanks
> >> > >
> >> > > --
> >> > > Florian HUSSONNOIS
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Ewen
> >>
> >
> >
> >
> > --
> > -- Guozhang
>