You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Stephen Powis <sp...@salesforce.com> on 2016/01/08 08:39:16 UTC
Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Hey!
So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this
morning, and it seems like everything went smoothly.
As the last step when I changed the inter.broker.protocol.version to
0.9.0.0 and did a rolling restart, I noticed that the FetchFollower request
times increased fairly significantly. Does anyone know if this is expected
with the new protocol version?
Link to graphite graph: http://i.imgur.com/uhvjCBH.png
Perhaps I'm just being paranoid and it will settle out? This is the same
graph over a larger time period - http://i.imgur.com/FB5Ey2a.png
Thanks!
Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Posted by Stephen Powis <sp...@salesforce.com>.
Hmm over the weekend looks like this slowly crept back up as shown in this
graph: http://i.imgur.com/DH9Qa09.png I performed the upgrade on 1/8
around 12am as shown by the spike in the graph. Here's a graph with a
little more history going back ~1 month. This was pretty stable prior to
the upgrade. http://i.imgur.com/95Gw0MS.png
It looks like its entirely "RemoteTime" - http://i.imgur.com/Uamy42q.png
Any ideas of what I should check? I have num.replica.fetchers set to 8
currently.
Thanks!
On Sat, Jan 9, 2016 at 12:19 AM, Ismael Juma <is...@gmail.com> wrote:
> Great!
> On 8 Jan 2016 13:53, "Stephen Powis" <sp...@salesforce.com> wrote:
>
> > Hey Ismael,
> >
> > It did, just took a few hours :) It seems like its just hit where it
> was
> > before the upgrade.
> >
> >
> >
> >
> > Thanks!
> > Stephen
> >
> > On Fri, Jan 8, 2016 at 8:03 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Stephen,
> > >
> > > Have things settled down since?
> > >
> > > Thanks,
> > > Ismael
> > >
> > > On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis <sp...@salesforce.com>
> > > wrote:
> > >
> > > > Hey!
> > > >
> > > > So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this
> > > > morning, and it seems like everything went smoothly.
> > > >
> > > > As the last step when I changed the inter.broker.protocol.version to
> > > > 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower
> > > request
> > > > times increased fairly significantly. Does anyone know if this is
> > > expected
> > > > with the new protocol version?
> > > >
> > > > Link to graphite graph: http://i.imgur.com/uhvjCBH.png
> > > >
> > > > Perhaps I'm just being paranoid and it will settle out? This is the
> > same
> > > > graph over a larger time period - http://i.imgur.com/FB5Ey2a.png
> > > >
> > > > Thanks!
> > > >
> > >
> >
>
Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Posted by Ismael Juma <is...@gmail.com>.
Great!
On 8 Jan 2016 13:53, "Stephen Powis" <sp...@salesforce.com> wrote:
> Hey Ismael,
>
> It did, just took a few hours :) It seems like its just hit where it was
> before the upgrade.
>
>
>
>
> Thanks!
> Stephen
>
> On Fri, Jan 8, 2016 at 8:03 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Stephen,
> >
> > Have things settled down since?
> >
> > Thanks,
> > Ismael
> >
> > On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis <sp...@salesforce.com>
> > wrote:
> >
> > > Hey!
> > >
> > > So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this
> > > morning, and it seems like everything went smoothly.
> > >
> > > As the last step when I changed the inter.broker.protocol.version to
> > > 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower
> > request
> > > times increased fairly significantly. Does anyone know if this is
> > expected
> > > with the new protocol version?
> > >
> > > Link to graphite graph: http://i.imgur.com/uhvjCBH.png
> > >
> > > Perhaps I'm just being paranoid and it will settle out? This is the
> same
> > > graph over a larger time period - http://i.imgur.com/FB5Ey2a.png
> > >
> > > Thanks!
> > >
> >
>
Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Posted by Stephen Powis <sp...@salesforce.com>.
Hey Ismael,
It did, just took a few hours :) It seems like its just hit where it was
before the upgrade.
Thanks!
Stephen
On Fri, Jan 8, 2016 at 8:03 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Stephen,
>
> Have things settled down since?
>
> Thanks,
> Ismael
>
> On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis <sp...@salesforce.com>
> wrote:
>
> > Hey!
> >
> > So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this
> > morning, and it seems like everything went smoothly.
> >
> > As the last step when I changed the inter.broker.protocol.version to
> > 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower
> request
> > times increased fairly significantly. Does anyone know if this is
> expected
> > with the new protocol version?
> >
> > Link to graphite graph: http://i.imgur.com/uhvjCBH.png
> >
> > Perhaps I'm just being paranoid and it will settle out? This is the same
> > graph over a larger time period - http://i.imgur.com/FB5Ey2a.png
> >
> > Thanks!
> >
>
Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Stephen,
Have things settled down since?
Thanks,
Ismael
On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis <sp...@salesforce.com> wrote:
> Hey!
>
> So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this
> morning, and it seems like everything went smoothly.
>
> As the last step when I changed the inter.broker.protocol.version to
> 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower request
> times increased fairly significantly. Does anyone know if this is expected
> with the new protocol version?
>
> Link to graphite graph: http://i.imgur.com/uhvjCBH.png
>
> Perhaps I'm just being paranoid and it will settle out? This is the same
> graph over a larger time period - http://i.imgur.com/FB5Ey2a.png
>
> Thanks!
>