You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tianji Li <sk...@gmail.com> on 2017/03/15 13:06:45 UTC
restart Kafka Streams application takes around 5 minutes
Hi there,
In the experiments I am doing now, if I restart the streams application, I
have to wait for around 5 minutes for some reason.
I can see something in the Kafka logs:
[2017-03-15 08:36:18,118] INFO [GroupCoordinator 0]: Preparing to
restabilize group xxx-test25 with old generation 2
(kafka.coordinator.GroupCoordinator)
[2017-03-15 08:41:08,449] INFO [GroupCoordinator 0]: Stabilized group
xxx-test25 generation 3 (kafka.coordinator.GroupCoordinator)
[2017-03-15 08:41:08,473] INFO [GroupCoordinator 0]: Assignment received
from leader for group xxx-test25 for generation 3
(kafka.coordinator.GroupCoordinator)
What is happening in these 5 minutes? How to reduce it?
Thanks
Tianji
Re: restart Kafka Streams application takes around 5 minutes
Posted by Tianji Li <sk...@gmail.com>.
Hi Guozhang,
I am using 0.10.2.0. Will try the trunk and report back.
Thanks
Tianji
On Wed, Mar 15, 2017 at 5:25 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Are you building with any released versions of Kafka or with a build from
> Kafka trunk? There are a few fixes we have made post 0.10.2 in trunk that
> has largely reduced the rebalance latency so I'd recommend try using a
> build from Kafka trunk for testing if possible.
>
>
> Guozhang
>
> On Wed, Mar 15, 2017 at 10:46 AM, Tianji Li <sk...@gmail.com> wrote:
>
> > It seems independent to the rocksdb sizes. It also took 5 minutes when
> > there are 375Mbytes this morning...
> >
> > On Wed, Mar 15, 2017 at 9:13 AM, Sachin Mittal <sj...@gmail.com>
> wrote:
> >
> > > rocksdb state store initialization may be taking up that time.
> > > Whats the size of your rockksdb state directory. May be partitioning
> the
> > > source topic, increasing the number of threads/instances processing the
> > > source and reducing the time window of aggregation can help in reducing
> > the
> > > startup time.
> > >
> > >
> > >
> > > On Wed, Mar 15, 2017 at 6:36 PM, Tianji Li <sk...@gmail.com> wrote:
> > >
> > > > Hi there,
> > > >
> > > > In the experiments I am doing now, if I restart the streams
> > application,
> > > I
> > > > have to wait for around 5 minutes for some reason.
> > > >
> > > > I can see something in the Kafka logs:
> > > >
> > > > [2017-03-15 08:36:18,118] INFO [GroupCoordinator 0]: Preparing to
> > > > restabilize group xxx-test25 with old generation 2
> > > > (kafka.coordinator.GroupCoordinator)
> > > > [2017-03-15 08:41:08,449] INFO [GroupCoordinator 0]: Stabilized group
> > > > xxx-test25 generation 3 (kafka.coordinator.GroupCoordinator)
> > > > [2017-03-15 08:41:08,473] INFO [GroupCoordinator 0]: Assignment
> > received
> > > > from leader for group xxx-test25 for generation 3
> > > > (kafka.coordinator.GroupCoordinator)
> > > >
> > > > What is happening in these 5 minutes? How to reduce it?
> > > >
> > > > Thanks
> > > > Tianji
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: restart Kafka Streams application takes around 5 minutes
Posted by Guozhang Wang <wa...@gmail.com>.
Are you building with any released versions of Kafka or with a build from
Kafka trunk? There are a few fixes we have made post 0.10.2 in trunk that
has largely reduced the rebalance latency so I'd recommend try using a
build from Kafka trunk for testing if possible.
Guozhang
On Wed, Mar 15, 2017 at 10:46 AM, Tianji Li <sk...@gmail.com> wrote:
> It seems independent to the rocksdb sizes. It also took 5 minutes when
> there are 375Mbytes this morning...
>
> On Wed, Mar 15, 2017 at 9:13 AM, Sachin Mittal <sj...@gmail.com> wrote:
>
> > rocksdb state store initialization may be taking up that time.
> > Whats the size of your rockksdb state directory. May be partitioning the
> > source topic, increasing the number of threads/instances processing the
> > source and reducing the time window of aggregation can help in reducing
> the
> > startup time.
> >
> >
> >
> > On Wed, Mar 15, 2017 at 6:36 PM, Tianji Li <sk...@gmail.com> wrote:
> >
> > > Hi there,
> > >
> > > In the experiments I am doing now, if I restart the streams
> application,
> > I
> > > have to wait for around 5 minutes for some reason.
> > >
> > > I can see something in the Kafka logs:
> > >
> > > [2017-03-15 08:36:18,118] INFO [GroupCoordinator 0]: Preparing to
> > > restabilize group xxx-test25 with old generation 2
> > > (kafka.coordinator.GroupCoordinator)
> > > [2017-03-15 08:41:08,449] INFO [GroupCoordinator 0]: Stabilized group
> > > xxx-test25 generation 3 (kafka.coordinator.GroupCoordinator)
> > > [2017-03-15 08:41:08,473] INFO [GroupCoordinator 0]: Assignment
> received
> > > from leader for group xxx-test25 for generation 3
> > > (kafka.coordinator.GroupCoordinator)
> > >
> > > What is happening in these 5 minutes? How to reduce it?
> > >
> > > Thanks
> > > Tianji
> > >
> >
>
--
-- Guozhang
Re: restart Kafka Streams application takes around 5 minutes
Posted by Tianji Li <sk...@gmail.com>.
It seems independent to the rocksdb sizes. It also took 5 minutes when
there are 375Mbytes this morning...
On Wed, Mar 15, 2017 at 9:13 AM, Sachin Mittal <sj...@gmail.com> wrote:
> rocksdb state store initialization may be taking up that time.
> Whats the size of your rockksdb state directory. May be partitioning the
> source topic, increasing the number of threads/instances processing the
> source and reducing the time window of aggregation can help in reducing the
> startup time.
>
>
>
> On Wed, Mar 15, 2017 at 6:36 PM, Tianji Li <sk...@gmail.com> wrote:
>
> > Hi there,
> >
> > In the experiments I am doing now, if I restart the streams application,
> I
> > have to wait for around 5 minutes for some reason.
> >
> > I can see something in the Kafka logs:
> >
> > [2017-03-15 08:36:18,118] INFO [GroupCoordinator 0]: Preparing to
> > restabilize group xxx-test25 with old generation 2
> > (kafka.coordinator.GroupCoordinator)
> > [2017-03-15 08:41:08,449] INFO [GroupCoordinator 0]: Stabilized group
> > xxx-test25 generation 3 (kafka.coordinator.GroupCoordinator)
> > [2017-03-15 08:41:08,473] INFO [GroupCoordinator 0]: Assignment received
> > from leader for group xxx-test25 for generation 3
> > (kafka.coordinator.GroupCoordinator)
> >
> > What is happening in these 5 minutes? How to reduce it?
> >
> > Thanks
> > Tianji
> >
>
Re: restart Kafka Streams application takes around 5 minutes
Posted by Sachin Mittal <sj...@gmail.com>.
rocksdb state store initialization may be taking up that time.
Whats the size of your rockksdb state directory. May be partitioning the
source topic, increasing the number of threads/instances processing the
source and reducing the time window of aggregation can help in reducing the
startup time.
On Wed, Mar 15, 2017 at 6:36 PM, Tianji Li <sk...@gmail.com> wrote:
> Hi there,
>
> In the experiments I am doing now, if I restart the streams application, I
> have to wait for around 5 minutes for some reason.
>
> I can see something in the Kafka logs:
>
> [2017-03-15 08:36:18,118] INFO [GroupCoordinator 0]: Preparing to
> restabilize group xxx-test25 with old generation 2
> (kafka.coordinator.GroupCoordinator)
> [2017-03-15 08:41:08,449] INFO [GroupCoordinator 0]: Stabilized group
> xxx-test25 generation 3 (kafka.coordinator.GroupCoordinator)
> [2017-03-15 08:41:08,473] INFO [GroupCoordinator 0]: Assignment received
> from leader for group xxx-test25 for generation 3
> (kafka.coordinator.GroupCoordinator)
>
> What is happening in these 5 minutes? How to reduce it?
>
> Thanks
> Tianji
>