You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sameer Kumar <sa...@gmail.com> on 2017/06/23 16:31:01 UTC
Kafka 10.2.1 KStreams rebalancing issue
Hi,
Came across a rebalancing issue in using KafkaStreams. I have two machines,
Machine1 and Machine2, machine1 is consuming all partitions and machine2 is
completely free and not processing any partitions. If I shutdown machine1,
then machine2 will take over and would start consuming all partitions.
But, in this scenario, its not using the complete cluster.
*Machine 1*
2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
[StreamThread-27] Creating active task 1_2 with assigned partitions
[LIC3-43-lic3-deb-ci2-43-repartition-2,
LIC3-43-lic3-cnt-ci-43-repartition-2]
2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
[StreamThread-20] State transition from PARTITIONS_REVOKED to
ASSIGNING_PARTITIONS.
2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
[StreamThread-31] State transition from PARTITIONS_REVOKED to
ASSIGNING_PARTITIONS.
2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
[StreamThread-13] State transition from PARTITIONS_REVOKED to
ASSIGNING_PARTITIONS.
2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
[StreamThread-16] State transition from ASSIGNING_PARTITIONS to RUNNING.
2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
[StreamThread-18] Creating active task 0_4 with assigned partitions
[testS5-4]
2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
[StreamThread-13] Creating active task 0_2 with assigned partitions
[testS5-2]
2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
[StreamThread-31] Creating active task 2_0 with assigned partitions
[LIC3-43-lic3-cnt-li-43-repartition-0,
LIC3-43-lic3-cnt-li-43_1-repartition-0]
2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
[StreamThread-20] Creating active task 0_9 with assigned partitions
[testS5-9]
2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
[LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
[StreamThread-29] Creating active task 2_5 with assigned partitions
[LIC3-43-lic3-cnt-li-43-repartition-5,
LIC3-43-lic3-cnt-li-43_1-repartition-5]
*Machine2*
21:54:08 INFO StreamThread:248 - stream-thread [StreamThread-8] at state
RUNNING: partitions [] revoked at the beginning of consumer rebalance.
2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-5] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
[StreamThread-5] Removing all active tasks []
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
[StreamThread-5] Removing all standby tasks []
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-18] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO StreamThread:163 - stream-thread [StreamThread-8]
State transition from RUNNING to PARTITIONS_REVOKED.
2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
[StreamThread-18] Removing all active tasks []
2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
[StreamThread-18] Removing all standby tasks []
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-1] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-15] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
[StreamThread-15] Removing all active tasks []
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-30] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-14] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
[StreamThread-15] Removing all standby tasks []
2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
[StreamThread-1] Removing all active tasks []
2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
[StreamThread-1] Removing all standby tasks []
2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
[StreamThread-14] Removing all active tasks []
2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
[StreamThread-30] Removing all active tasks []
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
[StreamThread-30] Removing all standby tasks []
2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
[StreamThread-14] Removing all standby tasks []
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-20] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
[StreamThread-20] Removing all active tasks []
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-6] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-9] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
[StreamThread-9] Removing all active tasks []
2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
[StreamThread-9] Removing all standby tasks []
2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
LIC3-43
2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
[LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
REBALANCING to REBALANCING.
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-4] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
[StreamThread-31] Updating suspended tasks to contain active tasks []
2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
[StreamThread-31] Removing all active tasks []
2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
[StreamThread-31] Removing all standby tasks []
Regards,
-Sameer.
Re: Kafka 10.2.1 KStreams rebalancing issue
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Sameer,
From your shared logs it seems the number of "threads" is actually larger
than the number of topic partitions (note each machine seem to have lots of
threads). More specifically, I have only saw max. of 6 partitions for any
topics, e.g.:
LIC3-43-lic3-deb-ci2-43-repartition-0
LIC3-43-lic3-deb-ci2-43-repartition-1
LIC3-43-lic3-deb-ci2-43-repartition-2
...
LIC3-43-lic3-deb-ci2-43-repartition-5
If that is the case, and assume you only have one sub-topology, then only 6
threads out of the TOTAL number of threads across your machines will ever
get any data: this is because within a sub-topology, the number of tasks
(i.e. the unit of parallelism that can be distributed across threads) will
but the MAX(#.partitions of input topics).
Guozhang
On Fri, Jun 23, 2017 at 7:32 PM, Sameer Kumar <sa...@gmail.com>
wrote:
> Hi Eno,
>
> Yes, I was concerned that machine2 ispite of being free is not utilized at
> all.
>
> Partitions would be 60 and streams would have 63 threads. So, I understand
> now why threads were not started on machine2.
>
> For others, 1 partition would be consumed by 1 thread. So, thus the
> mismatch.
>
> Thanks,
> -Sameer.
>
> On Fri, Jun 23, 2017 at 11:02 PM, Eno Thereska <en...@gmail.com>
> wrote:
>
> > Hi Sameer,
> >
> > Could you elaborate on your question? Are you concerned that machine2
> does
> > not have any tasks in the beginning?
> >
> > Could you share your streams configuration? In particular how many
> threads
> > does each stream instance have? Also how many topics and partitions do
> you
> > have?
> >
> >
> > Thanks,
> > Eno
> >
> > > On 23 Jun 2017, at 17:31, Sameer Kumar <sa...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > Came across a rebalancing issue in using KafkaStreams. I have two
> > machines,
> > > Machine1 and Machine2, machine1 is consuming all partitions and
> machine2
> > is
> > > completely free and not processing any partitions. If I shutdown
> > machine1,
> > > then machine2 will take over and would start consuming all partitions.
> > >
> > > But, in this scenario, its not using the complete cluster.
> > >
> > > *Machine 1*
> > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > > [StreamThread-27] Creating active task 1_2 with assigned partitions
> > > [LIC3-43-lic3-deb-ci2-43-repartition-2,
> > > LIC3-43-lic3-cnt-ci-43-repartition-2]
> > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> > > [StreamThread-20] State transition from PARTITIONS_REVOKED to
> > > ASSIGNING_PARTITIONS.
> > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> > > [StreamThread-31] State transition from PARTITIONS_REVOKED to
> > > ASSIGNING_PARTITIONS.
> > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> > > [StreamThread-13] State transition from PARTITIONS_REVOKED to
> > > ASSIGNING_PARTITIONS.
> > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> > > [StreamThread-16] State transition from ASSIGNING_PARTITIONS to
> RUNNING.
> > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > > [StreamThread-18] Creating active task 0_4 with assigned partitions
> > > [testS5-4]
> > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > > [StreamThread-13] Creating active task 0_2 with assigned partitions
> > > [testS5-2]
> > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > > [StreamThread-31] Creating active task 2_0 with assigned partitions
> > > [LIC3-43-lic3-cnt-li-43-repartition-0,
> > > LIC3-43-lic3-cnt-li-43_1-repartition-0]
> > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > > [StreamThread-20] Creating active task 0_9 with assigned partitions
> > > [testS5-9]
> > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > > [StreamThread-29] Creating active task 2_5 with assigned partitions
> > > [LIC3-43-lic3-cnt-li-43-repartition-5,
> > > LIC3-43-lic3-cnt-li-43_1-repartition-5]
> > >
> > >
> > > *Machine2*
> > >
> > > 21:54:08 INFO StreamThread:248 - stream-thread [StreamThread-8] at
> state
> > > RUNNING: partitions [] revoked at the beginning of consumer rebalance.
> > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > > LIC3-43
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-5] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > > [StreamThread-5] Removing all active tasks []
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > > [StreamThread-5] Removing all standby tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-18] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:163 - stream-thread
> > [StreamThread-8]
> > > State transition from RUNNING to PARTITIONS_REVOKED.
> > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > > [StreamThread-18] Removing all active tasks []
> > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > > LIC3-43
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > > [StreamThread-18] Removing all standby tasks []
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-1] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-15] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > > [StreamThread-15] Removing all active tasks []
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > > LIC3-43
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-30] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-14] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > > [StreamThread-15] Removing all standby tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > > [StreamThread-1] Removing all active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > > [StreamThread-1] Removing all standby tasks []
> > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > > LIC3-43
> > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > > LIC3-43
> > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > > [StreamThread-14] Removing all active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > > [StreamThread-30] Removing all active tasks []
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > > [StreamThread-30] Removing all standby tasks []
> > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > > LIC3-43
> > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > > [StreamThread-14] Removing all standby tasks []
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-20] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > > [StreamThread-20] Removing all active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-6] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-9] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > > LIC3-43
> > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > > [StreamThread-9] Removing all active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > > [StreamThread-9] Removing all standby tasks []
> > > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > > LIC3-43
> > > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > > REBALANCING to REBALANCING.
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-4] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > > [StreamThread-31] Updating suspended tasks to contain active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > > [StreamThread-31] Removing all active tasks []
> > > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > > [StreamThread-31] Removing all standby tasks []
> > >
> > >
> > > Regards,
> > > -Sameer.
> >
> >
>
--
-- Guozhang
Re: Kafka 10.2.1 KStreams rebalancing issue
Posted by Sameer Kumar <sa...@gmail.com>.
Hi Eno,
Yes, I was concerned that machine2 ispite of being free is not utilized at
all.
Partitions would be 60 and streams would have 63 threads. So, I understand
now why threads were not started on machine2.
For others, 1 partition would be consumed by 1 thread. So, thus the
mismatch.
Thanks,
-Sameer.
On Fri, Jun 23, 2017 at 11:02 PM, Eno Thereska <en...@gmail.com>
wrote:
> Hi Sameer,
>
> Could you elaborate on your question? Are you concerned that machine2 does
> not have any tasks in the beginning?
>
> Could you share your streams configuration? In particular how many threads
> does each stream instance have? Also how many topics and partitions do you
> have?
>
>
> Thanks,
> Eno
>
> > On 23 Jun 2017, at 17:31, Sameer Kumar <sa...@gmail.com> wrote:
> >
> > Hi,
> >
> > Came across a rebalancing issue in using KafkaStreams. I have two
> machines,
> > Machine1 and Machine2, machine1 is consuming all partitions and machine2
> is
> > completely free and not processing any partitions. If I shutdown
> machine1,
> > then machine2 will take over and would start consuming all partitions.
> >
> > But, in this scenario, its not using the complete cluster.
> >
> > *Machine 1*
> > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > [StreamThread-27] Creating active task 1_2 with assigned partitions
> > [LIC3-43-lic3-deb-ci2-43-repartition-2,
> > LIC3-43-lic3-cnt-ci-43-repartition-2]
> > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> > [StreamThread-20] State transition from PARTITIONS_REVOKED to
> > ASSIGNING_PARTITIONS.
> > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> > [StreamThread-31] State transition from PARTITIONS_REVOKED to
> > ASSIGNING_PARTITIONS.
> > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> > [StreamThread-13] State transition from PARTITIONS_REVOKED to
> > ASSIGNING_PARTITIONS.
> > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> > [StreamThread-16] State transition from ASSIGNING_PARTITIONS to RUNNING.
> > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > [StreamThread-18] Creating active task 0_4 with assigned partitions
> > [testS5-4]
> > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > [StreamThread-13] Creating active task 0_2 with assigned partitions
> > [testS5-2]
> > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > [StreamThread-31] Creating active task 2_0 with assigned partitions
> > [LIC3-43-lic3-cnt-li-43-repartition-0,
> > LIC3-43-lic3-cnt-li-43_1-repartition-0]
> > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > [StreamThread-20] Creating active task 0_9 with assigned partitions
> > [testS5-9]
> > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> > [StreamThread-29] Creating active task 2_5 with assigned partitions
> > [LIC3-43-lic3-cnt-li-43-repartition-5,
> > LIC3-43-lic3-cnt-li-43_1-repartition-5]
> >
> >
> > *Machine2*
> >
> > 21:54:08 INFO StreamThread:248 - stream-thread [StreamThread-8] at state
> > RUNNING: partitions [] revoked at the beginning of consumer rebalance.
> > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-5] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > [StreamThread-5] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > [StreamThread-5] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-18] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:163 - stream-thread
> [StreamThread-8]
> > State transition from RUNNING to PARTITIONS_REVOKED.
> > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > [StreamThread-18] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > [StreamThread-18] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-1] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-15] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > [StreamThread-15] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-30] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-14] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > [StreamThread-15] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > [StreamThread-1] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > [StreamThread-1] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > [StreamThread-14] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > [StreamThread-30] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > [StreamThread-30] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > [StreamThread-14] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-20] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > [StreamThread-20] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-6] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-9] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > [StreamThread-9] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > [StreamThread-9] Removing all standby tasks []
> > 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> > LIC3-43
> > 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> > [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> > REBALANCING to REBALANCING.
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-4] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> > [StreamThread-31] Updating suspended tasks to contain active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> > [StreamThread-31] Removing all active tasks []
> > 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> > [StreamThread-31] Removing all standby tasks []
> >
> >
> > Regards,
> > -Sameer.
>
>
Re: Kafka 10.2.1 KStreams rebalancing issue
Posted by Eno Thereska <en...@gmail.com>.
Hi Sameer,
Could you elaborate on your question? Are you concerned that machine2 does not have any tasks in the beginning?
Could you share your streams configuration? In particular how many threads does each stream instance have? Also how many topics and partitions do you have?
Thanks,
Eno
> On 23 Jun 2017, at 17:31, Sameer Kumar <sa...@gmail.com> wrote:
>
> Hi,
>
> Came across a rebalancing issue in using KafkaStreams. I have two machines,
> Machine1 and Machine2, machine1 is consuming all partitions and machine2 is
> completely free and not processing any partitions. If I shutdown machine1,
> then machine2 will take over and would start consuming all partitions.
>
> But, in this scenario, its not using the complete cluster.
>
> *Machine 1*
> 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> [StreamThread-27] Creating active task 1_2 with assigned partitions
> [LIC3-43-lic3-deb-ci2-43-repartition-2,
> LIC3-43-lic3-cnt-ci-43-repartition-2]
> 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> [StreamThread-20] State transition from PARTITIONS_REVOKED to
> ASSIGNING_PARTITIONS.
> 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> [StreamThread-31] State transition from PARTITIONS_REVOKED to
> ASSIGNING_PARTITIONS.
> 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> [StreamThread-13] State transition from PARTITIONS_REVOKED to
> ASSIGNING_PARTITIONS.
> 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:07 INFO StreamThread:163 - stream-thread
> [StreamThread-16] State transition from ASSIGNING_PARTITIONS to RUNNING.
> 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> [StreamThread-18] Creating active task 0_4 with assigned partitions
> [testS5-4]
> 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> [StreamThread-13] Creating active task 0_2 with assigned partitions
> [testS5-2]
> 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> [StreamThread-31] Creating active task 2_0 with assigned partitions
> [LIC3-43-lic3-cnt-li-43-repartition-0,
> LIC3-43-lic3-cnt-li-43_1-repartition-0]
> 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> [StreamThread-20] Creating active task 0_9 with assigned partitions
> [testS5-9]
> 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:07 INFO KafkaStreams:224 - stream-client
> [LIC3-43-2b05186c-cbf2-4f9c-acc4-0dd929c7d647] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:07 INFO StreamThread:858 - stream-thread
> [StreamThread-29] Creating active task 2_5 with assigned partitions
> [LIC3-43-lic3-cnt-li-43-repartition-5,
> LIC3-43-lic3-cnt-li-43_1-repartition-5]
>
>
> *Machine2*
>
> 21:54:08 INFO StreamThread:248 - stream-thread [StreamThread-8] at state
> RUNNING: partitions [] revoked at the beginning of consumer rebalance.
> 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> LIC3-43
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-5] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> [StreamThread-5] Removing all active tasks []
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> [StreamThread-5] Removing all standby tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-18] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:163 - stream-thread [StreamThread-8]
> State transition from RUNNING to PARTITIONS_REVOKED.
> 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> [StreamThread-18] Removing all active tasks []
> 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> LIC3-43
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> [StreamThread-18] Removing all standby tasks []
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-1] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-15] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> [StreamThread-15] Removing all active tasks []
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> LIC3-43
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-30] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-14] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> [StreamThread-15] Removing all standby tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> [StreamThread-1] Removing all active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> [StreamThread-1] Removing all standby tasks []
> 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> LIC3-43
> 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> LIC3-43
> 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> [StreamThread-14] Removing all active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> [StreamThread-30] Removing all active tasks []
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> [StreamThread-30] Removing all standby tasks []
> 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> LIC3-43
> 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> [StreamThread-14] Removing all standby tasks []
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-20] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> [StreamThread-20] Removing all active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-6] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-9] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> LIC3-43
> 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> [StreamThread-9] Removing all active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> [StreamThread-9] Removing all standby tasks []
> 2017-06-23 21:54:08 INFO AbstractCoordinator:420 - (Re-)joining group
> LIC3-43
> 2017-06-23 21:54:08 INFO KafkaStreams:224 - stream-client
> [LIC3-43-47a995d4-2f53-4837-8b8a-c5550bc7b3eb] State transition from
> REBALANCING to REBALANCING.
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-4] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1042 - stream-thread
> [StreamThread-31] Updating suspended tasks to contain active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1049 - stream-thread
> [StreamThread-31] Removing all active tasks []
> 2017-06-23 21:54:08 INFO StreamThread:1064 - stream-thread
> [StreamThread-31] Removing all standby tasks []
>
>
> Regards,
> -Sameer.