You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yonghui Zhao <zh...@gmail.com> on 2014/05/12 09:48:13 UTC
owner info in zk is not correct
Hi,
We are using kafka 0.7.
2 brokers, each broker has 10 partitions for one topic
3 consumers in one consumer group, each consumer create 10 streams.
Today, when we want to rollout new service.
After we restart one consumer we find exceptions and warning.
kafka.common.ConsumerRebalanceFailedException:
RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't rebalance
after 4 retries
[INFO 2014-05-12 15:17:47.364]
kafka.utils.Logging$class.info(Logging.scala:61)
[conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored data:
RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
[INFO 2014-05-12 15:17:47.366]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the
partition ownership to be deleted: 1-2]
[INFO 2014-05-12 15:17:47.375]
kafka.utils.Logging$class.info(Logging.scala:61)
[conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored data:
RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
[INFO 2014-05-12 15:17:47.375]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the
partition ownership to be deleted: 1-3]
[INFO 2014-05-12 15:17:47.385]
kafka.utils.Logging$class.info(Logging.scala:61)
[conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored data:
RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
[INFO 2014-05-12 15:17:47.386]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the
partition ownership to be deleted: 1-5]
And I opened zk viewer.
In zk, we found 2 consumers in ConsumerGroup/ids:
RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
And in owners/topic/ we found all partitions are assigned to
sd-sns-relation03.bj:
Here is the owner info:
1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
So all partitions are assigned to sd-sns-relation03.bj, but from logs and
counter, we are sure sd-sns-relation02.bj has input too.
My question is:
1. why rebalance failed?
2. why owner info is wrong? btw: zkclient is 0.2
Re: owner info in zk is not correct
Posted by Helin Xiang <xk...@gmail.com>.
Thanks Jun.
That's really helpful.
On Fri, May 23, 2014 at 10:38 PM, Jun Rao <ju...@gmail.com> wrote:
> This is true in 0.7.x. In 0.8.x, the fetcher will be restarted
> automatically on such failures.
>
> Thanks,
>
> Jun
>
>
> On Thu, May 22, 2014 at 10:40 PM, Helin Xiang <xk...@gmail.com> wrote:
>
> > Hi Guozhang,
> >
> > sorry for asking a little unrelated problem here.
> >
> > We found a consumer stopping fetching data when doing an network upgrade,
> > if the consumer have connection problem with one broker (but OK with
> > zookeeper and other brokers)
> > the fetcherrunnable will stop, but there is no chance to restart the
> thread
> > again ( since there is no zookeeper timeout so no reblance will be
> > triggered)
> >
> > kafka version is 0.7.2 and we are using high level consumer.
> >
> > I did a simulation test using iptables and the result is same( no chance
> to
> > restart fetcherrunnable).
> > I have read the code, seems the exception will not be handled outside.
> Am I
> > wrong about this?
> >
> > THANKS.
> >
> >
> >
> >
> >
> > On Sat, May 17, 2014 at 12:19 AM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> >
> > > Hi Yonghui,
> > >
> > > Could you check if consumer2's fetcher thread is still alive? Also we
> > have
> > > an entry in FAQ wiki page about "consumer stopped consuming", current
> > > Apache has some issues with the wiki page, but you may want to check it
> > out
> > > once Apache page resumes.
> > >
> > > Guozhang
> > >
> > >
> > > On Wed, May 14, 2014 at 8:59 PM, Yonghui Zhao <zh...@gmail.com>
> > > wrote:
> > >
> > > > Thanks GuoZhang.
> > > >
> > > > After last accident, we stop all consumers and then restart all
> > consumers
> > > > one by one then it is ok.
> > > >
> > > > 2 brokers, 10 partitions / broker, 3 consumers, each create 10
> > stream
> > > >
> > > > So consumer1 consumes 10 partitions, consumer2 consumers another 10
> > > > partitions, consumer3 is idle.
> > > >
> > > > Today we find some exceptions in consumer2, after these exceptions
> > > > consumer2 doesn't work, no message is consumed.
> > > >
> > > > But in zk I found the owner ship doesn't change, consumer1 own 10
> > > > partitions consumer 2 own another 10 partitions/
> > > >
> > > > How can we avoid this happen again?
> > > >
> > > >
> > > > From the log we see the error sequence:
> > > >
> > > >
> > > >
> > > >
> > > > *exception during commitOffsets, Reconnect in multifetch due to
> socket
> > > > error, rebalance 2 times, error in FetcherRunnable'Seems the last
> > > error
> > > > in FetcherRunnable is deadly, after this error no message is
> consumed,
> > > but
> > > > zk ownership doesn't release.*Here is all the kafka related log in
> that
> > > > time
> > > > *:*
> > > >
> > > > [WARN 2014-05-13 16:19:05.020]
> > > > kafka.utils.Logging$class.warn(Logging.scala:79)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> > > > during commitOffsets]
> > > > at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> > > > at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> > > > [INFO 2014-05-13 16:19:08.991]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c ZK
> expired;
> > > > release old broker parition ownership; re-register consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c]
> > > > [INFO 2014-05-13 16:19:08.991]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > > registering consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
> > > > [WARN 2014-05-13 16:19:09.001]
> > > > kafka.utils.Logging$class.warn(Logging.scala:79)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> > > > during commitOffsets]
> > > > at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> > > > at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> > > > [INFO 2014-05-13 16:19:09.002]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > > registering
> > > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> in
> > > ZK]
> > > > [INFO 2014-05-13 16:19:09.003]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > > rebalancing consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > > > [INFO 2014-05-13 16:19:09.063]
> > > > kafka.utils.Logging$class.info(Logging.scala:69)
> > > > [Reconnect in multifetch due to socket error: ]
> > > > at kafka.utils.Utils$.read(Utils.scala:538)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> > > > at
> > kafka.network.Receive$class.readCompletely(Transmission.scala:55)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > > at
> > > kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
> > > > at
> > > > kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:117)
> > > > at
> > kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
> > > > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > > > [INFO 2014-05-13 16:19:09.068]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > > > [INFO 2014-05-13 16:19:09.069]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > > [INFO 2014-05-13 16:19:09.071]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared all relevant queues for this fetcher]
> > > > [INFO 2014-05-13 16:19:09.071]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.071]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.071]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared the data chunks in all the consumer message iterators]
> > > > [INFO 2014-05-13 16:19:09.073]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> Committing
> > > all
> > > > offsets after clearing the fetcher queues]
> > > > [INFO 2014-05-13 16:19:09.073]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > > > partition ownership]
> > > > [INFO 2014-05-13 16:19:09.073]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> rebalancing
> > > the
> > > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7,
> 1-8,
> > > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic
> sensei
> > > > with consumers:
> > > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > > > [INFO 2014-05-13 16:19:09.073]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> attempting
> > > to
> > > > claim partition 1-8]
> > > > [INFO 2014-05-13 16:19:09.077]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> attempting
> > > to
> > > > claim partition 1-4]
> > > > [INFO 2014-05-13 16:19:09.081]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> attempting
> > > to
> > > > claim partition 1-3]
> > > > [INFO 2014-05-13 16:19:09.085]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> attempting
> > > to
> > > > claim partition 1-2]
> > > > [INFO 2014-05-13 16:19:09.089]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> attempting
> > > to
> > > > claim partition 1-1]
> > > > [INFO 2014-05-13 16:19:09.093]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> attempting
> > > to
> > > > claim partition 1-5]
> > > > [INFO 2014-05-13 16:19:09.097]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> attempting
> > > to
> > > > claim partition 1-6]
> > > > [INFO 2014-05-13 16:19:09.101]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> attempting
> > > to
> > > > claim partition 1-9]
> > > > [INFO 2014-05-13 16:19:09.105]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> attempting
> > > to
> > > > claim partition 1-7]
> > > > [INFO 2014-05-13 16:19:09.109]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> attempting
> > > to
> > > > claim partition 1-0]
> > > > [INFO 2014-05-13 16:19:09.118]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> > successfully
> > > > owned partition 1-2 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.122]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> > successfully
> > > > owned partition 1-3 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.127]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> > successfully
> > > > owned partition 1-5 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.132]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> > successfully
> > > > owned partition 1-0 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.137]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> > successfully
> > > > owned partition 1-9 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.153]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> > successfully
> > > > owned partition 1-6 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.157]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> > successfully
> > > > owned partition 1-7 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.161]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> > successfully
> > > > owned partition 1-1 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.166]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> > successfully
> > > > owned partition 1-4 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.170]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> > successfully
> > > > owned partition 1-8 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.170]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating
> > the
> > > > cache]
> > > > [INFO 2014-05-13 16:19:09.171]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > > > partitions : sensei:1-0: fetched offset = 1555136587: consumed
> offset =
> > > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > > > 1558487487]
> > > > [INFO 2014-05-13 16:19:09.171]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > > rebalancing
> > > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> try
> > > #0]
> > > > [INFO 2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset:
> > 1563630746
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset:
> > 1552261483
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset:
> 10316617
> > > from
> > > > 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset:
> > 1550214323
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset:
> > 1551158177
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset:
> > 1552939302
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset:
> > 1550960225
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset:
> > 1556683052
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset:
> > 1555136587
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset:
> > 1558487487
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.188]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > > rebalancing consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > > > [INFO 2014-05-13 16:19:09.240]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > > > [INFO 2014-05-13 16:19:09.240]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > > [INFO 2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared all relevant queues for this fetcher]
> > > > [INFO 2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared the data chunks in all the consumer message iterators]
> > > > [INFO 2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> Committing
> > > all
> > > > offsets after clearing the fetcher queues]
> > > > [INFO 2014-05-13 16:19:09.285]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > > > partition ownership]
> > > > [INFO 2014-05-13 16:19:09.333]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> rebalancing
> > > the
> > > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7,
> 1-8,
> > > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic
> sensei
> > > > with consumers:
> > > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > > > [INFO 2014-05-13 16:19:09.333]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> attempting
> > > to
> > > > claim partition 1-8]
> > > > [INFO 2014-05-13 16:19:09.337]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> attempting
> > > to
> > > > claim partition 1-4]
> > > > [INFO 2014-05-13 16:19:09.341]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> attempting
> > > to
> > > > claim partition 1-3]
> > > > [INFO 2014-05-13 16:19:09.345]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> attempting
> > > to
> > > > claim partition 1-2]
> > > > [INFO 2014-05-13 16:19:09.349]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> attempting
> > > to
> > > > claim partition 1-1]
> > > > [INFO 2014-05-13 16:19:09.354]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> attempting
> > > to
> > > > claim partition 1-5]
> > > > [INFO 2014-05-13 16:19:09.358]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> attempting
> > > to
> > > > claim partition 1-6]
> > > > [INFO 2014-05-13 16:19:09.362]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> attempting
> > > to
> > > > claim partition 1-9]
> > > > [INFO 2014-05-13 16:19:09.366]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> attempting
> > > to
> > > > claim partition 1-7]
> > > > [INFO 2014-05-13 16:19:09.370]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> attempting
> > > to
> > > > claim partition 1-0]
> > > > [INFO 2014-05-13 16:19:09.379]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> > successfully
> > > > owned partition 1-2 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.390]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [/consumers/RecommendEvent/owners/sensei/1-3 exists with value
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 during
> > > > connection loss; this is ok]
> > > > [INFO 2014-05-13 16:19:09.390]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> > successfully
> > > > owned partition 1-3 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.395]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> > successfully
> > > > owned partition 1-5 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.400]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> > successfully
> > > > owned partition 1-0 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.405]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> > successfully
> > > > owned partition 1-9 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.409]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> > successfully
> > > > owned partition 1-6 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.417]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [/consumers/RecommendEvent/owners/sensei/1-7 exists with value
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 during
> > > > connection loss; this is ok]
> > > > [INFO 2014-05-13 16:19:09.417]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> > successfully
> > > > owned partition 1-7 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.422]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> > successfully
> > > > owned partition 1-1 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.430]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [/consumers/RecommendEvent/owners/sensei/1-4 exists with value
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 during
> > > > connection loss; this is ok]
> > > > [INFO 2014-05-13 16:19:09.430]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> > successfully
> > > > owned partition 1-4 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.435]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> > successfully
> > > > owned partition 1-8 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.435]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating
> > the
> > > > cache]
> > > > [INFO 2014-05-13 16:19:09.435]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > > > partitions : sensei:1-0: fetched offset = 1555136587: consumed
> offset =
> > > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > > > 1558487487]
> > > > [INFO 2014-05-13 16:19:09.436]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > > rebalancing
> > > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> try
> > > #0]
> > > > [INFO 2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset:
> > 1563630746
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > > rebalancing consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > > > [INFO 2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset:
> > 1552261483
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset:
> 10316617
> > > from
> > > > 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset:
> > 1550214323
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset:
> > 1551158177
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset:
> > 1552939302
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset:
> > 1550960225
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.439]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset:
> > 1556683052
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.439]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset:
> > 1555136587
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.439]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset:
> > 1558487487
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > > > [INFO 2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > > [INFO 2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared all relevant queues for this fetcher]
> > > > [INFO 2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.494]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.494]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO 2014-05-13 16:19:09.494]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared the data chunks in all the consumer message iterators]
> > > > [INFO 2014-05-13 16:19:09.494]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> Committing
> > > all
> > > > offsets after clearing the fetcher queues]
> > > > [INFO 2014-05-13 16:19:09.539]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > > > partition ownership]
> > > > [INFO 2014-05-13 16:19:09.582]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> rebalancing
> > > the
> > > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7,
> 1-8,
> > > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic
> sensei
> > > > with consumers:
> > > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > > > [INFO 2014-05-13 16:19:09.582]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> attempting
> > > to
> > > > claim partition 1-8]
> > > > [INFO 2014-05-13 16:19:09.586]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> attempting
> > > to
> > > > claim partition 1-4]
> > > > [INFO 2014-05-13 16:19:09.590]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> attempting
> > > to
> > > > claim partition 1-3]
> > > > [INFO 2014-05-13 16:19:09.594]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> attempting
> > > to
> > > > claim partition 1-2]
> > > > [INFO 2014-05-13 16:19:09.598]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> attempting
> > > to
> > > > claim partition 1-1]
> > > > [INFO 2014-05-13 16:19:09.602]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> attempting
> > > to
> > > > claim partition 1-5]
> > > > [INFO 2014-05-13 16:19:09.606]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> attempting
> > > to
> > > > claim partition 1-6]
> > > > [INFO 2014-05-13 16:19:09.610]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> attempting
> > > to
> > > > claim partition 1-9]
> > > > [INFO 2014-05-13 16:19:09.614]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> attempting
> > > to
> > > > claim partition 1-7]
> > > > [INFO 2014-05-13 16:19:09.618]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> attempting
> > > to
> > > > claim partition 1-0]
> > > > [INFO 2014-05-13 16:19:09.633]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> > successfully
> > > > owned partition 1-2 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.638]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> > successfully
> > > > owned partition 1-3 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.646]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> > successfully
> > > > owned partition 1-5 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.656]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> > successfully
> > > > owned partition 1-0 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.660]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> > successfully
> > > > owned partition 1-9 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.665]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> > successfully
> > > > owned partition 1-6 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.669]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> > successfully
> > > > owned partition 1-7 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.674]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> > successfully
> > > > owned partition 1-1 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.679]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> > successfully
> > > > owned partition 1-4 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.684]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> > successfully
> > > > owned partition 1-8 for topic sensei]
> > > > [INFO 2014-05-13 16:19:09.684]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating
> > the
> > > > cache]
> > > > [INFO 2014-05-13 16:19:09.684]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > > > partitions : sensei:1-0: fetched offset = 1555136587: consumed
> offset =
> > > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > > > 1558487487]
> > > > [INFO 2014-05-13 16:19:09.684]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > > rebalancing
> > > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> try
> > > #0]
> > > > [INFO 2014-05-13 16:19:09.685]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset:
> > 1563630746
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.685]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset:
> > 1552261483
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.685]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset:
> 10316617
> > > from
> > > > 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.685]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset:
> > 1550214323
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset:
> > 1551158177
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset:
> > 1552939302
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset:
> > 1550960225
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset:
> > 1556683052
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset:
> > 1555136587
> > > > from 10.101.10.12:9092]
> > > > [INFO 2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset:
> > 1558487487
> > > > from 10.101.10.12:9092]
> > > > [ERROR 2014-05-13 16:19:18.687]
> > > > kafka.utils.Logging$class.error(Logging.scala:89) [error in
> > > FetcherRunnable
> > > > ]
> > > > at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:49)
> > > > at
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:186)
> > > > at
> > kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:113)
> > > > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > > > [INFO 2014-05-13 16:19:18.687]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > >
> > > >
> > > >
> > > > 2014-05-12 23:22 GMT+08:00 Guozhang Wang <wa...@gmail.com>:
> > > >
> > > > > Hello Yonghui,
> > > > >
> > > > > In 0.7 the consumer rebalance logic is distributed and in some
> corner
> > > > cases
> > > > > such as soft-failure-caused-consecutive rebalances some consumer
> may
> > > > > consider the rebalance as complete while others are still trying
> the
> > > > > rebalance process. You can check the GC logs on your consumer to
> > verify
> > > > if
> > > > > that is the case:
> > > > >
> > > > > https://issues.apache.org/jira/browse/KAFKA-242
> > > > >
> > > > > If you bounce the consumers to trigger another rebalance, this
> issue
> > > > would
> > > > > likely to be resolved.
> > > > >
> > > > > To solve this issue in 0.9 we are moving the group management like
> > load
> > > > > rebalance from the ZK-based distributed logic into a centralized
> > > > > coordiantor. Details can be found here:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <
> > zhaoyonghui@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are using kafka 0.7.
> > > > > >
> > > > > > 2 brokers, each broker has 10 partitions for one topic
> > > > > > 3 consumers in one consumer group, each consumer create 10
> streams.
> > > > > >
> > > > > >
> > > > > > Today, when we want to rollout new service.
> > > > > > After we restart one consumer we find exceptions and warning.
> > > > > >
> > > > > > kafka.common.ConsumerRebalanceFailedException:
> > > > > > RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't
> > > > > rebalance
> > > > > > after 4 retries
> > > > > >
> > > > > >
> > > > > > [INFO 2014-05-12 15:17:47.364]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> > > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2
> stored
> > > > data:
> > > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > > > > [INFO 2014-05-12 15:17:47.366]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e
> waiting
> > > for
> > > > > the
> > > > > > partition ownership to be deleted: 1-2]
> > > > > > [INFO 2014-05-12 15:17:47.375]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> > > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3
> stored
> > > > data:
> > > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > > > > [INFO 2014-05-12 15:17:47.375]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e
> waiting
> > > for
> > > > > the
> > > > > > partition ownership to be deleted: 1-3]
> > > > > > [INFO 2014-05-12 15:17:47.385]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> > > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5
> stored
> > > > data:
> > > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> > > > > > [INFO 2014-05-12 15:17:47.386]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e
> waiting
> > > for
> > > > > the
> > > > > > partition ownership to be deleted: 1-5]
> > > > > >
> > > > > >
> > > > > >
> > > > > > And I opened zk viewer.
> > > > > >
> > > > > > In zk, we found 2 consumers in ConsumerGroup/ids:
> > > > > >
> > > > > > RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> > > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
> > > > > >
> > > > > >
> > > > > > And in owners/topic/ we found all partitions are assigned to
> > > > > > sd-sns-relation03.bj:
> > > > > >
> > > > > > Here is the owner info:
> > > > > > 1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > > > 1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > > > 1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > > > 1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > > > 1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > > > 1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > > > 1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > > > 1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > > > 1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > > > 1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > > >
> > > > > > 2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > > > 2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > > > 2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > > > 2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > > > 2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > > > 2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> > > > > > 2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> > > > > > 2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> > > > > > 2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> > > > > > 2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
> > > > > >
> > > > > >
> > > > > > So all partitions are assigned to sd-sns-relation03.bj, but
> from
> > > logs
> > > > > and
> > > > > > counter, we are sure sd-sns-relation02.bj has input too.
> > > > > >
> > > > > >
> > > > > > My question is:
> > > > > >
> > > > > > 1. why rebalance failed?
> > > > > > 2. why owner info is wrong? btw: zkclient is 0.2
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> >
> >
> > *Best Regards 向河林*
> >
>
> --
>
>
> *Best Regards 向河林*
>
>
Re: owner info in zk is not correct
Posted by Jun Rao <ju...@gmail.com>.
This is true in 0.7.x. In 0.8.x, the fetcher will be restarted
automatically on such failures.
Thanks,
Jun
On Thu, May 22, 2014 at 10:40 PM, Helin Xiang <xk...@gmail.com> wrote:
> Hi Guozhang,
>
> sorry for asking a little unrelated problem here.
>
> We found a consumer stopping fetching data when doing an network upgrade,
> if the consumer have connection problem with one broker (but OK with
> zookeeper and other brokers)
> the fetcherrunnable will stop, but there is no chance to restart the thread
> again ( since there is no zookeeper timeout so no reblance will be
> triggered)
>
> kafka version is 0.7.2 and we are using high level consumer.
>
> I did a simulation test using iptables and the result is same( no chance to
> restart fetcherrunnable).
> I have read the code, seems the exception will not be handled outside. Am I
> wrong about this?
>
> THANKS.
>
>
>
>
>
> On Sat, May 17, 2014 at 12:19 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Hi Yonghui,
> >
> > Could you check if consumer2's fetcher thread is still alive? Also we
> have
> > an entry in FAQ wiki page about "consumer stopped consuming", current
> > Apache has some issues with the wiki page, but you may want to check it
> out
> > once Apache page resumes.
> >
> > Guozhang
> >
> >
> > On Wed, May 14, 2014 at 8:59 PM, Yonghui Zhao <zh...@gmail.com>
> > wrote:
> >
> > > Thanks GuoZhang.
> > >
> > > After last accident, we stop all consumers and then restart all
> consumers
> > > one by one then it is ok.
> > >
> > > 2 brokers, 10 partitions / broker, 3 consumers, each create 10
> stream
> > >
> > > So consumer1 consumes 10 partitions, consumer2 consumers another 10
> > > partitions, consumer3 is idle.
> > >
> > > Today we find some exceptions in consumer2, after these exceptions
> > > consumer2 doesn't work, no message is consumed.
> > >
> > > But in zk I found the owner ship doesn't change, consumer1 own 10
> > > partitions consumer 2 own another 10 partitions/
> > >
> > > How can we avoid this happen again?
> > >
> > >
> > > From the log we see the error sequence:
> > >
> > >
> > >
> > >
> > > *exception during commitOffsets, Reconnect in multifetch due to socket
> > > error, rebalance 2 times, error in FetcherRunnable'Seems the last
> > error
> > > in FetcherRunnable is deadly, after this error no message is consumed,
> > but
> > > zk ownership doesn't release.*Here is all the kafka related log in that
> > > time
> > > *:*
> > >
> > > [WARN 2014-05-13 16:19:05.020]
> > > kafka.utils.Logging$class.warn(Logging.scala:79)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> > > during commitOffsets]
> > > at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> > > at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> > > [INFO 2014-05-13 16:19:08.991]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c ZK expired;
> > > release old broker parition ownership; re-register consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c]
> > > [INFO 2014-05-13 16:19:08.991]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > registering consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
> > > [WARN 2014-05-13 16:19:09.001]
> > > kafka.utils.Logging$class.warn(Logging.scala:79)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> > > during commitOffsets]
> > > at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> > > at
> > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> > > at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> > > [INFO 2014-05-13 16:19:09.002]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > registering
> > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in
> > ZK]
> > > [INFO 2014-05-13 16:19:09.003]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > rebalancing consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > > [INFO 2014-05-13 16:19:09.063]
> > > kafka.utils.Logging$class.info(Logging.scala:69)
> > > [Reconnect in multifetch due to socket error: ]
> > > at kafka.utils.Utils$.read(Utils.scala:538)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> > > at
> kafka.network.Receive$class.readCompletely(Transmission.scala:55)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > at
> > kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
> > > at
> > > kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:117)
> > > at
> kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
> > > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > > [INFO 2014-05-13 16:19:09.068]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > > [INFO 2014-05-13 16:19:09.069]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > [INFO 2014-05-13 16:19:09.071]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Cleared all relevant queues for this fetcher]
> > > [INFO 2014-05-13 16:19:09.071]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.071]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.071]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.072]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.072]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.072]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.072]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.072]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.072]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.072]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.072]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Cleared the data chunks in all the consumer message iterators]
> > > [INFO 2014-05-13 16:19:09.073]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing
> > all
> > > offsets after clearing the fetcher queues]
> > > [INFO 2014-05-13 16:19:09.073]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > > partition ownership]
> > > [INFO 2014-05-13 16:19:09.073]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing
> > the
> > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> > > with consumers:
> > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > > [INFO 2014-05-13 16:19:09.073]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting
> > to
> > > claim partition 1-8]
> > > [INFO 2014-05-13 16:19:09.077]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting
> > to
> > > claim partition 1-4]
> > > [INFO 2014-05-13 16:19:09.081]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting
> > to
> > > claim partition 1-3]
> > > [INFO 2014-05-13 16:19:09.085]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting
> > to
> > > claim partition 1-2]
> > > [INFO 2014-05-13 16:19:09.089]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting
> > to
> > > claim partition 1-1]
> > > [INFO 2014-05-13 16:19:09.093]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting
> > to
> > > claim partition 1-5]
> > > [INFO 2014-05-13 16:19:09.097]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting
> > to
> > > claim partition 1-6]
> > > [INFO 2014-05-13 16:19:09.101]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting
> > to
> > > claim partition 1-9]
> > > [INFO 2014-05-13 16:19:09.105]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting
> > to
> > > claim partition 1-7]
> > > [INFO 2014-05-13 16:19:09.109]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting
> > to
> > > claim partition 1-0]
> > > [INFO 2014-05-13 16:19:09.118]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> successfully
> > > owned partition 1-2 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.122]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> successfully
> > > owned partition 1-3 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.127]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> successfully
> > > owned partition 1-5 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.132]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> successfully
> > > owned partition 1-0 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.137]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> successfully
> > > owned partition 1-9 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.153]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> successfully
> > > owned partition 1-6 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.157]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> successfully
> > > owned partition 1-7 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.161]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> successfully
> > > owned partition 1-1 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.166]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> successfully
> > > owned partition 1-4 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.170]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> successfully
> > > owned partition 1-8 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.170]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating
> the
> > > cache]
> > > [INFO 2014-05-13 16:19:09.171]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > > partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > > 1558487487]
> > > [INFO 2014-05-13 16:19:09.171]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > rebalancing
> > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try
> > #0]
> > > [INFO 2014-05-13 16:19:09.172]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset:
> 1563630746
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.172]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset:
> 1552261483
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.172]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617
> > from
> > > 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.172]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset:
> 1550214323
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.172]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset:
> 1551158177
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.173]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset:
> 1552939302
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.173]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset:
> 1550960225
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.173]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset:
> 1556683052
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.173]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset:
> 1555136587
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.173]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset:
> 1558487487
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.188]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > rebalancing consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > > [INFO 2014-05-13 16:19:09.240]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > > [INFO 2014-05-13 16:19:09.240]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > [INFO 2014-05-13 16:19:09.241]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Cleared all relevant queues for this fetcher]
> > > [INFO 2014-05-13 16:19:09.241]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.241]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.241]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.241]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.242]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.242]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.242]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.242]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.242]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.242]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.242]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Cleared the data chunks in all the consumer message iterators]
> > > [INFO 2014-05-13 16:19:09.242]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing
> > all
> > > offsets after clearing the fetcher queues]
> > > [INFO 2014-05-13 16:19:09.285]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > > partition ownership]
> > > [INFO 2014-05-13 16:19:09.333]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing
> > the
> > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> > > with consumers:
> > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > > [INFO 2014-05-13 16:19:09.333]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting
> > to
> > > claim partition 1-8]
> > > [INFO 2014-05-13 16:19:09.337]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting
> > to
> > > claim partition 1-4]
> > > [INFO 2014-05-13 16:19:09.341]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting
> > to
> > > claim partition 1-3]
> > > [INFO 2014-05-13 16:19:09.345]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting
> > to
> > > claim partition 1-2]
> > > [INFO 2014-05-13 16:19:09.349]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting
> > to
> > > claim partition 1-1]
> > > [INFO 2014-05-13 16:19:09.354]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting
> > to
> > > claim partition 1-5]
> > > [INFO 2014-05-13 16:19:09.358]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting
> > to
> > > claim partition 1-6]
> > > [INFO 2014-05-13 16:19:09.362]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting
> > to
> > > claim partition 1-9]
> > > [INFO 2014-05-13 16:19:09.366]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting
> > to
> > > claim partition 1-7]
> > > [INFO 2014-05-13 16:19:09.370]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting
> > to
> > > claim partition 1-0]
> > > [INFO 2014-05-13 16:19:09.379]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> successfully
> > > owned partition 1-2 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.390]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [/consumers/RecommendEvent/owners/sensei/1-3 exists with value
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 during
> > > connection loss; this is ok]
> > > [INFO 2014-05-13 16:19:09.390]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> successfully
> > > owned partition 1-3 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.395]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> successfully
> > > owned partition 1-5 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.400]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> successfully
> > > owned partition 1-0 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.405]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> successfully
> > > owned partition 1-9 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.409]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> successfully
> > > owned partition 1-6 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.417]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [/consumers/RecommendEvent/owners/sensei/1-7 exists with value
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 during
> > > connection loss; this is ok]
> > > [INFO 2014-05-13 16:19:09.417]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> successfully
> > > owned partition 1-7 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.422]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> successfully
> > > owned partition 1-1 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.430]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [/consumers/RecommendEvent/owners/sensei/1-4 exists with value
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 during
> > > connection loss; this is ok]
> > > [INFO 2014-05-13 16:19:09.430]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> successfully
> > > owned partition 1-4 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.435]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> successfully
> > > owned partition 1-8 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.435]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating
> the
> > > cache]
> > > [INFO 2014-05-13 16:19:09.435]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > > partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > > 1558487487]
> > > [INFO 2014-05-13 16:19:09.436]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > rebalancing
> > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try
> > #0]
> > > [INFO 2014-05-13 16:19:09.438]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset:
> 1563630746
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.438]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > rebalancing consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > > [INFO 2014-05-13 16:19:09.438]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset:
> 1552261483
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.438]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617
> > from
> > > 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.438]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset:
> 1550214323
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.438]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset:
> 1551158177
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.438]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset:
> 1552939302
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.438]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset:
> 1550960225
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.439]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset:
> 1556683052
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.439]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset:
> 1555136587
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.439]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset:
> 1558487487
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.492]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > > [INFO 2014-05-13 16:19:09.492]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > [INFO 2014-05-13 16:19:09.492]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Cleared all relevant queues for this fetcher]
> > > [INFO 2014-05-13 16:19:09.492]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.492]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.493]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.493]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.493]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.493]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.493]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.493]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.494]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.494]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Clearing the current data chunk for this consumer iterator]
> > > [INFO 2014-05-13 16:19:09.494]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [Cleared the data chunks in all the consumer message iterators]
> > > [INFO 2014-05-13 16:19:09.494]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing
> > all
> > > offsets after clearing the fetcher queues]
> > > [INFO 2014-05-13 16:19:09.539]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > > partition ownership]
> > > [INFO 2014-05-13 16:19:09.582]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing
> > the
> > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> > > with consumers:
> > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > > [INFO 2014-05-13 16:19:09.582]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting
> > to
> > > claim partition 1-8]
> > > [INFO 2014-05-13 16:19:09.586]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting
> > to
> > > claim partition 1-4]
> > > [INFO 2014-05-13 16:19:09.590]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting
> > to
> > > claim partition 1-3]
> > > [INFO 2014-05-13 16:19:09.594]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting
> > to
> > > claim partition 1-2]
> > > [INFO 2014-05-13 16:19:09.598]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting
> > to
> > > claim partition 1-1]
> > > [INFO 2014-05-13 16:19:09.602]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting
> > to
> > > claim partition 1-5]
> > > [INFO 2014-05-13 16:19:09.606]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting
> > to
> > > claim partition 1-6]
> > > [INFO 2014-05-13 16:19:09.610]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting
> > to
> > > claim partition 1-9]
> > > [INFO 2014-05-13 16:19:09.614]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting
> > to
> > > claim partition 1-7]
> > > [INFO 2014-05-13 16:19:09.618]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting
> > to
> > > claim partition 1-0]
> > > [INFO 2014-05-13 16:19:09.633]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> successfully
> > > owned partition 1-2 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.638]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> successfully
> > > owned partition 1-3 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.646]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> successfully
> > > owned partition 1-5 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.656]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> successfully
> > > owned partition 1-0 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.660]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> successfully
> > > owned partition 1-9 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.665]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> successfully
> > > owned partition 1-6 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.669]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> successfully
> > > owned partition 1-7 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.674]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> successfully
> > > owned partition 1-1 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.679]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> successfully
> > > owned partition 1-4 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.684]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> successfully
> > > owned partition 1-8 for topic sensei]
> > > [INFO 2014-05-13 16:19:09.684]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating
> the
> > > cache]
> > > [INFO 2014-05-13 16:19:09.684]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > > partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > > 1558487487]
> > > [INFO 2014-05-13 16:19:09.684]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > rebalancing
> > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try
> > #0]
> > > [INFO 2014-05-13 16:19:09.685]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset:
> 1563630746
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.685]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset:
> 1552261483
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.685]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617
> > from
> > > 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.685]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset:
> 1550214323
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.686]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset:
> 1551158177
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.686]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset:
> 1552939302
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.686]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset:
> 1550960225
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.686]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset:
> 1556683052
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.686]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset:
> 1555136587
> > > from 10.101.10.12:9092]
> > > [INFO 2014-05-13 16:19:09.686]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset:
> 1558487487
> > > from 10.101.10.12:9092]
> > > [ERROR 2014-05-13 16:19:18.687]
> > > kafka.utils.Logging$class.error(Logging.scala:89) [error in
> > FetcherRunnable
> > > ]
> > > at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:49)
> > > at
> > >
> >
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:186)
> > > at
> kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:113)
> > > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > > [INFO 2014-05-13 16:19:18.687]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > >
> > >
> > >
> > > 2014-05-12 23:22 GMT+08:00 Guozhang Wang <wa...@gmail.com>:
> > >
> > > > Hello Yonghui,
> > > >
> > > > In 0.7 the consumer rebalance logic is distributed and in some corner
> > > cases
> > > > such as soft-failure-caused-consecutive rebalances some consumer may
> > > > consider the rebalance as complete while others are still trying the
> > > > rebalance process. You can check the GC logs on your consumer to
> verify
> > > if
> > > > that is the case:
> > > >
> > > > https://issues.apache.org/jira/browse/KAFKA-242
> > > >
> > > > If you bounce the consumers to trigger another rebalance, this issue
> > > would
> > > > likely to be resolved.
> > > >
> > > > To solve this issue in 0.9 we are moving the group management like
> load
> > > > rebalance from the ZK-based distributed logic into a centralized
> > > > coordiantor. Details can be found here:
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <
> zhaoyonghui@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are using kafka 0.7.
> > > > >
> > > > > 2 brokers, each broker has 10 partitions for one topic
> > > > > 3 consumers in one consumer group, each consumer create 10 streams.
> > > > >
> > > > >
> > > > > Today, when we want to rollout new service.
> > > > > After we restart one consumer we find exceptions and warning.
> > > > >
> > > > > kafka.common.ConsumerRebalanceFailedException:
> > > > > RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't
> > > > rebalance
> > > > > after 4 retries
> > > > >
> > > > >
> > > > > [INFO 2014-05-12 15:17:47.364]
> > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored
> > > data:
> > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > > > [INFO 2014-05-12 15:17:47.366]
> > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting
> > for
> > > > the
> > > > > partition ownership to be deleted: 1-2]
> > > > > [INFO 2014-05-12 15:17:47.375]
> > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored
> > > data:
> > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > > > [INFO 2014-05-12 15:17:47.375]
> > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting
> > for
> > > > the
> > > > > partition ownership to be deleted: 1-3]
> > > > > [INFO 2014-05-12 15:17:47.385]
> > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored
> > > data:
> > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> > > > > [INFO 2014-05-12 15:17:47.386]
> > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting
> > for
> > > > the
> > > > > partition ownership to be deleted: 1-5]
> > > > >
> > > > >
> > > > >
> > > > > And I opened zk viewer.
> > > > >
> > > > > In zk, we found 2 consumers in ConsumerGroup/ids:
> > > > >
> > > > > RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
> > > > >
> > > > >
> > > > > And in owners/topic/ we found all partitions are assigned to
> > > > > sd-sns-relation03.bj:
> > > > >
> > > > > Here is the owner info:
> > > > > 1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > > 1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > > 1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > > 1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > > 1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > > 1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > > 1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > > 1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > > 1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > > 1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > >
> > > > > 2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > > 2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > > 2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > > 2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > > 2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > > 2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> > > > > 2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> > > > > 2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> > > > > 2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> > > > > 2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
> > > > >
> > > > >
> > > > > So all partitions are assigned to sd-sns-relation03.bj, but from
> > logs
> > > > and
> > > > > counter, we are sure sd-sns-relation02.bj has input too.
> > > > >
> > > > >
> > > > > My question is:
> > > > >
> > > > > 1. why rebalance failed?
> > > > > 2. why owner info is wrong? btw: zkclient is 0.2
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
>
>
> *Best Regards 向河林*
>
Re: owner info in zk is not correct
Posted by Helin Xiang <xk...@gmail.com>.
Hi Guozhang,
sorry for asking a little unrelated problem here.
We found a consumer stopping fetching data when doing an network upgrade,
if the consumer have connection problem with one broker (but OK with
zookeeper and other brokers)
the fetcherrunnable will stop, but there is no chance to restart the thread
again ( since there is no zookeeper timeout so no reblance will be
triggered)
kafka version is 0.7.2 and we are using high level consumer.
I did a simulation test using iptables and the result is same( no chance to
restart fetcherrunnable).
I have read the code, seems the exception will not be handled outside. Am I
wrong about this?
THANKS.
On Sat, May 17, 2014 at 12:19 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Yonghui,
>
> Could you check if consumer2's fetcher thread is still alive? Also we have
> an entry in FAQ wiki page about "consumer stopped consuming", current
> Apache has some issues with the wiki page, but you may want to check it out
> once Apache page resumes.
>
> Guozhang
>
>
> On Wed, May 14, 2014 at 8:59 PM, Yonghui Zhao <zh...@gmail.com>
> wrote:
>
> > Thanks GuoZhang.
> >
> > After last accident, we stop all consumers and then restart all consumers
> > one by one then it is ok.
> >
> > 2 brokers, 10 partitions / broker, 3 consumers, each create 10 stream
> >
> > So consumer1 consumes 10 partitions, consumer2 consumers another 10
> > partitions, consumer3 is idle.
> >
> > Today we find some exceptions in consumer2, after these exceptions
> > consumer2 doesn't work, no message is consumed.
> >
> > But in zk I found the owner ship doesn't change, consumer1 own 10
> > partitions consumer 2 own another 10 partitions/
> >
> > How can we avoid this happen again?
> >
> >
> > From the log we see the error sequence:
> >
> >
> >
> >
> > *exception during commitOffsets, Reconnect in multifetch due to socket
> > error, rebalance 2 times, error in FetcherRunnable'Seems the last
> error
> > in FetcherRunnable is deadly, after this error no message is consumed,
> but
> > zk ownership doesn't release.*Here is all the kafka related log in that
> > time
> > *:*
> >
> > [WARN 2014-05-13 16:19:05.020]
> > kafka.utils.Logging$class.warn(Logging.scala:79)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> > during commitOffsets]
> > at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> > at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> > [INFO 2014-05-13 16:19:08.991]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c ZK expired;
> > release old broker parition ownership; re-register consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c]
> > [INFO 2014-05-13 16:19:08.991]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > registering consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
> > [WARN 2014-05-13 16:19:09.001]
> > kafka.utils.Logging$class.warn(Logging.scala:79)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> > during commitOffsets]
> > at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> > at
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> > at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> > [INFO 2014-05-13 16:19:09.002]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> registering
> > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in
> ZK]
> > [INFO 2014-05-13 16:19:09.003]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > rebalancing consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > [INFO 2014-05-13 16:19:09.063]
> > kafka.utils.Logging$class.info(Logging.scala:69)
> > [Reconnect in multifetch due to socket error: ]
> > at kafka.utils.Utils$.read(Utils.scala:538)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> > at kafka.network.Receive$class.readCompletely(Transmission.scala:55)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > at
> kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
> > at
> > kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:117)
> > at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
> > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > [INFO 2014-05-13 16:19:09.068]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > [INFO 2014-05-13 16:19:09.069]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > [INFO 2014-05-13 16:19:09.071]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Cleared all relevant queues for this fetcher]
> > [INFO 2014-05-13 16:19:09.071]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.071]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.071]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.072]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.072]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.072]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.072]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.072]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.072]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.072]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.072]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Cleared the data chunks in all the consumer message iterators]
> > [INFO 2014-05-13 16:19:09.073]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing
> all
> > offsets after clearing the fetcher queues]
> > [INFO 2014-05-13 16:19:09.073]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > partition ownership]
> > [INFO 2014-05-13 16:19:09.073]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing
> the
> > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> > with consumers:
> > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > [INFO 2014-05-13 16:19:09.073]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting
> to
> > claim partition 1-8]
> > [INFO 2014-05-13 16:19:09.077]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting
> to
> > claim partition 1-4]
> > [INFO 2014-05-13 16:19:09.081]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting
> to
> > claim partition 1-3]
> > [INFO 2014-05-13 16:19:09.085]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting
> to
> > claim partition 1-2]
> > [INFO 2014-05-13 16:19:09.089]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting
> to
> > claim partition 1-1]
> > [INFO 2014-05-13 16:19:09.093]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting
> to
> > claim partition 1-5]
> > [INFO 2014-05-13 16:19:09.097]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting
> to
> > claim partition 1-6]
> > [INFO 2014-05-13 16:19:09.101]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting
> to
> > claim partition 1-9]
> > [INFO 2014-05-13 16:19:09.105]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting
> to
> > claim partition 1-7]
> > [INFO 2014-05-13 16:19:09.109]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting
> to
> > claim partition 1-0]
> > [INFO 2014-05-13 16:19:09.118]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
> > owned partition 1-2 for topic sensei]
> > [INFO 2014-05-13 16:19:09.122]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
> > owned partition 1-3 for topic sensei]
> > [INFO 2014-05-13 16:19:09.127]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
> > owned partition 1-5 for topic sensei]
> > [INFO 2014-05-13 16:19:09.132]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
> > owned partition 1-0 for topic sensei]
> > [INFO 2014-05-13 16:19:09.137]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
> > owned partition 1-9 for topic sensei]
> > [INFO 2014-05-13 16:19:09.153]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
> > owned partition 1-6 for topic sensei]
> > [INFO 2014-05-13 16:19:09.157]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
> > owned partition 1-7 for topic sensei]
> > [INFO 2014-05-13 16:19:09.161]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
> > owned partition 1-1 for topic sensei]
> > [INFO 2014-05-13 16:19:09.166]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
> > owned partition 1-4 for topic sensei]
> > [INFO 2014-05-13 16:19:09.170]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
> > owned partition 1-8 for topic sensei]
> > [INFO 2014-05-13 16:19:09.170]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
> > cache]
> > [INFO 2014-05-13 16:19:09.171]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > 1558487487]
> > [INFO 2014-05-13 16:19:09.171]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> rebalancing
> > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try
> #0]
> > [INFO 2014-05-13 16:19:09.172]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.172]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.172]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617
> from
> > 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.172]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.172]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.173]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.173]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.173]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.173]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.173]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.188]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > rebalancing consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > [INFO 2014-05-13 16:19:09.240]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > [INFO 2014-05-13 16:19:09.240]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > [INFO 2014-05-13 16:19:09.241]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Cleared all relevant queues for this fetcher]
> > [INFO 2014-05-13 16:19:09.241]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.241]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.241]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.241]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.242]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.242]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.242]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.242]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.242]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.242]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.242]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Cleared the data chunks in all the consumer message iterators]
> > [INFO 2014-05-13 16:19:09.242]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing
> all
> > offsets after clearing the fetcher queues]
> > [INFO 2014-05-13 16:19:09.285]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > partition ownership]
> > [INFO 2014-05-13 16:19:09.333]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing
> the
> > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> > with consumers:
> > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > [INFO 2014-05-13 16:19:09.333]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting
> to
> > claim partition 1-8]
> > [INFO 2014-05-13 16:19:09.337]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting
> to
> > claim partition 1-4]
> > [INFO 2014-05-13 16:19:09.341]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting
> to
> > claim partition 1-3]
> > [INFO 2014-05-13 16:19:09.345]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting
> to
> > claim partition 1-2]
> > [INFO 2014-05-13 16:19:09.349]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting
> to
> > claim partition 1-1]
> > [INFO 2014-05-13 16:19:09.354]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting
> to
> > claim partition 1-5]
> > [INFO 2014-05-13 16:19:09.358]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting
> to
> > claim partition 1-6]
> > [INFO 2014-05-13 16:19:09.362]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting
> to
> > claim partition 1-9]
> > [INFO 2014-05-13 16:19:09.366]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting
> to
> > claim partition 1-7]
> > [INFO 2014-05-13 16:19:09.370]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting
> to
> > claim partition 1-0]
> > [INFO 2014-05-13 16:19:09.379]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
> > owned partition 1-2 for topic sensei]
> > [INFO 2014-05-13 16:19:09.390]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [/consumers/RecommendEvent/owners/sensei/1-3 exists with value
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 during
> > connection loss; this is ok]
> > [INFO 2014-05-13 16:19:09.390]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
> > owned partition 1-3 for topic sensei]
> > [INFO 2014-05-13 16:19:09.395]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
> > owned partition 1-5 for topic sensei]
> > [INFO 2014-05-13 16:19:09.400]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
> > owned partition 1-0 for topic sensei]
> > [INFO 2014-05-13 16:19:09.405]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
> > owned partition 1-9 for topic sensei]
> > [INFO 2014-05-13 16:19:09.409]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
> > owned partition 1-6 for topic sensei]
> > [INFO 2014-05-13 16:19:09.417]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [/consumers/RecommendEvent/owners/sensei/1-7 exists with value
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 during
> > connection loss; this is ok]
> > [INFO 2014-05-13 16:19:09.417]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
> > owned partition 1-7 for topic sensei]
> > [INFO 2014-05-13 16:19:09.422]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
> > owned partition 1-1 for topic sensei]
> > [INFO 2014-05-13 16:19:09.430]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [/consumers/RecommendEvent/owners/sensei/1-4 exists with value
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 during
> > connection loss; this is ok]
> > [INFO 2014-05-13 16:19:09.430]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
> > owned partition 1-4 for topic sensei]
> > [INFO 2014-05-13 16:19:09.435]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
> > owned partition 1-8 for topic sensei]
> > [INFO 2014-05-13 16:19:09.435]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
> > cache]
> > [INFO 2014-05-13 16:19:09.435]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > 1558487487]
> > [INFO 2014-05-13 16:19:09.436]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> rebalancing
> > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try
> #0]
> > [INFO 2014-05-13 16:19:09.438]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.438]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > rebalancing consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > [INFO 2014-05-13 16:19:09.438]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.438]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617
> from
> > 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.438]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.438]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.438]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.438]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.439]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.439]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.439]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.492]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > [INFO 2014-05-13 16:19:09.492]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > [INFO 2014-05-13 16:19:09.492]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Cleared all relevant queues for this fetcher]
> > [INFO 2014-05-13 16:19:09.492]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.492]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.493]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.493]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.493]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.493]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.493]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.493]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.494]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.494]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Clearing the current data chunk for this consumer iterator]
> > [INFO 2014-05-13 16:19:09.494]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [Cleared the data chunks in all the consumer message iterators]
> > [INFO 2014-05-13 16:19:09.494]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing
> all
> > offsets after clearing the fetcher queues]
> > [INFO 2014-05-13 16:19:09.539]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > partition ownership]
> > [INFO 2014-05-13 16:19:09.582]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing
> the
> > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> > with consumers:
> > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > [INFO 2014-05-13 16:19:09.582]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting
> to
> > claim partition 1-8]
> > [INFO 2014-05-13 16:19:09.586]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting
> to
> > claim partition 1-4]
> > [INFO 2014-05-13 16:19:09.590]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting
> to
> > claim partition 1-3]
> > [INFO 2014-05-13 16:19:09.594]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting
> to
> > claim partition 1-2]
> > [INFO 2014-05-13 16:19:09.598]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting
> to
> > claim partition 1-1]
> > [INFO 2014-05-13 16:19:09.602]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting
> to
> > claim partition 1-5]
> > [INFO 2014-05-13 16:19:09.606]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting
> to
> > claim partition 1-6]
> > [INFO 2014-05-13 16:19:09.610]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting
> to
> > claim partition 1-9]
> > [INFO 2014-05-13 16:19:09.614]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting
> to
> > claim partition 1-7]
> > [INFO 2014-05-13 16:19:09.618]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting
> to
> > claim partition 1-0]
> > [INFO 2014-05-13 16:19:09.633]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
> > owned partition 1-2 for topic sensei]
> > [INFO 2014-05-13 16:19:09.638]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
> > owned partition 1-3 for topic sensei]
> > [INFO 2014-05-13 16:19:09.646]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
> > owned partition 1-5 for topic sensei]
> > [INFO 2014-05-13 16:19:09.656]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
> > owned partition 1-0 for topic sensei]
> > [INFO 2014-05-13 16:19:09.660]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
> > owned partition 1-9 for topic sensei]
> > [INFO 2014-05-13 16:19:09.665]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
> > owned partition 1-6 for topic sensei]
> > [INFO 2014-05-13 16:19:09.669]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
> > owned partition 1-7 for topic sensei]
> > [INFO 2014-05-13 16:19:09.674]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
> > owned partition 1-1 for topic sensei]
> > [INFO 2014-05-13 16:19:09.679]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
> > owned partition 1-4 for topic sensei]
> > [INFO 2014-05-13 16:19:09.684]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
> > owned partition 1-8 for topic sensei]
> > [INFO 2014-05-13 16:19:09.684]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
> > cache]
> > [INFO 2014-05-13 16:19:09.684]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > 1558487487]
> > [INFO 2014-05-13 16:19:09.684]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> rebalancing
> > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try
> #0]
> > [INFO 2014-05-13 16:19:09.685]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.685]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.685]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617
> from
> > 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.685]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.686]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.686]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.686]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.686]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.686]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
> > from 10.101.10.12:9092]
> > [INFO 2014-05-13 16:19:09.686]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
> > from 10.101.10.12:9092]
> > [ERROR 2014-05-13 16:19:18.687]
> > kafka.utils.Logging$class.error(Logging.scala:89) [error in
> FetcherRunnable
> > ]
> > at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:49)
> > at
> >
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:186)
> > at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:113)
> > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > [INFO 2014-05-13 16:19:18.687]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> >
> >
> >
> > 2014-05-12 23:22 GMT+08:00 Guozhang Wang <wa...@gmail.com>:
> >
> > > Hello Yonghui,
> > >
> > > In 0.7 the consumer rebalance logic is distributed and in some corner
> > cases
> > > such as soft-failure-caused-consecutive rebalances some consumer may
> > > consider the rebalance as complete while others are still trying the
> > > rebalance process. You can check the GC logs on your consumer to verify
> > if
> > > that is the case:
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-242
> > >
> > > If you bounce the consumers to trigger another rebalance, this issue
> > would
> > > likely to be resolved.
> > >
> > > To solve this issue in 0.9 we are moving the group management like load
> > > rebalance from the ZK-based distributed logic into a centralized
> > > coordiantor. Details can be found here:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> > >
> > > Guozhang
> > >
> > >
> > > On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <zhaoyonghui@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > We are using kafka 0.7.
> > > >
> > > > 2 brokers, each broker has 10 partitions for one topic
> > > > 3 consumers in one consumer group, each consumer create 10 streams.
> > > >
> > > >
> > > > Today, when we want to rollout new service.
> > > > After we restart one consumer we find exceptions and warning.
> > > >
> > > > kafka.common.ConsumerRebalanceFailedException:
> > > > RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't
> > > rebalance
> > > > after 4 retries
> > > >
> > > >
> > > > [INFO 2014-05-12 15:17:47.364]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored
> > data:
> > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > > [INFO 2014-05-12 15:17:47.366]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting
> for
> > > the
> > > > partition ownership to be deleted: 1-2]
> > > > [INFO 2014-05-12 15:17:47.375]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored
> > data:
> > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > > [INFO 2014-05-12 15:17:47.375]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting
> for
> > > the
> > > > partition ownership to be deleted: 1-3]
> > > > [INFO 2014-05-12 15:17:47.385]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored
> > data:
> > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> > > > [INFO 2014-05-12 15:17:47.386]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting
> for
> > > the
> > > > partition ownership to be deleted: 1-5]
> > > >
> > > >
> > > >
> > > > And I opened zk viewer.
> > > >
> > > > In zk, we found 2 consumers in ConsumerGroup/ids:
> > > >
> > > > RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
> > > >
> > > >
> > > > And in owners/topic/ we found all partitions are assigned to
> > > > sd-sns-relation03.bj:
> > > >
> > > > Here is the owner info:
> > > > 1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > 1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > 1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > 1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > 1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > 1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > 1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > 1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > 1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > 1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > >
> > > > 2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > 2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > 2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > 2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > 2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > 2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> > > > 2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> > > > 2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> > > > 2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> > > > 2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
> > > >
> > > >
> > > > So all partitions are assigned to sd-sns-relation03.bj, but from
> logs
> > > and
> > > > counter, we are sure sd-sns-relation02.bj has input too.
> > > >
> > > >
> > > > My question is:
> > > >
> > > > 1. why rebalance failed?
> > > > 2. why owner info is wrong? btw: zkclient is 0.2
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
--
*Best Regards 向河林*
Re: owner info in zk is not correct
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Yonghui,
Could you check if consumer2's fetcher thread is still alive? Also we have
an entry in FAQ wiki page about "consumer stopped consuming", current
Apache has some issues with the wiki page, but you may want to check it out
once Apache page resumes.
Guozhang
On Wed, May 14, 2014 at 8:59 PM, Yonghui Zhao <zh...@gmail.com> wrote:
> Thanks GuoZhang.
>
> After last accident, we stop all consumers and then restart all consumers
> one by one then it is ok.
>
> 2 brokers, 10 partitions / broker, 3 consumers, each create 10 stream
>
> So consumer1 consumes 10 partitions, consumer2 consumers another 10
> partitions, consumer3 is idle.
>
> Today we find some exceptions in consumer2, after these exceptions
> consumer2 doesn't work, no message is consumed.
>
> But in zk I found the owner ship doesn't change, consumer1 own 10
> partitions consumer 2 own another 10 partitions/
>
> How can we avoid this happen again?
>
>
> From the log we see the error sequence:
>
>
>
>
> *exception during commitOffsets, Reconnect in multifetch due to socket
> error, rebalance 2 times, error in FetcherRunnable'Seems the last error
> in FetcherRunnable is deadly, after this error no message is consumed, but
> zk ownership doesn't release.*Here is all the kafka related log in that
> time
> *:*
>
> [WARN 2014-05-13 16:19:05.020]
> kafka.utils.Logging$class.warn(Logging.scala:79)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> during commitOffsets]
> at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> [INFO 2014-05-13 16:19:08.991]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c ZK expired;
> release old broker parition ownership; re-register consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c]
> [INFO 2014-05-13 16:19:08.991]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> registering consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
> [WARN 2014-05-13 16:19:09.001]
> kafka.utils.Logging$class.warn(Logging.scala:79)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> during commitOffsets]
> at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> [INFO 2014-05-13 16:19:09.002]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end registering
> consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
> [INFO 2014-05-13 16:19:09.003]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> rebalancing consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO 2014-05-13 16:19:09.063]
> kafka.utils.Logging$class.info(Logging.scala:69)
> [Reconnect in multifetch due to socket error: ]
> at kafka.utils.Utils$.read(Utils.scala:538)
> at
>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Receive$class.readCompletely(Transmission.scala:55)
> at
>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> at kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
> at
> kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:117)
> at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
> at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> [INFO 2014-05-13 16:19:09.068]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> [INFO 2014-05-13 16:19:09.069]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> [INFO 2014-05-13 16:19:09.071]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared all relevant queues for this fetcher]
> [INFO 2014-05-13 16:19:09.071]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.071]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.071]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared the data chunks in all the consumer message iterators]
> [INFO 2014-05-13 16:19:09.073]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
> offsets after clearing the fetcher queues]
> [INFO 2014-05-13 16:19:09.073]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> partition ownership]
> [INFO 2014-05-13 16:19:09.073]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
> following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> with consumers:
> List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> [INFO 2014-05-13 16:19:09.073]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
> claim partition 1-8]
> [INFO 2014-05-13 16:19:09.077]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
> claim partition 1-4]
> [INFO 2014-05-13 16:19:09.081]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
> claim partition 1-3]
> [INFO 2014-05-13 16:19:09.085]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
> claim partition 1-2]
> [INFO 2014-05-13 16:19:09.089]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
> claim partition 1-1]
> [INFO 2014-05-13 16:19:09.093]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
> claim partition 1-5]
> [INFO 2014-05-13 16:19:09.097]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
> claim partition 1-6]
> [INFO 2014-05-13 16:19:09.101]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
> claim partition 1-9]
> [INFO 2014-05-13 16:19:09.105]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
> claim partition 1-7]
> [INFO 2014-05-13 16:19:09.109]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
> claim partition 1-0]
> [INFO 2014-05-13 16:19:09.118]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
> owned partition 1-2 for topic sensei]
> [INFO 2014-05-13 16:19:09.122]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
> owned partition 1-3 for topic sensei]
> [INFO 2014-05-13 16:19:09.127]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
> owned partition 1-5 for topic sensei]
> [INFO 2014-05-13 16:19:09.132]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
> owned partition 1-0 for topic sensei]
> [INFO 2014-05-13 16:19:09.137]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
> owned partition 1-9 for topic sensei]
> [INFO 2014-05-13 16:19:09.153]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
> owned partition 1-6 for topic sensei]
> [INFO 2014-05-13 16:19:09.157]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
> owned partition 1-7 for topic sensei]
> [INFO 2014-05-13 16:19:09.161]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
> owned partition 1-1 for topic sensei]
> [INFO 2014-05-13 16:19:09.166]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
> owned partition 1-4 for topic sensei]
> [INFO 2014-05-13 16:19:09.170]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
> owned partition 1-8 for topic sensei]
> [INFO 2014-05-13 16:19:09.170]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
> cache]
> [INFO 2014-05-13 16:19:09.171]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> 1558487487]
> [INFO 2014-05-13 16:19:09.171]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
> consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO 2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
> 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.188]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> rebalancing consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO 2014-05-13 16:19:09.240]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> [INFO 2014-05-13 16:19:09.240]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> [INFO 2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared all relevant queues for this fetcher]
> [INFO 2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared the data chunks in all the consumer message iterators]
> [INFO 2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
> offsets after clearing the fetcher queues]
> [INFO 2014-05-13 16:19:09.285]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> partition ownership]
> [INFO 2014-05-13 16:19:09.333]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
> following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> with consumers:
> List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> [INFO 2014-05-13 16:19:09.333]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
> claim partition 1-8]
> [INFO 2014-05-13 16:19:09.337]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
> claim partition 1-4]
> [INFO 2014-05-13 16:19:09.341]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
> claim partition 1-3]
> [INFO 2014-05-13 16:19:09.345]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
> claim partition 1-2]
> [INFO 2014-05-13 16:19:09.349]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
> claim partition 1-1]
> [INFO 2014-05-13 16:19:09.354]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
> claim partition 1-5]
> [INFO 2014-05-13 16:19:09.358]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
> claim partition 1-6]
> [INFO 2014-05-13 16:19:09.362]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
> claim partition 1-9]
> [INFO 2014-05-13 16:19:09.366]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
> claim partition 1-7]
> [INFO 2014-05-13 16:19:09.370]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
> claim partition 1-0]
> [INFO 2014-05-13 16:19:09.379]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
> owned partition 1-2 for topic sensei]
> [INFO 2014-05-13 16:19:09.390]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [/consumers/RecommendEvent/owners/sensei/1-3 exists with value
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 during
> connection loss; this is ok]
> [INFO 2014-05-13 16:19:09.390]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
> owned partition 1-3 for topic sensei]
> [INFO 2014-05-13 16:19:09.395]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
> owned partition 1-5 for topic sensei]
> [INFO 2014-05-13 16:19:09.400]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
> owned partition 1-0 for topic sensei]
> [INFO 2014-05-13 16:19:09.405]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
> owned partition 1-9 for topic sensei]
> [INFO 2014-05-13 16:19:09.409]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
> owned partition 1-6 for topic sensei]
> [INFO 2014-05-13 16:19:09.417]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [/consumers/RecommendEvent/owners/sensei/1-7 exists with value
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 during
> connection loss; this is ok]
> [INFO 2014-05-13 16:19:09.417]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
> owned partition 1-7 for topic sensei]
> [INFO 2014-05-13 16:19:09.422]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
> owned partition 1-1 for topic sensei]
> [INFO 2014-05-13 16:19:09.430]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [/consumers/RecommendEvent/owners/sensei/1-4 exists with value
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 during
> connection loss; this is ok]
> [INFO 2014-05-13 16:19:09.430]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
> owned partition 1-4 for topic sensei]
> [INFO 2014-05-13 16:19:09.435]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
> owned partition 1-8 for topic sensei]
> [INFO 2014-05-13 16:19:09.435]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
> cache]
> [INFO 2014-05-13 16:19:09.435]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> 1558487487]
> [INFO 2014-05-13 16:19:09.436]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
> consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO 2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> rebalancing consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO 2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
> 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.439]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.439]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.439]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> [INFO 2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> [INFO 2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared all relevant queues for this fetcher]
> [INFO 2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.494]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.494]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO 2014-05-13 16:19:09.494]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared the data chunks in all the consumer message iterators]
> [INFO 2014-05-13 16:19:09.494]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
> offsets after clearing the fetcher queues]
> [INFO 2014-05-13 16:19:09.539]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> partition ownership]
> [INFO 2014-05-13 16:19:09.582]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
> following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> with consumers:
> List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> [INFO 2014-05-13 16:19:09.582]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
> claim partition 1-8]
> [INFO 2014-05-13 16:19:09.586]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
> claim partition 1-4]
> [INFO 2014-05-13 16:19:09.590]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
> claim partition 1-3]
> [INFO 2014-05-13 16:19:09.594]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
> claim partition 1-2]
> [INFO 2014-05-13 16:19:09.598]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
> claim partition 1-1]
> [INFO 2014-05-13 16:19:09.602]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
> claim partition 1-5]
> [INFO 2014-05-13 16:19:09.606]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
> claim partition 1-6]
> [INFO 2014-05-13 16:19:09.610]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
> claim partition 1-9]
> [INFO 2014-05-13 16:19:09.614]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
> claim partition 1-7]
> [INFO 2014-05-13 16:19:09.618]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
> claim partition 1-0]
> [INFO 2014-05-13 16:19:09.633]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
> owned partition 1-2 for topic sensei]
> [INFO 2014-05-13 16:19:09.638]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
> owned partition 1-3 for topic sensei]
> [INFO 2014-05-13 16:19:09.646]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
> owned partition 1-5 for topic sensei]
> [INFO 2014-05-13 16:19:09.656]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
> owned partition 1-0 for topic sensei]
> [INFO 2014-05-13 16:19:09.660]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
> owned partition 1-9 for topic sensei]
> [INFO 2014-05-13 16:19:09.665]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
> owned partition 1-6 for topic sensei]
> [INFO 2014-05-13 16:19:09.669]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
> owned partition 1-7 for topic sensei]
> [INFO 2014-05-13 16:19:09.674]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
> owned partition 1-1 for topic sensei]
> [INFO 2014-05-13 16:19:09.679]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
> owned partition 1-4 for topic sensei]
> [INFO 2014-05-13 16:19:09.684]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
> owned partition 1-8 for topic sensei]
> [INFO 2014-05-13 16:19:09.684]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
> cache]
> [INFO 2014-05-13 16:19:09.684]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> 1558487487]
> [INFO 2014-05-13 16:19:09.684]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
> consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO 2014-05-13 16:19:09.685]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.685]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.685]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
> 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.685]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
> from 10.101.10.12:9092]
> [INFO 2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
> from 10.101.10.12:9092]
> [ERROR 2014-05-13 16:19:18.687]
> kafka.utils.Logging$class.error(Logging.scala:89) [error in FetcherRunnable
> ]
> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:49)
> at
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:186)
> at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:113)
> at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> [INFO 2014-05-13 16:19:18.687]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
>
>
>
> 2014-05-12 23:22 GMT+08:00 Guozhang Wang <wa...@gmail.com>:
>
> > Hello Yonghui,
> >
> > In 0.7 the consumer rebalance logic is distributed and in some corner
> cases
> > such as soft-failure-caused-consecutive rebalances some consumer may
> > consider the rebalance as complete while others are still trying the
> > rebalance process. You can check the GC logs on your consumer to verify
> if
> > that is the case:
> >
> > https://issues.apache.org/jira/browse/KAFKA-242
> >
> > If you bounce the consumers to trigger another rebalance, this issue
> would
> > likely to be resolved.
> >
> > To solve this issue in 0.9 we are moving the group management like load
> > rebalance from the ZK-based distributed logic into a centralized
> > coordiantor. Details can be found here:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> >
> > Guozhang
> >
> >
> > On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <zhaoyonghui@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > We are using kafka 0.7.
> > >
> > > 2 brokers, each broker has 10 partitions for one topic
> > > 3 consumers in one consumer group, each consumer create 10 streams.
> > >
> > >
> > > Today, when we want to rollout new service.
> > > After we restart one consumer we find exceptions and warning.
> > >
> > > kafka.common.ConsumerRebalanceFailedException:
> > > RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't
> > rebalance
> > > after 4 retries
> > >
> > >
> > > [INFO 2014-05-12 15:17:47.364]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored
> data:
> > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > [INFO 2014-05-12 15:17:47.366]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> > the
> > > partition ownership to be deleted: 1-2]
> > > [INFO 2014-05-12 15:17:47.375]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored
> data:
> > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > [INFO 2014-05-12 15:17:47.375]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> > the
> > > partition ownership to be deleted: 1-3]
> > > [INFO 2014-05-12 15:17:47.385]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored
> data:
> > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> > > [INFO 2014-05-12 15:17:47.386]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> > the
> > > partition ownership to be deleted: 1-5]
> > >
> > >
> > >
> > > And I opened zk viewer.
> > >
> > > In zk, we found 2 consumers in ConsumerGroup/ids:
> > >
> > > RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
> > >
> > >
> > > And in owners/topic/ we found all partitions are assigned to
> > > sd-sns-relation03.bj:
> > >
> > > Here is the owner info:
> > > 1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > 1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > 1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > 1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > 1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > 1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > 1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > 1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > 1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > 1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > >
> > > 2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > 2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > 2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > 2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > 2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > 2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> > > 2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> > > 2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> > > 2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> > > 2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
> > >
> > >
> > > So all partitions are assigned to sd-sns-relation03.bj, but from logs
> > and
> > > counter, we are sure sd-sns-relation02.bj has input too.
> > >
> > >
> > > My question is:
> > >
> > > 1. why rebalance failed?
> > > 2. why owner info is wrong? btw: zkclient is 0.2
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: owner info in zk is not correct
Posted by Yonghui Zhao <zh...@gmail.com>.
Thanks GuoZhang.
After last accident, we stop all consumers and then restart all consumers
one by one then it is ok.
2 brokers, 10 partitions / broker, 3 consumers, each create 10 stream
So consumer1 consumes 10 partitions, consumer2 consumers another 10
partitions, consumer3 is idle.
Today we find some exceptions in consumer2, after these exceptions
consumer2 doesn't work, no message is consumed.
But in zk I found the owner ship doesn't change, consumer1 own 10
partitions consumer 2 own another 10 partitions/
How can we avoid this happen again?
From the log we see the error sequence:
*exception during commitOffsets, Reconnect in multifetch due to socket
error, rebalance 2 times, error in FetcherRunnable'Seems the last error
in FetcherRunnable is deadly, after this error no message is consumed, but
zk ownership doesn't release.*Here is all the kafka related log in that time
*:*
[WARN 2014-05-13 16:19:05.020]
kafka.utils.Logging$class.warn(Logging.scala:79)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
during commitOffsets]
at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
at
kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
at
kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
[INFO 2014-05-13 16:19:08.991]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c ZK expired;
release old broker parition ownership; re-register consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c]
[INFO 2014-05-13 16:19:08.991]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
registering consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
[WARN 2014-05-13 16:19:09.001]
kafka.utils.Logging$class.warn(Logging.scala:79)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
during commitOffsets]
at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
at
kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
at
kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
[INFO 2014-05-13 16:19:09.002]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end registering
consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
[INFO 2014-05-13 16:19:09.003]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
rebalancing consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.063]
kafka.utils.Logging$class.info(Logging.scala:69)
[Reconnect in multifetch due to socket error: ]
at kafka.utils.Utils$.read(Utils.scala:538)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Receive$class.readCompletely(Transmission.scala:55)
at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
at kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:117)
at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
[INFO 2014-05-13 16:19:09.068]
kafka.utils.Logging$class.info(Logging.scala:61)
[FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
[INFO 2014-05-13 16:19:09.069]
kafka.utils.Logging$class.info(Logging.scala:61)
[stopping fetcher FetchRunnable-0 to host 10.101.10.12]
[INFO 2014-05-13 16:19:09.071]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared all relevant queues for this fetcher]
[INFO 2014-05-13 16:19:09.071]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.071]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.071]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared the data chunks in all the consumer message iterators]
[INFO 2014-05-13 16:19:09.073]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
offsets after clearing the fetcher queues]
[INFO 2014-05-13 16:19:09.073]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
partition ownership]
[INFO 2014-05-13 16:19:09.073]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
with consumers:
List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
[INFO 2014-05-13 16:19:09.073]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
claim partition 1-8]
[INFO 2014-05-13 16:19:09.077]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
claim partition 1-4]
[INFO 2014-05-13 16:19:09.081]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
claim partition 1-3]
[INFO 2014-05-13 16:19:09.085]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
claim partition 1-2]
[INFO 2014-05-13 16:19:09.089]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
claim partition 1-1]
[INFO 2014-05-13 16:19:09.093]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
claim partition 1-5]
[INFO 2014-05-13 16:19:09.097]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
claim partition 1-6]
[INFO 2014-05-13 16:19:09.101]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
claim partition 1-9]
[INFO 2014-05-13 16:19:09.105]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
claim partition 1-7]
[INFO 2014-05-13 16:19:09.109]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
claim partition 1-0]
[INFO 2014-05-13 16:19:09.118]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
owned partition 1-2 for topic sensei]
[INFO 2014-05-13 16:19:09.122]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
owned partition 1-3 for topic sensei]
[INFO 2014-05-13 16:19:09.127]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
owned partition 1-5 for topic sensei]
[INFO 2014-05-13 16:19:09.132]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
owned partition 1-0 for topic sensei]
[INFO 2014-05-13 16:19:09.137]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
owned partition 1-9 for topic sensei]
[INFO 2014-05-13 16:19:09.153]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
owned partition 1-6 for topic sensei]
[INFO 2014-05-13 16:19:09.157]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
owned partition 1-7 for topic sensei]
[INFO 2014-05-13 16:19:09.161]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
owned partition 1-1 for topic sensei]
[INFO 2014-05-13 16:19:09.166]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
owned partition 1-4 for topic sensei]
[INFO 2014-05-13 16:19:09.170]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
owned partition 1-8 for topic sensei]
[INFO 2014-05-13 16:19:09.170]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
cache]
[INFO 2014-05-13 16:19:09.171]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
1558487487]
[INFO 2014-05-13 16:19:09.171]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.188]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
rebalancing consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.240]
kafka.utils.Logging$class.info(Logging.scala:61)
[FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
[INFO 2014-05-13 16:19:09.240]
kafka.utils.Logging$class.info(Logging.scala:61)
[stopping fetcher FetchRunnable-0 to host 10.101.10.12]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared all relevant queues for this fetcher]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared the data chunks in all the consumer message iterators]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
offsets after clearing the fetcher queues]
[INFO 2014-05-13 16:19:09.285]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
partition ownership]
[INFO 2014-05-13 16:19:09.333]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
with consumers:
List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
[INFO 2014-05-13 16:19:09.333]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
claim partition 1-8]
[INFO 2014-05-13 16:19:09.337]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
claim partition 1-4]
[INFO 2014-05-13 16:19:09.341]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
claim partition 1-3]
[INFO 2014-05-13 16:19:09.345]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
claim partition 1-2]
[INFO 2014-05-13 16:19:09.349]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
claim partition 1-1]
[INFO 2014-05-13 16:19:09.354]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
claim partition 1-5]
[INFO 2014-05-13 16:19:09.358]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
claim partition 1-6]
[INFO 2014-05-13 16:19:09.362]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
claim partition 1-9]
[INFO 2014-05-13 16:19:09.366]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
claim partition 1-7]
[INFO 2014-05-13 16:19:09.370]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
claim partition 1-0]
[INFO 2014-05-13 16:19:09.379]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
owned partition 1-2 for topic sensei]
[INFO 2014-05-13 16:19:09.390]
kafka.utils.Logging$class.info(Logging.scala:61)
[/consumers/RecommendEvent/owners/sensei/1-3 exists with value
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 during
connection loss; this is ok]
[INFO 2014-05-13 16:19:09.390]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
owned partition 1-3 for topic sensei]
[INFO 2014-05-13 16:19:09.395]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
owned partition 1-5 for topic sensei]
[INFO 2014-05-13 16:19:09.400]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
owned partition 1-0 for topic sensei]
[INFO 2014-05-13 16:19:09.405]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
owned partition 1-9 for topic sensei]
[INFO 2014-05-13 16:19:09.409]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
owned partition 1-6 for topic sensei]
[INFO 2014-05-13 16:19:09.417]
kafka.utils.Logging$class.info(Logging.scala:61)
[/consumers/RecommendEvent/owners/sensei/1-7 exists with value
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 during
connection loss; this is ok]
[INFO 2014-05-13 16:19:09.417]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
owned partition 1-7 for topic sensei]
[INFO 2014-05-13 16:19:09.422]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
owned partition 1-1 for topic sensei]
[INFO 2014-05-13 16:19:09.430]
kafka.utils.Logging$class.info(Logging.scala:61)
[/consumers/RecommendEvent/owners/sensei/1-4 exists with value
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 during
connection loss; this is ok]
[INFO 2014-05-13 16:19:09.430]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
owned partition 1-4 for topic sensei]
[INFO 2014-05-13 16:19:09.435]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
owned partition 1-8 for topic sensei]
[INFO 2014-05-13 16:19:09.435]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
cache]
[INFO 2014-05-13 16:19:09.435]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
1558487487]
[INFO 2014-05-13 16:19:09.436]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
rebalancing consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.439]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.439]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.439]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[stopping fetcher FetchRunnable-0 to host 10.101.10.12]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared all relevant queues for this fetcher]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.494]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.494]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.494]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared the data chunks in all the consumer message iterators]
[INFO 2014-05-13 16:19:09.494]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
offsets after clearing the fetcher queues]
[INFO 2014-05-13 16:19:09.539]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
partition ownership]
[INFO 2014-05-13 16:19:09.582]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
with consumers:
List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
[INFO 2014-05-13 16:19:09.582]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
claim partition 1-8]
[INFO 2014-05-13 16:19:09.586]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
claim partition 1-4]
[INFO 2014-05-13 16:19:09.590]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
claim partition 1-3]
[INFO 2014-05-13 16:19:09.594]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
claim partition 1-2]
[INFO 2014-05-13 16:19:09.598]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
claim partition 1-1]
[INFO 2014-05-13 16:19:09.602]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
claim partition 1-5]
[INFO 2014-05-13 16:19:09.606]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
claim partition 1-6]
[INFO 2014-05-13 16:19:09.610]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
claim partition 1-9]
[INFO 2014-05-13 16:19:09.614]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
claim partition 1-7]
[INFO 2014-05-13 16:19:09.618]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
claim partition 1-0]
[INFO 2014-05-13 16:19:09.633]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
owned partition 1-2 for topic sensei]
[INFO 2014-05-13 16:19:09.638]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
owned partition 1-3 for topic sensei]
[INFO 2014-05-13 16:19:09.646]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
owned partition 1-5 for topic sensei]
[INFO 2014-05-13 16:19:09.656]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
owned partition 1-0 for topic sensei]
[INFO 2014-05-13 16:19:09.660]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
owned partition 1-9 for topic sensei]
[INFO 2014-05-13 16:19:09.665]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
owned partition 1-6 for topic sensei]
[INFO 2014-05-13 16:19:09.669]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
owned partition 1-7 for topic sensei]
[INFO 2014-05-13 16:19:09.674]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
owned partition 1-1 for topic sensei]
[INFO 2014-05-13 16:19:09.679]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
owned partition 1-4 for topic sensei]
[INFO 2014-05-13 16:19:09.684]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
owned partition 1-8 for topic sensei]
[INFO 2014-05-13 16:19:09.684]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
cache]
[INFO 2014-05-13 16:19:09.684]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
1558487487]
[INFO 2014-05-13 16:19:09.684]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.685]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.685]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.685]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.685]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
from 10.101.10.12:9092]
[ERROR 2014-05-13 16:19:18.687]
kafka.utils.Logging$class.error(Logging.scala:89) [error in FetcherRunnable
]
at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:49)
at
kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:186)
at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:113)
at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
[INFO 2014-05-13 16:19:18.687]
kafka.utils.Logging$class.info(Logging.scala:61)
[stopping fetcher FetchRunnable-0 to host 10.101.10.12]
2014-05-12 23:22 GMT+08:00 Guozhang Wang <wa...@gmail.com>:
> Hello Yonghui,
>
> In 0.7 the consumer rebalance logic is distributed and in some corner cases
> such as soft-failure-caused-consecutive rebalances some consumer may
> consider the rebalance as complete while others are still trying the
> rebalance process. You can check the GC logs on your consumer to verify if
> that is the case:
>
> https://issues.apache.org/jira/browse/KAFKA-242
>
> If you bounce the consumers to trigger another rebalance, this issue would
> likely to be resolved.
>
> To solve this issue in 0.9 we are moving the group management like load
> rebalance from the ZK-based distributed logic into a centralized
> coordiantor. Details can be found here:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
>
> Guozhang
>
>
> On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <zhaoyonghui@gmail.com
> >wrote:
>
> > Hi,
> >
> > We are using kafka 0.7.
> >
> > 2 brokers, each broker has 10 partitions for one topic
> > 3 consumers in one consumer group, each consumer create 10 streams.
> >
> >
> > Today, when we want to rollout new service.
> > After we restart one consumer we find exceptions and warning.
> >
> > kafka.common.ConsumerRebalanceFailedException:
> > RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't
> rebalance
> > after 4 retries
> >
> >
> > [INFO 2014-05-12 15:17:47.364]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored data:
> > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > [INFO 2014-05-12 15:17:47.366]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> the
> > partition ownership to be deleted: 1-2]
> > [INFO 2014-05-12 15:17:47.375]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored data:
> > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > [INFO 2014-05-12 15:17:47.375]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> the
> > partition ownership to be deleted: 1-3]
> > [INFO 2014-05-12 15:17:47.385]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored data:
> > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> > [INFO 2014-05-12 15:17:47.386]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> the
> > partition ownership to be deleted: 1-5]
> >
> >
> >
> > And I opened zk viewer.
> >
> > In zk, we found 2 consumers in ConsumerGroup/ids:
> >
> > RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
> >
> >
> > And in owners/topic/ we found all partitions are assigned to
> > sd-sns-relation03.bj:
> >
> > Here is the owner info:
> > 1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > 1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > 1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > 1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > 1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > 1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > 1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > 1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > 1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > 1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> >
> > 2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > 2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > 2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > 2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > 2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > 2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> > 2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> > 2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> > 2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> > 2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
> >
> >
> > So all partitions are assigned to sd-sns-relation03.bj, but from logs
> and
> > counter, we are sure sd-sns-relation02.bj has input too.
> >
> >
> > My question is:
> >
> > 1. why rebalance failed?
> > 2. why owner info is wrong? btw: zkclient is 0.2
> >
>
>
>
> --
> -- Guozhang
>
Re: owner info in zk is not correct
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Yonghui,
In 0.7 the consumer rebalance logic is distributed and in some corner cases
such as soft-failure-caused-consecutive rebalances some consumer may
consider the rebalance as complete while others are still trying the
rebalance process. You can check the GC logs on your consumer to verify if
that is the case:
https://issues.apache.org/jira/browse/KAFKA-242
If you bounce the consumers to trigger another rebalance, this issue would
likely to be resolved.
To solve this issue in 0.9 we are moving the group management like load
rebalance from the ZK-based distributed logic into a centralized
coordiantor. Details can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
Guozhang
On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <zh...@gmail.com>wrote:
> Hi,
>
> We are using kafka 0.7.
>
> 2 brokers, each broker has 10 partitions for one topic
> 3 consumers in one consumer group, each consumer create 10 streams.
>
>
> Today, when we want to rollout new service.
> After we restart one consumer we find exceptions and warning.
>
> kafka.common.ConsumerRebalanceFailedException:
> RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't rebalance
> after 4 retries
>
>
> [INFO 2014-05-12 15:17:47.364]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored data:
> RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> [INFO 2014-05-12 15:17:47.366]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the
> partition ownership to be deleted: 1-2]
> [INFO 2014-05-12 15:17:47.375]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored data:
> RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> [INFO 2014-05-12 15:17:47.375]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the
> partition ownership to be deleted: 1-3]
> [INFO 2014-05-12 15:17:47.385]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored data:
> RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> [INFO 2014-05-12 15:17:47.386]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the
> partition ownership to be deleted: 1-5]
>
>
>
> And I opened zk viewer.
>
> In zk, we found 2 consumers in ConsumerGroup/ids:
>
> RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
>
>
> And in owners/topic/ we found all partitions are assigned to
> sd-sns-relation03.bj:
>
> Here is the owner info:
> 1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> 1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> 1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> 1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> 1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> 1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> 1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> 1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> 1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> 1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
>
> 2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> 2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> 2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> 2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> 2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> 2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> 2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> 2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> 2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> 2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
>
>
> So all partitions are assigned to sd-sns-relation03.bj, but from logs and
> counter, we are sure sd-sns-relation02.bj has input too.
>
>
> My question is:
>
> 1. why rebalance failed?
> 2. why owner info is wrong? btw: zkclient is 0.2
>
--
-- Guozhang