You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by ankit tyagi <an...@gmail.com> on 2014/04/18 12:58:43 UTC
too many rebalances in my consumer log and fetcher threads are
getting stopped
Hi,
I am seeing consumer re-balances very frequently and getting socket
reconnect exception. log is given below for more insights
[2014-04-18
16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
Disconnecting from kafka_leader_coms07.snapdeal.com:9092
[2014-04-18
16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
[ConsumerFetcherManager-1397812122507] *Stopping leader finder thread*
[2014-04-18
16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
[kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
Shutting down
[2014-04-18
16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
[kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
Shutdown completed
[2014-04-18
16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
[ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
[2014-04-18
16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
[ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
Shutting down
[2014-04-18
16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
Reconnect due to socket error: *
*java.nio.channels.ClosedByInterruptException*
* at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
* at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
* at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
* at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
at kafka.utils.Utils$.read(Utils.scala:394)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-04-18
16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
[ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
Stopped
[2014-04-18
16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
[ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
Shutdown completed*
[2014-04-18
16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
[ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
Shutting down
[2014-04-18
16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
[kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
Stopped
[*2014-04-18
16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
Reconnect due to socket error: *
*java.nio.channels.ClosedByInterruptException*
* at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
* at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
* at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
* at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
at kafka.utils.Utils$.read(Utils.scala:394)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
currently we are using 3 node kafka cluster(0.8.0_beta) with 1 topic of 100
partition. According to kakfa documentation, GC maybe the reason of too may
rebalances so i monitored my app with jstat, couldn't any issue over there.
[coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
S0 S1 E O P YGC YGCT FGC FGCT GCT
35.09 8.88 100.00 66.89 90.73 1119 820.300 102 1093.934 1914.234
0.00 59.06 21.02 70.93 90.89 1119 821.362 102 1093.934 1915.296
0.00 59.06 84.61 70.93 91.85 1119 821.362 102 1093.934 1915.296
35.08 59.06 100.00 79.29 92.00 1120 821.362 102 1093.934 1915.296
53.67 0.00 43.22 80.63 92.55 1120 822.511 102 1093.934 1916.445
53.67 0.00 89.50 80.63 93.55 1120 822.511 102 1093.934 1916.445
53.67 31.42 100.00 86.79 93.97 1121 822.511 102 1093.934 1916.445
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
0.00 0.00 65.02 41.18 90.02 1121 823.580 103 1104.702 1928.282
36.43 0.00 4.34 41.18 90.58 1122 823.979 103 1104.702 1928.681
36.43 0.00 69.49 41.18 91.02 1122 823.979 103 1104.702 1928.681
36.43 26.60 100.00 42.34 91.20 1123 823.979 103 1104.702 1928.681
0.00 35.07 30.98 46.02 91.41 1123 824.749 103 1104.702 1929.451
0.00 35.07 86.93 46.02 91.60 1123 824.749 103 1104.702 1929.451
1.60 0.00 14.58 51.85 91.74 1124 825.198 103 1104.702 1929.900
1.60 0.00 62.30 51.85 91.98 1124 825.198 103 1104.702 1929.900
0.00 9.65 6.49 52.08 92.19 1125 825.360 103 1104.702 1930.063
0.00 9.65 58.28 52.08 92.47 1125 825.360 103 1104.702 1930.063
19.19 9.65 100.00 53.38 92.63 1126 825.360 103 1104.702 1930.063
25.31 0.00 51.89 53.48 92.87 1126 825.784 103 1104.702 1930.487
25.31 11.21 100.00 55.24 93.04 1127 825.784 103 1104.702 1930.487
0.00 99.97 39.63 57.69 93.22 1127 826.513 103 1104.702 1931.216
0.00 99.97 86.22 57.69 93.40 1127 826.513 103 1104.702 1931.216
72.36 99.97 100.00 57.69 93.51 1128 826.513 103 1104.702 1931.216
85.99 0.00 44.82 57.69 93.76 1128 827.339 103 1104.702 1932.041
85.99 0.83 100.00 66.34 94.09 1129 827.339 103 1104.702 1932.041
0.00 36.82 12.21 73.88 94.17 1129 828.588 103 1104.702 1933.290
0.00 36.82 100.00 73.88 94.94 1130 828.588 103 1104.702 1933.290
47.42 0.00 5.23 79.99 95.02 1130 829.497 103 1104.702 1934.199
47.42 0.00 69.50 79.99 96.14 1130 829.497 103 1104.702 1934.199
47.42 19.36 100.00 80.78 97.13 1131 829.497 103 1104.702 1934.199
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
0.00 0.00 33.49 59.22 89.53 1131 830.454 104 1119.378 1949.831
0.00 0.00 87.06 59.22 90.67 1131 830.454 104 1119.378 1949.831
24.49 0.00 19.01 59.22 91.22 1132 830.770 104 1119.378 1950.147
24.49 0.00 72.05 59.22 91.46 1132 830.770 104 1119.378 1950.147
24.49 39.18 100.00 59.22 91.64 1133 830.770 104 1119.378 1950.147
0.00 99.99 44.95 59.22 91.82 1133 831.338 104 1119.378 1950.715
0.00 99.99 94.30 59.22 92.17 1133 831.338 104 1119.378 1950.715
58.40 0.00 11.63 59.22 92.22 1134 831.993 104 1119.378 1951.370
58.40 0.00 58.77 59.22 92.46 1134 831.993 104 1119.378 1951.370
58.40 27.41 100.00 59.22 92.55 1135 831.993 104 1119.378 1951.370
0.00 63.03 30.83 59.22 92.71 1135 832.675 104 1119.378 1952.053
0.00 63.03 81.31 59.22 92.94 1135 832.675 104 1119.378 1952.053
55.01 63.03 100.00 59.22 92.98 1136 832.675 104 1119.378 1952.053
79.90 0.00 37.84 59.22 93.14 1136 833.582 104 1119.378 1952.960
79.90 0.00 92.75 59.22 93.24 1136 833.582 104 1119.378 1952.960
79.90 69.59 100.00 59.22 93.30 1137 833.582 104 1119.378 1952.960
0.00 100.00 54.86 59.77 93.65 1137 834.765 104 1119.378 1954.143
31.63 100.00 98.64 60.31 94.04 1138 834.765 104 1119.378 1954.143
100.00 100.00 98.64 63.88 94.04 1138 834.765 104 1119.378 1954.143
100.00 0.00 50.43 65.69 94.50 1138 836.225 104 1119.378 1955.603
100.00 14.88 100.00 66.31 95.25 1139 836.225 104 1119.378 1955.603
100.00 100.00 100.00 67.98 95.25 1139 836.225 104 1119.378 1955.603
0.00 100.00 43.37 72.29 95.82 1139 837.636 104 1119.378 1957.014
24.55 100.00 100.00 76.41 96.40 1140 837.636 104 1119.378 1957.014
93.50 100.00 100.00 80.50 96.40 1140 837.636 104 1119.378 1957.014
100.00 0.00 62.48 82.20 96.79 1140 839.089 104 1119.378 1958.467
100.00 22.78 100.00 88.93 97.05 1141 839.089 104 1119.378 1958.467
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
0.00 0.00 53.78 46.66 89.50 1141 840.580 105 1129.520 1970.101
0.27 0.00 100.00 46.66 89.75 1142 840.580 105 1129.520 1970.101
26.45 0.00 33.61 46.66 89.77 1142 840.924 105 1129.520 1970.444
26.45 0.00 84.87 46.66 89.81 1142 840.924 105 1129.520 1970.444
0.00 2.68 16.33 50.87 89.83 1143 841.276 105 1129.520 1970.796
0.00 2.68 63.77 50.87 89.87 1143 841.276 105 1129.520 1970.796
Not sure what is happening over here. any leads would be helpful..
Regards,
Ankit TYagi
Re: too many rebalances in my consumer log and fetcher threads are
getting stopped
Posted by Jun Rao <ju...@gmail.com>.
Yes, you could increase zk session timeout too. The impact is that a hard
kill of the consumer could take up to that timeout to be detected.
Thanks,
Jun
On Sun, Apr 20, 2014 at 9:50 AM, ankit tyagi <an...@gmail.com>wrote:
> Can i reduce the consumer balances if i increase the zookeeper session
> timeout?
>
>
>
>
> On Sat, Apr 19, 2014 at 9:11 PM, ankit tyagi <ankittyagi.mnnit@gmail.com
> >wrote:
>
> > Hi Jun,
> >
> > i grabbed for 'begin rebalancing consumer' in my log and got around
> *5000*times in today's logs.
> >
> > Can there be any other reason instead full GC to initiate re-balances.
> > Are there any broker properties which can be tune to reduce rebalances?
> >
> >
> > On Sat, Apr 19, 2014 at 9:31 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> >> Do you see any rebalances? The fetcher was stopped because it was shut
> >> down, which typically happens during rebalances.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Fri, Apr 18, 2014 at 11:08 AM, ankit tyagi <
> ankittyagi.mnnit@gmail.com
> >> >wrote:
> >>
> >> > I have checked that. There was no full gc at that time. I have
> attached
> >> > jstat ouput too in my mail.
> >> >
> >> > I have concerned over why consumer fetcher thread are getting stopped.
> >> > On 18 Apr 2014 22:51, "Jun Rao" <ju...@gmail.com> wrote:
> >> >
> >> > > Have you looked at
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> >> > > ?
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > > On Fri, Apr 18, 2014 at 3:58 AM, ankit tyagi <
> >> ankittyagi.mnnit@gmail.com
> >> > > >wrote:
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > I am seeing consumer re-balances very frequently and getting
> socket
> >> > > > reconnect exception. log is given below for more insights
> >> > > >
> >> > > >
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
> >> > > > Disconnecting from kafka_leader_coms07.snapdeal.com:9092
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> >> > > > [ConsumerFetcherManager-1397812122507] *Stopping leader finder
> >> thread*
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> >> > > > Shutting down
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> >> > > > Shutdown completed
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> >> > > > [ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> >> > > > Shutting down
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
> >> > > > Reconnect due to socket error: *
> >> > > > *java.nio.channels.ClosedByInterruptException*
> >> > > > * at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> >> > > > * at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> >> > > > * at
> >> > > >
> >> >
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> >> > > > * at
> >> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> >> > > > at
> >> > > >
> >> > >
> >> >
> >>
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> >> > > > at kafka.utils.Utils$.read(Utils.scala:394)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> >> > > > at
> kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> >> > > > at
> kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> >> > > > at
> >> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> >> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> >> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> >> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> >> > > > at
> >> > > >
> >> >
> >>
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> >> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> >> > > > Stopped
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> >> > > > Shutdown completed*
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
> >> > > > Shutting down
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> >> > > > Stopped
> >> > > > [*2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
> >> > > > Reconnect due to socket error: *
> >> > > > *java.nio.channels.ClosedByInterruptException*
> >> > > > * at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> >> > > > * at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> >> > > > * at
> >> > > >
> >> >
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> >> > > > * at
> >> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> >> > > > at
> >> > > >
> >> > >
> >> >
> >>
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> >> > > > at kafka.utils.Utils$.read(Utils.scala:394)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> >> > > > at
> kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> >> > > > at
> kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> >> > > > at
> >> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> >> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> >> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> >> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> >> > > > at
> >> > > >
> >> >
> >>
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> >> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >> > > >
> >> > > >
> >> > > > currently we are using 3 node kafka cluster(0.8.0_beta) with 1
> >> topic of
> >> > > 100
> >> > > > partition. According to kakfa documentation, GC maybe the reason
> of
> >> too
> >> > > may
> >> > > > rebalances so i monitored my app with jstat, couldn't any issue
> over
> >> > > there.
> >> > > >
> >> > > > [coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
> >> > > > S0 S1 E O P YGC YGCT FGC FGCT
> >> GCT
> >> > > > 35.09 8.88 100.00 66.89 90.73 1119 820.300 102 1093.934
> >> > > 1914.234
> >> > > > 0.00 59.06 21.02 70.93 90.89 1119 821.362 102 1093.934
> >> > > 1915.296
> >> > > > 0.00 59.06 84.61 70.93 91.85 1119 821.362 102 1093.934
> >> > > 1915.296
> >> > > > 35.08 59.06 100.00 79.29 92.00 1120 821.362 102 1093.934
> >> > > 1915.296
> >> > > > 53.67 0.00 43.22 80.63 92.55 1120 822.511 102 1093.934
> >> > > 1916.445
> >> > > > 53.67 0.00 89.50 80.63 93.55 1120 822.511 102 1093.934
> >> > > 1916.445
> >> > > > 53.67 31.42 100.00 86.79 93.97 1121 822.511 102 1093.934
> >> > > 1916.445
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> >> > > 1917.514
> >> > > > 0.00 0.00 65.02 41.18 90.02 1121 823.580 103 1104.702
> >> > > 1928.282
> >> > > > 36.43 0.00 4.34 41.18 90.58 1122 823.979 103 1104.702
> >> > > 1928.681
> >> > > > 36.43 0.00 69.49 41.18 91.02 1122 823.979 103 1104.702
> >> > > 1928.681
> >> > > > 36.43 26.60 100.00 42.34 91.20 1123 823.979 103 1104.702
> >> > > 1928.681
> >> > > > 0.00 35.07 30.98 46.02 91.41 1123 824.749 103 1104.702
> >> > > 1929.451
> >> > > > 0.00 35.07 86.93 46.02 91.60 1123 824.749 103 1104.702
> >> > > 1929.451
> >> > > > 1.60 0.00 14.58 51.85 91.74 1124 825.198 103 1104.702
> >> > > 1929.900
> >> > > > 1.60 0.00 62.30 51.85 91.98 1124 825.198 103 1104.702
> >> > > 1929.900
> >> > > > 0.00 9.65 6.49 52.08 92.19 1125 825.360 103 1104.702
> >> > > 1930.063
> >> > > > 0.00 9.65 58.28 52.08 92.47 1125 825.360 103 1104.702
> >> > > 1930.063
> >> > > > 19.19 9.65 100.00 53.38 92.63 1126 825.360 103 1104.702
> >> > > 1930.063
> >> > > > 25.31 0.00 51.89 53.48 92.87 1126 825.784 103 1104.702
> >> > > 1930.487
> >> > > > 25.31 11.21 100.00 55.24 93.04 1127 825.784 103 1104.702
> >> > > 1930.487
> >> > > > 0.00 99.97 39.63 57.69 93.22 1127 826.513 103 1104.702
> >> > > 1931.216
> >> > > > 0.00 99.97 86.22 57.69 93.40 1127 826.513 103 1104.702
> >> > > 1931.216
> >> > > > 72.36 99.97 100.00 57.69 93.51 1128 826.513 103 1104.702
> >> > > 1931.216
> >> > > > 85.99 0.00 44.82 57.69 93.76 1128 827.339 103 1104.702
> >> > > 1932.041
> >> > > > 85.99 0.83 100.00 66.34 94.09 1129 827.339 103 1104.702
> >> > > 1932.041
> >> > > > 0.00 36.82 12.21 73.88 94.17 1129 828.588 103 1104.702
> >> > > 1933.290
> >> > > > 0.00 36.82 100.00 73.88 94.94 1130 828.588 103 1104.702
> >> > > 1933.290
> >> > > > 47.42 0.00 5.23 79.99 95.02 1130 829.497 103 1104.702
> >> > > 1934.199
> >> > > > 47.42 0.00 69.50 79.99 96.14 1130 829.497 103 1104.702
> >> > > 1934.199
> >> > > > 47.42 19.36 100.00 80.78 97.13 1131 829.497 103 1104.702
> >> > > 1934.199
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> >> > > 1935.156
> >> > > > 0.00 0.00 33.49 59.22 89.53 1131 830.454 104 1119.378
> >> > > 1949.831
> >> > > > 0.00 0.00 87.06 59.22 90.67 1131 830.454 104 1119.378
> >> > > 1949.831
> >> > > > 24.49 0.00 19.01 59.22 91.22 1132 830.770 104 1119.378
> >> > > 1950.147
> >> > > > 24.49 0.00 72.05 59.22 91.46 1132 830.770 104 1119.378
> >> > > 1950.147
> >> > > > 24.49 39.18 100.00 59.22 91.64 1133 830.770 104 1119.378
> >> > > 1950.147
> >> > > > 0.00 99.99 44.95 59.22 91.82 1133 831.338 104 1119.378
> >> > > 1950.715
> >> > > > 0.00 99.99 94.30 59.22 92.17 1133 831.338 104 1119.378
> >> > > 1950.715
> >> > > > 58.40 0.00 11.63 59.22 92.22 1134 831.993 104 1119.378
> >> > > 1951.370
> >> > > > 58.40 0.00 58.77 59.22 92.46 1134 831.993 104 1119.378
> >> > > 1951.370
> >> > > > 58.40 27.41 100.00 59.22 92.55 1135 831.993 104 1119.378
> >> > > 1951.370
> >> > > > 0.00 63.03 30.83 59.22 92.71 1135 832.675 104 1119.378
> >> > > 1952.053
> >> > > > 0.00 63.03 81.31 59.22 92.94 1135 832.675 104 1119.378
> >> > > 1952.053
> >> > > > 55.01 63.03 100.00 59.22 92.98 1136 832.675 104 1119.378
> >> > > 1952.053
> >> > > > 79.90 0.00 37.84 59.22 93.14 1136 833.582 104 1119.378
> >> > > 1952.960
> >> > > > 79.90 0.00 92.75 59.22 93.24 1136 833.582 104 1119.378
> >> > > 1952.960
> >> > > > 79.90 69.59 100.00 59.22 93.30 1137 833.582 104 1119.378
> >> > > 1952.960
> >> > > > 0.00 100.00 54.86 59.77 93.65 1137 834.765 104 1119.378
> >> > > 1954.143
> >> > > > 31.63 100.00 98.64 60.31 94.04 1138 834.765 104 1119.378
> >> > > 1954.143
> >> > > > 100.00 100.00 98.64 63.88 94.04 1138 834.765 104 1119.378
> >> > > 1954.143
> >> > > > 100.00 0.00 50.43 65.69 94.50 1138 836.225 104 1119.378
> >> > > 1955.603
> >> > > > 100.00 14.88 100.00 66.31 95.25 1139 836.225 104 1119.378
> >> > > 1955.603
> >> > > > 100.00 100.00 100.00 67.98 95.25 1139 836.225 104 1119.378
> >> > > 1955.603
> >> > > > 0.00 100.00 43.37 72.29 95.82 1139 837.636 104 1119.378
> >> > > 1957.014
> >> > > > 24.55 100.00 100.00 76.41 96.40 1140 837.636 104 1119.378
> >> > > 1957.014
> >> > > > 93.50 100.00 100.00 80.50 96.40 1140 837.636 104 1119.378
> >> > > 1957.014
> >> > > > 100.00 0.00 62.48 82.20 96.79 1140 839.089 104 1119.378
> >> > > 1958.467
> >> > > > 100.00 22.78 100.00 88.93 97.05 1141 839.089 104 1119.378
> >> > > 1958.467
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> >> > > 1959.958
> >> > > > 0.00 0.00 53.78 46.66 89.50 1141 840.580 105 1129.520
> >> > > 1970.101
> >> > > > 0.27 0.00 100.00 46.66 89.75 1142 840.580 105 1129.520
> >> > > 1970.101
> >> > > > 26.45 0.00 33.61 46.66 89.77 1142 840.924 105 1129.520
> >> > > 1970.444
> >> > > > 26.45 0.00 84.87 46.66 89.81 1142 840.924 105 1129.520
> >> > > 1970.444
> >> > > > 0.00 2.68 16.33 50.87 89.83 1143 841.276 105 1129.520
> >> > > 1970.796
> >> > > > 0.00 2.68 63.77 50.87 89.87 1143 841.276 105 1129.520
> >> > > 1970.796
> >> > > >
> >> > > >
> >> > > > Not sure what is happening over here. any leads would be helpful..
> >> > > >
> >> > > > Regards,
> >> > > > Ankit TYagi
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: too many rebalances in my consumer log and fetcher threads are
getting stopped
Posted by ankit tyagi <an...@gmail.com>.
Can i reduce the consumer balances if i increase the zookeeper session
timeout?
On Sat, Apr 19, 2014 at 9:11 PM, ankit tyagi <an...@gmail.com>wrote:
> Hi Jun,
>
> i grabbed for 'begin rebalancing consumer' in my log and got around *5000*times in today's logs.
>
> Can there be any other reason instead full GC to initiate re-balances.
> Are there any broker properties which can be tune to reduce rebalances?
>
>
> On Sat, Apr 19, 2014 at 9:31 AM, Jun Rao <ju...@gmail.com> wrote:
>
>> Do you see any rebalances? The fetcher was stopped because it was shut
>> down, which typically happens during rebalances.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Fri, Apr 18, 2014 at 11:08 AM, ankit tyagi <ankittyagi.mnnit@gmail.com
>> >wrote:
>>
>> > I have checked that. There was no full gc at that time. I have attached
>> > jstat ouput too in my mail.
>> >
>> > I have concerned over why consumer fetcher thread are getting stopped.
>> > On 18 Apr 2014 22:51, "Jun Rao" <ju...@gmail.com> wrote:
>> >
>> > > Have you looked at
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
>> > > ?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Fri, Apr 18, 2014 at 3:58 AM, ankit tyagi <
>> ankittyagi.mnnit@gmail.com
>> > > >wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > I am seeing consumer re-balances very frequently and getting socket
>> > > > reconnect exception. log is given below for more insights
>> > > >
>> > > >
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
>> > > > Disconnecting from kafka_leader_coms07.snapdeal.com:9092
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
>> > > > [ConsumerFetcherManager-1397812122507] *Stopping leader finder
>> thread*
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
>> > > >
>> > > >
>> > >
>> >
>> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
>> > > > Shutting down
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
>> > > >
>> > > >
>> > >
>> >
>> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
>> > > > Shutdown completed
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
>> > > > [ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
>> > > >
>> > > >
>> > >
>> >
>> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
>> > > > Shutting down
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
>> > > > Reconnect due to socket error: *
>> > > > *java.nio.channels.ClosedByInterruptException*
>> > > > * at
>> > > >
>> > > >
>> > >
>> >
>> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
>> > > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
>> > > > * at
>> > > >
>> > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
>> > > > * at
>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
>> > > > at
>> > > >
>> > >
>> >
>> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>> > > > at kafka.utils.Utils$.read(Utils.scala:394)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
>> > > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>> > > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
>> > > > at
>> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
>> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
>> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
>> > > > at
>> > > >
>> >
>> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
>> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
>> > > >
>> > > >
>> > >
>> >
>> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
>> > > > Stopped
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
>> > > >
>> > > >
>> > >
>> >
>> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
>> > > > Shutdown completed*
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
>> > > >
>> > > >
>> > >
>> >
>> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
>> > > > Shutting down
>> > > > [2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
>> > > >
>> > > >
>> > >
>> >
>> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
>> > > > Stopped
>> > > > [*2014-04-18
>> > > >
>> > > >
>> > >
>> >
>> 16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
>> > > > Reconnect due to socket error: *
>> > > > *java.nio.channels.ClosedByInterruptException*
>> > > > * at
>> > > >
>> > > >
>> > >
>> >
>> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
>> > > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
>> > > > * at
>> > > >
>> > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
>> > > > * at
>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
>> > > > at
>> > > >
>> > >
>> >
>> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>> > > > at kafka.utils.Utils$.read(Utils.scala:394)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
>> > > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>> > > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
>> > > > at
>> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
>> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
>> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
>> > > > at
>> > > >
>> >
>> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
>> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>> > > >
>> > > >
>> > > > currently we are using 3 node kafka cluster(0.8.0_beta) with 1
>> topic of
>> > > 100
>> > > > partition. According to kakfa documentation, GC maybe the reason of
>> too
>> > > may
>> > > > rebalances so i monitored my app with jstat, couldn't any issue over
>> > > there.
>> > > >
>> > > > [coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
>> > > > S0 S1 E O P YGC YGCT FGC FGCT
>> GCT
>> > > > 35.09 8.88 100.00 66.89 90.73 1119 820.300 102 1093.934
>> > > 1914.234
>> > > > 0.00 59.06 21.02 70.93 90.89 1119 821.362 102 1093.934
>> > > 1915.296
>> > > > 0.00 59.06 84.61 70.93 91.85 1119 821.362 102 1093.934
>> > > 1915.296
>> > > > 35.08 59.06 100.00 79.29 92.00 1120 821.362 102 1093.934
>> > > 1915.296
>> > > > 53.67 0.00 43.22 80.63 92.55 1120 822.511 102 1093.934
>> > > 1916.445
>> > > > 53.67 0.00 89.50 80.63 93.55 1120 822.511 102 1093.934
>> > > 1916.445
>> > > > 53.67 31.42 100.00 86.79 93.97 1121 822.511 102 1093.934
>> > > 1916.445
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
>> > > 1917.514
>> > > > 0.00 0.00 65.02 41.18 90.02 1121 823.580 103 1104.702
>> > > 1928.282
>> > > > 36.43 0.00 4.34 41.18 90.58 1122 823.979 103 1104.702
>> > > 1928.681
>> > > > 36.43 0.00 69.49 41.18 91.02 1122 823.979 103 1104.702
>> > > 1928.681
>> > > > 36.43 26.60 100.00 42.34 91.20 1123 823.979 103 1104.702
>> > > 1928.681
>> > > > 0.00 35.07 30.98 46.02 91.41 1123 824.749 103 1104.702
>> > > 1929.451
>> > > > 0.00 35.07 86.93 46.02 91.60 1123 824.749 103 1104.702
>> > > 1929.451
>> > > > 1.60 0.00 14.58 51.85 91.74 1124 825.198 103 1104.702
>> > > 1929.900
>> > > > 1.60 0.00 62.30 51.85 91.98 1124 825.198 103 1104.702
>> > > 1929.900
>> > > > 0.00 9.65 6.49 52.08 92.19 1125 825.360 103 1104.702
>> > > 1930.063
>> > > > 0.00 9.65 58.28 52.08 92.47 1125 825.360 103 1104.702
>> > > 1930.063
>> > > > 19.19 9.65 100.00 53.38 92.63 1126 825.360 103 1104.702
>> > > 1930.063
>> > > > 25.31 0.00 51.89 53.48 92.87 1126 825.784 103 1104.702
>> > > 1930.487
>> > > > 25.31 11.21 100.00 55.24 93.04 1127 825.784 103 1104.702
>> > > 1930.487
>> > > > 0.00 99.97 39.63 57.69 93.22 1127 826.513 103 1104.702
>> > > 1931.216
>> > > > 0.00 99.97 86.22 57.69 93.40 1127 826.513 103 1104.702
>> > > 1931.216
>> > > > 72.36 99.97 100.00 57.69 93.51 1128 826.513 103 1104.702
>> > > 1931.216
>> > > > 85.99 0.00 44.82 57.69 93.76 1128 827.339 103 1104.702
>> > > 1932.041
>> > > > 85.99 0.83 100.00 66.34 94.09 1129 827.339 103 1104.702
>> > > 1932.041
>> > > > 0.00 36.82 12.21 73.88 94.17 1129 828.588 103 1104.702
>> > > 1933.290
>> > > > 0.00 36.82 100.00 73.88 94.94 1130 828.588 103 1104.702
>> > > 1933.290
>> > > > 47.42 0.00 5.23 79.99 95.02 1130 829.497 103 1104.702
>> > > 1934.199
>> > > > 47.42 0.00 69.50 79.99 96.14 1130 829.497 103 1104.702
>> > > 1934.199
>> > > > 47.42 19.36 100.00 80.78 97.13 1131 829.497 103 1104.702
>> > > 1934.199
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
>> > > 1935.156
>> > > > 0.00 0.00 33.49 59.22 89.53 1131 830.454 104 1119.378
>> > > 1949.831
>> > > > 0.00 0.00 87.06 59.22 90.67 1131 830.454 104 1119.378
>> > > 1949.831
>> > > > 24.49 0.00 19.01 59.22 91.22 1132 830.770 104 1119.378
>> > > 1950.147
>> > > > 24.49 0.00 72.05 59.22 91.46 1132 830.770 104 1119.378
>> > > 1950.147
>> > > > 24.49 39.18 100.00 59.22 91.64 1133 830.770 104 1119.378
>> > > 1950.147
>> > > > 0.00 99.99 44.95 59.22 91.82 1133 831.338 104 1119.378
>> > > 1950.715
>> > > > 0.00 99.99 94.30 59.22 92.17 1133 831.338 104 1119.378
>> > > 1950.715
>> > > > 58.40 0.00 11.63 59.22 92.22 1134 831.993 104 1119.378
>> > > 1951.370
>> > > > 58.40 0.00 58.77 59.22 92.46 1134 831.993 104 1119.378
>> > > 1951.370
>> > > > 58.40 27.41 100.00 59.22 92.55 1135 831.993 104 1119.378
>> > > 1951.370
>> > > > 0.00 63.03 30.83 59.22 92.71 1135 832.675 104 1119.378
>> > > 1952.053
>> > > > 0.00 63.03 81.31 59.22 92.94 1135 832.675 104 1119.378
>> > > 1952.053
>> > > > 55.01 63.03 100.00 59.22 92.98 1136 832.675 104 1119.378
>> > > 1952.053
>> > > > 79.90 0.00 37.84 59.22 93.14 1136 833.582 104 1119.378
>> > > 1952.960
>> > > > 79.90 0.00 92.75 59.22 93.24 1136 833.582 104 1119.378
>> > > 1952.960
>> > > > 79.90 69.59 100.00 59.22 93.30 1137 833.582 104 1119.378
>> > > 1952.960
>> > > > 0.00 100.00 54.86 59.77 93.65 1137 834.765 104 1119.378
>> > > 1954.143
>> > > > 31.63 100.00 98.64 60.31 94.04 1138 834.765 104 1119.378
>> > > 1954.143
>> > > > 100.00 100.00 98.64 63.88 94.04 1138 834.765 104 1119.378
>> > > 1954.143
>> > > > 100.00 0.00 50.43 65.69 94.50 1138 836.225 104 1119.378
>> > > 1955.603
>> > > > 100.00 14.88 100.00 66.31 95.25 1139 836.225 104 1119.378
>> > > 1955.603
>> > > > 100.00 100.00 100.00 67.98 95.25 1139 836.225 104 1119.378
>> > > 1955.603
>> > > > 0.00 100.00 43.37 72.29 95.82 1139 837.636 104 1119.378
>> > > 1957.014
>> > > > 24.55 100.00 100.00 76.41 96.40 1140 837.636 104 1119.378
>> > > 1957.014
>> > > > 93.50 100.00 100.00 80.50 96.40 1140 837.636 104 1119.378
>> > > 1957.014
>> > > > 100.00 0.00 62.48 82.20 96.79 1140 839.089 104 1119.378
>> > > 1958.467
>> > > > 100.00 22.78 100.00 88.93 97.05 1141 839.089 104 1119.378
>> > > 1958.467
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
>> > > 1959.958
>> > > > 0.00 0.00 53.78 46.66 89.50 1141 840.580 105 1129.520
>> > > 1970.101
>> > > > 0.27 0.00 100.00 46.66 89.75 1142 840.580 105 1129.520
>> > > 1970.101
>> > > > 26.45 0.00 33.61 46.66 89.77 1142 840.924 105 1129.520
>> > > 1970.444
>> > > > 26.45 0.00 84.87 46.66 89.81 1142 840.924 105 1129.520
>> > > 1970.444
>> > > > 0.00 2.68 16.33 50.87 89.83 1143 841.276 105 1129.520
>> > > 1970.796
>> > > > 0.00 2.68 63.77 50.87 89.87 1143 841.276 105 1129.520
>> > > 1970.796
>> > > >
>> > > >
>> > > > Not sure what is happening over here. any leads would be helpful..
>> > > >
>> > > > Regards,
>> > > > Ankit TYagi
>> > > >
>> > >
>> >
>>
>
>
Re: too many rebalances in my consumer log and fetcher threads are
getting stopped
Posted by ankit tyagi <an...@gmail.com>.
Hi Jun,
i grabbed for 'begin rebalancing consumer' in my log and got around
*5000*times in today's logs.
Can there be any other reason instead full GC to initiate re-balances. Are
there any broker properties which can be tune to reduce rebalances?
On Sat, Apr 19, 2014 at 9:31 AM, Jun Rao <ju...@gmail.com> wrote:
> Do you see any rebalances? The fetcher was stopped because it was shut
> down, which typically happens during rebalances.
>
> Thanks,
>
> Jun
>
>
> On Fri, Apr 18, 2014 at 11:08 AM, ankit tyagi <ankittyagi.mnnit@gmail.com
> >wrote:
>
> > I have checked that. There was no full gc at that time. I have attached
> > jstat ouput too in my mail.
> >
> > I have concerned over why consumer fetcher thread are getting stopped.
> > On 18 Apr 2014 22:51, "Jun Rao" <ju...@gmail.com> wrote:
> >
> > > Have you looked at
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> > > ?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, Apr 18, 2014 at 3:58 AM, ankit tyagi <
> ankittyagi.mnnit@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > I am seeing consumer re-balances very frequently and getting socket
> > > > reconnect exception. log is given below for more insights
> > > >
> > > >
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
> > > > Disconnecting from kafka_leader_coms07.snapdeal.com:9092
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > > > [ConsumerFetcherManager-1397812122507] *Stopping leader finder
> thread*
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > > >
> > > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > > Shutting down
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > > >
> > > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > > Shutdown completed
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > > > [ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > > Shutting down
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
> > > > Reconnect due to socket error: *
> > > > *java.nio.channels.ClosedByInterruptException*
> > > > * at
> > > >
> > > >
> > >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > > > * at
> > > >
> > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > > > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > > > at
> > > >
> > >
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > > > at kafka.utils.Utils$.read(Utils.scala:394)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > > > at
> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > > > at
> > > >
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > > Stopped
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > > Shutdown completed*
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
> > > > Shutting down
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > > >
> > > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > > Stopped
> > > > [*2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
> > > > Reconnect due to socket error: *
> > > > *java.nio.channels.ClosedByInterruptException*
> > > > * at
> > > >
> > > >
> > >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > > > * at
> > > >
> > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > > > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > > > at
> > > >
> > >
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > > > at kafka.utils.Utils$.read(Utils.scala:394)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > > > at
> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > > > at
> > > >
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > >
> > > >
> > > > currently we are using 3 node kafka cluster(0.8.0_beta) with 1 topic
> of
> > > 100
> > > > partition. According to kakfa documentation, GC maybe the reason of
> too
> > > may
> > > > rebalances so i monitored my app with jstat, couldn't any issue over
> > > there.
> > > >
> > > > [coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
> > > > S0 S1 E O P YGC YGCT FGC FGCT
> GCT
> > > > 35.09 8.88 100.00 66.89 90.73 1119 820.300 102 1093.934
> > > 1914.234
> > > > 0.00 59.06 21.02 70.93 90.89 1119 821.362 102 1093.934
> > > 1915.296
> > > > 0.00 59.06 84.61 70.93 91.85 1119 821.362 102 1093.934
> > > 1915.296
> > > > 35.08 59.06 100.00 79.29 92.00 1120 821.362 102 1093.934
> > > 1915.296
> > > > 53.67 0.00 43.22 80.63 92.55 1120 822.511 102 1093.934
> > > 1916.445
> > > > 53.67 0.00 89.50 80.63 93.55 1120 822.511 102 1093.934
> > > 1916.445
> > > > 53.67 31.42 100.00 86.79 93.97 1121 822.511 102 1093.934
> > > 1916.445
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > > 1917.514
> > > > 0.00 0.00 65.02 41.18 90.02 1121 823.580 103 1104.702
> > > 1928.282
> > > > 36.43 0.00 4.34 41.18 90.58 1122 823.979 103 1104.702
> > > 1928.681
> > > > 36.43 0.00 69.49 41.18 91.02 1122 823.979 103 1104.702
> > > 1928.681
> > > > 36.43 26.60 100.00 42.34 91.20 1123 823.979 103 1104.702
> > > 1928.681
> > > > 0.00 35.07 30.98 46.02 91.41 1123 824.749 103 1104.702
> > > 1929.451
> > > > 0.00 35.07 86.93 46.02 91.60 1123 824.749 103 1104.702
> > > 1929.451
> > > > 1.60 0.00 14.58 51.85 91.74 1124 825.198 103 1104.702
> > > 1929.900
> > > > 1.60 0.00 62.30 51.85 91.98 1124 825.198 103 1104.702
> > > 1929.900
> > > > 0.00 9.65 6.49 52.08 92.19 1125 825.360 103 1104.702
> > > 1930.063
> > > > 0.00 9.65 58.28 52.08 92.47 1125 825.360 103 1104.702
> > > 1930.063
> > > > 19.19 9.65 100.00 53.38 92.63 1126 825.360 103 1104.702
> > > 1930.063
> > > > 25.31 0.00 51.89 53.48 92.87 1126 825.784 103 1104.702
> > > 1930.487
> > > > 25.31 11.21 100.00 55.24 93.04 1127 825.784 103 1104.702
> > > 1930.487
> > > > 0.00 99.97 39.63 57.69 93.22 1127 826.513 103 1104.702
> > > 1931.216
> > > > 0.00 99.97 86.22 57.69 93.40 1127 826.513 103 1104.702
> > > 1931.216
> > > > 72.36 99.97 100.00 57.69 93.51 1128 826.513 103 1104.702
> > > 1931.216
> > > > 85.99 0.00 44.82 57.69 93.76 1128 827.339 103 1104.702
> > > 1932.041
> > > > 85.99 0.83 100.00 66.34 94.09 1129 827.339 103 1104.702
> > > 1932.041
> > > > 0.00 36.82 12.21 73.88 94.17 1129 828.588 103 1104.702
> > > 1933.290
> > > > 0.00 36.82 100.00 73.88 94.94 1130 828.588 103 1104.702
> > > 1933.290
> > > > 47.42 0.00 5.23 79.99 95.02 1130 829.497 103 1104.702
> > > 1934.199
> > > > 47.42 0.00 69.50 79.99 96.14 1130 829.497 103 1104.702
> > > 1934.199
> > > > 47.42 19.36 100.00 80.78 97.13 1131 829.497 103 1104.702
> > > 1934.199
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > > 1935.156
> > > > 0.00 0.00 33.49 59.22 89.53 1131 830.454 104 1119.378
> > > 1949.831
> > > > 0.00 0.00 87.06 59.22 90.67 1131 830.454 104 1119.378
> > > 1949.831
> > > > 24.49 0.00 19.01 59.22 91.22 1132 830.770 104 1119.378
> > > 1950.147
> > > > 24.49 0.00 72.05 59.22 91.46 1132 830.770 104 1119.378
> > > 1950.147
> > > > 24.49 39.18 100.00 59.22 91.64 1133 830.770 104 1119.378
> > > 1950.147
> > > > 0.00 99.99 44.95 59.22 91.82 1133 831.338 104 1119.378
> > > 1950.715
> > > > 0.00 99.99 94.30 59.22 92.17 1133 831.338 104 1119.378
> > > 1950.715
> > > > 58.40 0.00 11.63 59.22 92.22 1134 831.993 104 1119.378
> > > 1951.370
> > > > 58.40 0.00 58.77 59.22 92.46 1134 831.993 104 1119.378
> > > 1951.370
> > > > 58.40 27.41 100.00 59.22 92.55 1135 831.993 104 1119.378
> > > 1951.370
> > > > 0.00 63.03 30.83 59.22 92.71 1135 832.675 104 1119.378
> > > 1952.053
> > > > 0.00 63.03 81.31 59.22 92.94 1135 832.675 104 1119.378
> > > 1952.053
> > > > 55.01 63.03 100.00 59.22 92.98 1136 832.675 104 1119.378
> > > 1952.053
> > > > 79.90 0.00 37.84 59.22 93.14 1136 833.582 104 1119.378
> > > 1952.960
> > > > 79.90 0.00 92.75 59.22 93.24 1136 833.582 104 1119.378
> > > 1952.960
> > > > 79.90 69.59 100.00 59.22 93.30 1137 833.582 104 1119.378
> > > 1952.960
> > > > 0.00 100.00 54.86 59.77 93.65 1137 834.765 104 1119.378
> > > 1954.143
> > > > 31.63 100.00 98.64 60.31 94.04 1138 834.765 104 1119.378
> > > 1954.143
> > > > 100.00 100.00 98.64 63.88 94.04 1138 834.765 104 1119.378
> > > 1954.143
> > > > 100.00 0.00 50.43 65.69 94.50 1138 836.225 104 1119.378
> > > 1955.603
> > > > 100.00 14.88 100.00 66.31 95.25 1139 836.225 104 1119.378
> > > 1955.603
> > > > 100.00 100.00 100.00 67.98 95.25 1139 836.225 104 1119.378
> > > 1955.603
> > > > 0.00 100.00 43.37 72.29 95.82 1139 837.636 104 1119.378
> > > 1957.014
> > > > 24.55 100.00 100.00 76.41 96.40 1140 837.636 104 1119.378
> > > 1957.014
> > > > 93.50 100.00 100.00 80.50 96.40 1140 837.636 104 1119.378
> > > 1957.014
> > > > 100.00 0.00 62.48 82.20 96.79 1140 839.089 104 1119.378
> > > 1958.467
> > > > 100.00 22.78 100.00 88.93 97.05 1141 839.089 104 1119.378
> > > 1958.467
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > > 1959.958
> > > > 0.00 0.00 53.78 46.66 89.50 1141 840.580 105 1129.520
> > > 1970.101
> > > > 0.27 0.00 100.00 46.66 89.75 1142 840.580 105 1129.520
> > > 1970.101
> > > > 26.45 0.00 33.61 46.66 89.77 1142 840.924 105 1129.520
> > > 1970.444
> > > > 26.45 0.00 84.87 46.66 89.81 1142 840.924 105 1129.520
> > > 1970.444
> > > > 0.00 2.68 16.33 50.87 89.83 1143 841.276 105 1129.520
> > > 1970.796
> > > > 0.00 2.68 63.77 50.87 89.87 1143 841.276 105 1129.520
> > > 1970.796
> > > >
> > > >
> > > > Not sure what is happening over here. any leads would be helpful..
> > > >
> > > > Regards,
> > > > Ankit TYagi
> > > >
> > >
> >
>
Re: too many rebalances in my consumer log and fetcher threads are
getting stopped
Posted by Jun Rao <ju...@gmail.com>.
Do you see any rebalances? The fetcher was stopped because it was shut
down, which typically happens during rebalances.
Thanks,
Jun
On Fri, Apr 18, 2014 at 11:08 AM, ankit tyagi <an...@gmail.com>wrote:
> I have checked that. There was no full gc at that time. I have attached
> jstat ouput too in my mail.
>
> I have concerned over why consumer fetcher thread are getting stopped.
> On 18 Apr 2014 22:51, "Jun Rao" <ju...@gmail.com> wrote:
>
> > Have you looked at
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> > ?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Apr 18, 2014 at 3:58 AM, ankit tyagi <ankittyagi.mnnit@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > I am seeing consumer re-balances very frequently and getting socket
> > > reconnect exception. log is given below for more insights
> > >
> > >
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
> > > Disconnecting from kafka_leader_coms07.snapdeal.com:9092
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > > [ConsumerFetcherManager-1397812122507] *Stopping leader finder thread*
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > Shutting down
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > Shutdown completed
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > > [ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > Shutting down
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
> > > Reconnect due to socket error: *
> > > *java.nio.channels.ClosedByInterruptException*
> > > * at
> > >
> > >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > > * at
> > >
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > > at
> > >
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > > at kafka.utils.Utils$.read(Utils.scala:394)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > > at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > > at
> > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > > at
> > >
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
> > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > Stopped
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > Shutdown completed*
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
> > > Shutting down
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > Stopped
> > > [*2014-04-18
> > >
> > >
> >
> 16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
> > > Reconnect due to socket error: *
> > > *java.nio.channels.ClosedByInterruptException*
> > > * at
> > >
> > >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > > * at
> > >
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > > at
> > >
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > > at kafka.utils.Utils$.read(Utils.scala:394)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > > at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > > at
> > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > > at
> > >
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > >
> > >
> > > currently we are using 3 node kafka cluster(0.8.0_beta) with 1 topic of
> > 100
> > > partition. According to kakfa documentation, GC maybe the reason of too
> > may
> > > rebalances so i monitored my app with jstat, couldn't any issue over
> > there.
> > >
> > > [coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
> > > S0 S1 E O P YGC YGCT FGC FGCT GCT
> > > 35.09 8.88 100.00 66.89 90.73 1119 820.300 102 1093.934
> > 1914.234
> > > 0.00 59.06 21.02 70.93 90.89 1119 821.362 102 1093.934
> > 1915.296
> > > 0.00 59.06 84.61 70.93 91.85 1119 821.362 102 1093.934
> > 1915.296
> > > 35.08 59.06 100.00 79.29 92.00 1120 821.362 102 1093.934
> > 1915.296
> > > 53.67 0.00 43.22 80.63 92.55 1120 822.511 102 1093.934
> > 1916.445
> > > 53.67 0.00 89.50 80.63 93.55 1120 822.511 102 1093.934
> > 1916.445
> > > 53.67 31.42 100.00 86.79 93.97 1121 822.511 102 1093.934
> > 1916.445
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> > 1917.514
> > > 0.00 0.00 65.02 41.18 90.02 1121 823.580 103 1104.702
> > 1928.282
> > > 36.43 0.00 4.34 41.18 90.58 1122 823.979 103 1104.702
> > 1928.681
> > > 36.43 0.00 69.49 41.18 91.02 1122 823.979 103 1104.702
> > 1928.681
> > > 36.43 26.60 100.00 42.34 91.20 1123 823.979 103 1104.702
> > 1928.681
> > > 0.00 35.07 30.98 46.02 91.41 1123 824.749 103 1104.702
> > 1929.451
> > > 0.00 35.07 86.93 46.02 91.60 1123 824.749 103 1104.702
> > 1929.451
> > > 1.60 0.00 14.58 51.85 91.74 1124 825.198 103 1104.702
> > 1929.900
> > > 1.60 0.00 62.30 51.85 91.98 1124 825.198 103 1104.702
> > 1929.900
> > > 0.00 9.65 6.49 52.08 92.19 1125 825.360 103 1104.702
> > 1930.063
> > > 0.00 9.65 58.28 52.08 92.47 1125 825.360 103 1104.702
> > 1930.063
> > > 19.19 9.65 100.00 53.38 92.63 1126 825.360 103 1104.702
> > 1930.063
> > > 25.31 0.00 51.89 53.48 92.87 1126 825.784 103 1104.702
> > 1930.487
> > > 25.31 11.21 100.00 55.24 93.04 1127 825.784 103 1104.702
> > 1930.487
> > > 0.00 99.97 39.63 57.69 93.22 1127 826.513 103 1104.702
> > 1931.216
> > > 0.00 99.97 86.22 57.69 93.40 1127 826.513 103 1104.702
> > 1931.216
> > > 72.36 99.97 100.00 57.69 93.51 1128 826.513 103 1104.702
> > 1931.216
> > > 85.99 0.00 44.82 57.69 93.76 1128 827.339 103 1104.702
> > 1932.041
> > > 85.99 0.83 100.00 66.34 94.09 1129 827.339 103 1104.702
> > 1932.041
> > > 0.00 36.82 12.21 73.88 94.17 1129 828.588 103 1104.702
> > 1933.290
> > > 0.00 36.82 100.00 73.88 94.94 1130 828.588 103 1104.702
> > 1933.290
> > > 47.42 0.00 5.23 79.99 95.02 1130 829.497 103 1104.702
> > 1934.199
> > > 47.42 0.00 69.50 79.99 96.14 1130 829.497 103 1104.702
> > 1934.199
> > > 47.42 19.36 100.00 80.78 97.13 1131 829.497 103 1104.702
> > 1934.199
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> > 1935.156
> > > 0.00 0.00 33.49 59.22 89.53 1131 830.454 104 1119.378
> > 1949.831
> > > 0.00 0.00 87.06 59.22 90.67 1131 830.454 104 1119.378
> > 1949.831
> > > 24.49 0.00 19.01 59.22 91.22 1132 830.770 104 1119.378
> > 1950.147
> > > 24.49 0.00 72.05 59.22 91.46 1132 830.770 104 1119.378
> > 1950.147
> > > 24.49 39.18 100.00 59.22 91.64 1133 830.770 104 1119.378
> > 1950.147
> > > 0.00 99.99 44.95 59.22 91.82 1133 831.338 104 1119.378
> > 1950.715
> > > 0.00 99.99 94.30 59.22 92.17 1133 831.338 104 1119.378
> > 1950.715
> > > 58.40 0.00 11.63 59.22 92.22 1134 831.993 104 1119.378
> > 1951.370
> > > 58.40 0.00 58.77 59.22 92.46 1134 831.993 104 1119.378
> > 1951.370
> > > 58.40 27.41 100.00 59.22 92.55 1135 831.993 104 1119.378
> > 1951.370
> > > 0.00 63.03 30.83 59.22 92.71 1135 832.675 104 1119.378
> > 1952.053
> > > 0.00 63.03 81.31 59.22 92.94 1135 832.675 104 1119.378
> > 1952.053
> > > 55.01 63.03 100.00 59.22 92.98 1136 832.675 104 1119.378
> > 1952.053
> > > 79.90 0.00 37.84 59.22 93.14 1136 833.582 104 1119.378
> > 1952.960
> > > 79.90 0.00 92.75 59.22 93.24 1136 833.582 104 1119.378
> > 1952.960
> > > 79.90 69.59 100.00 59.22 93.30 1137 833.582 104 1119.378
> > 1952.960
> > > 0.00 100.00 54.86 59.77 93.65 1137 834.765 104 1119.378
> > 1954.143
> > > 31.63 100.00 98.64 60.31 94.04 1138 834.765 104 1119.378
> > 1954.143
> > > 100.00 100.00 98.64 63.88 94.04 1138 834.765 104 1119.378
> > 1954.143
> > > 100.00 0.00 50.43 65.69 94.50 1138 836.225 104 1119.378
> > 1955.603
> > > 100.00 14.88 100.00 66.31 95.25 1139 836.225 104 1119.378
> > 1955.603
> > > 100.00 100.00 100.00 67.98 95.25 1139 836.225 104 1119.378
> > 1955.603
> > > 0.00 100.00 43.37 72.29 95.82 1139 837.636 104 1119.378
> > 1957.014
> > > 24.55 100.00 100.00 76.41 96.40 1140 837.636 104 1119.378
> > 1957.014
> > > 93.50 100.00 100.00 80.50 96.40 1140 837.636 104 1119.378
> > 1957.014
> > > 100.00 0.00 62.48 82.20 96.79 1140 839.089 104 1119.378
> > 1958.467
> > > 100.00 22.78 100.00 88.93 97.05 1141 839.089 104 1119.378
> > 1958.467
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> > 1959.958
> > > 0.00 0.00 53.78 46.66 89.50 1141 840.580 105 1129.520
> > 1970.101
> > > 0.27 0.00 100.00 46.66 89.75 1142 840.580 105 1129.520
> > 1970.101
> > > 26.45 0.00 33.61 46.66 89.77 1142 840.924 105 1129.520
> > 1970.444
> > > 26.45 0.00 84.87 46.66 89.81 1142 840.924 105 1129.520
> > 1970.444
> > > 0.00 2.68 16.33 50.87 89.83 1143 841.276 105 1129.520
> > 1970.796
> > > 0.00 2.68 63.77 50.87 89.87 1143 841.276 105 1129.520
> > 1970.796
> > >
> > >
> > > Not sure what is happening over here. any leads would be helpful..
> > >
> > > Regards,
> > > Ankit TYagi
> > >
> >
>
Re: too many rebalances in my consumer log and fetcher threads are
getting stopped
Posted by ankit tyagi <an...@gmail.com>.
I have checked that. There was no full gc at that time. I have attached
jstat ouput too in my mail.
I have concerned over why consumer fetcher thread are getting stopped.
On 18 Apr 2014 22:51, "Jun Rao" <ju...@gmail.com> wrote:
> Have you looked at
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> ?
>
> Thanks,
>
> Jun
>
>
> On Fri, Apr 18, 2014 at 3:58 AM, ankit tyagi <ankittyagi.mnnit@gmail.com
> >wrote:
>
> > Hi,
> >
> > I am seeing consumer re-balances very frequently and getting socket
> > reconnect exception. log is given below for more insights
> >
> >
> > [2014-04-18
> >
> >
> 16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
> > Disconnecting from kafka_leader_coms07.snapdeal.com:9092
> > [2014-04-18
> >
> >
> 16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > [ConsumerFetcherManager-1397812122507] *Stopping leader finder thread*
> > [2014-04-18
> >
> >
> 16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > Shutting down
> > [2014-04-18
> >
> >
> 16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > Shutdown completed
> > [2014-04-18
> >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > [ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
> > [2014-04-18
> >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > Shutting down
> > [2014-04-18
> >
> >
> 16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
> > Reconnect due to socket error: *
> > *java.nio.channels.ClosedByInterruptException*
> > * at
> >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > * at
> > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > at
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > at kafka.utils.Utils$.read(Utils.scala:394)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > at
> >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > at
> >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > at
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > [2014-04-18
> >
> >
> 16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
> >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > Stopped
> > [2014-04-18
> >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > Shutdown completed*
> > [2014-04-18
> >
> >
> 16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
> > Shutting down
> > [2014-04-18
> >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > Stopped
> > [*2014-04-18
> >
> >
> 16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
> > Reconnect due to socket error: *
> > *java.nio.channels.ClosedByInterruptException*
> > * at
> >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > * at
> > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > at
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > at kafka.utils.Utils$.read(Utils.scala:394)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > at
> >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > at
> >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > at
> >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > at
> >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > at
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >
> >
> > currently we are using 3 node kafka cluster(0.8.0_beta) with 1 topic of
> 100
> > partition. According to kakfa documentation, GC maybe the reason of too
> may
> > rebalances so i monitored my app with jstat, couldn't any issue over
> there.
> >
> > [coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
> > S0 S1 E O P YGC YGCT FGC FGCT GCT
> > 35.09 8.88 100.00 66.89 90.73 1119 820.300 102 1093.934
> 1914.234
> > 0.00 59.06 21.02 70.93 90.89 1119 821.362 102 1093.934
> 1915.296
> > 0.00 59.06 84.61 70.93 91.85 1119 821.362 102 1093.934
> 1915.296
> > 35.08 59.06 100.00 79.29 92.00 1120 821.362 102 1093.934
> 1915.296
> > 53.67 0.00 43.22 80.63 92.55 1120 822.511 102 1093.934
> 1916.445
> > 53.67 0.00 89.50 80.63 93.55 1120 822.511 102 1093.934
> 1916.445
> > 53.67 31.42 100.00 86.79 93.97 1121 822.511 102 1093.934
> 1916.445
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934
> 1917.514
> > 0.00 0.00 65.02 41.18 90.02 1121 823.580 103 1104.702
> 1928.282
> > 36.43 0.00 4.34 41.18 90.58 1122 823.979 103 1104.702
> 1928.681
> > 36.43 0.00 69.49 41.18 91.02 1122 823.979 103 1104.702
> 1928.681
> > 36.43 26.60 100.00 42.34 91.20 1123 823.979 103 1104.702
> 1928.681
> > 0.00 35.07 30.98 46.02 91.41 1123 824.749 103 1104.702
> 1929.451
> > 0.00 35.07 86.93 46.02 91.60 1123 824.749 103 1104.702
> 1929.451
> > 1.60 0.00 14.58 51.85 91.74 1124 825.198 103 1104.702
> 1929.900
> > 1.60 0.00 62.30 51.85 91.98 1124 825.198 103 1104.702
> 1929.900
> > 0.00 9.65 6.49 52.08 92.19 1125 825.360 103 1104.702
> 1930.063
> > 0.00 9.65 58.28 52.08 92.47 1125 825.360 103 1104.702
> 1930.063
> > 19.19 9.65 100.00 53.38 92.63 1126 825.360 103 1104.702
> 1930.063
> > 25.31 0.00 51.89 53.48 92.87 1126 825.784 103 1104.702
> 1930.487
> > 25.31 11.21 100.00 55.24 93.04 1127 825.784 103 1104.702
> 1930.487
> > 0.00 99.97 39.63 57.69 93.22 1127 826.513 103 1104.702
> 1931.216
> > 0.00 99.97 86.22 57.69 93.40 1127 826.513 103 1104.702
> 1931.216
> > 72.36 99.97 100.00 57.69 93.51 1128 826.513 103 1104.702
> 1931.216
> > 85.99 0.00 44.82 57.69 93.76 1128 827.339 103 1104.702
> 1932.041
> > 85.99 0.83 100.00 66.34 94.09 1129 827.339 103 1104.702
> 1932.041
> > 0.00 36.82 12.21 73.88 94.17 1129 828.588 103 1104.702
> 1933.290
> > 0.00 36.82 100.00 73.88 94.94 1130 828.588 103 1104.702
> 1933.290
> > 47.42 0.00 5.23 79.99 95.02 1130 829.497 103 1104.702
> 1934.199
> > 47.42 0.00 69.50 79.99 96.14 1130 829.497 103 1104.702
> 1934.199
> > 47.42 19.36 100.00 80.78 97.13 1131 829.497 103 1104.702
> 1934.199
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702
> 1935.156
> > 0.00 0.00 33.49 59.22 89.53 1131 830.454 104 1119.378
> 1949.831
> > 0.00 0.00 87.06 59.22 90.67 1131 830.454 104 1119.378
> 1949.831
> > 24.49 0.00 19.01 59.22 91.22 1132 830.770 104 1119.378
> 1950.147
> > 24.49 0.00 72.05 59.22 91.46 1132 830.770 104 1119.378
> 1950.147
> > 24.49 39.18 100.00 59.22 91.64 1133 830.770 104 1119.378
> 1950.147
> > 0.00 99.99 44.95 59.22 91.82 1133 831.338 104 1119.378
> 1950.715
> > 0.00 99.99 94.30 59.22 92.17 1133 831.338 104 1119.378
> 1950.715
> > 58.40 0.00 11.63 59.22 92.22 1134 831.993 104 1119.378
> 1951.370
> > 58.40 0.00 58.77 59.22 92.46 1134 831.993 104 1119.378
> 1951.370
> > 58.40 27.41 100.00 59.22 92.55 1135 831.993 104 1119.378
> 1951.370
> > 0.00 63.03 30.83 59.22 92.71 1135 832.675 104 1119.378
> 1952.053
> > 0.00 63.03 81.31 59.22 92.94 1135 832.675 104 1119.378
> 1952.053
> > 55.01 63.03 100.00 59.22 92.98 1136 832.675 104 1119.378
> 1952.053
> > 79.90 0.00 37.84 59.22 93.14 1136 833.582 104 1119.378
> 1952.960
> > 79.90 0.00 92.75 59.22 93.24 1136 833.582 104 1119.378
> 1952.960
> > 79.90 69.59 100.00 59.22 93.30 1137 833.582 104 1119.378
> 1952.960
> > 0.00 100.00 54.86 59.77 93.65 1137 834.765 104 1119.378
> 1954.143
> > 31.63 100.00 98.64 60.31 94.04 1138 834.765 104 1119.378
> 1954.143
> > 100.00 100.00 98.64 63.88 94.04 1138 834.765 104 1119.378
> 1954.143
> > 100.00 0.00 50.43 65.69 94.50 1138 836.225 104 1119.378
> 1955.603
> > 100.00 14.88 100.00 66.31 95.25 1139 836.225 104 1119.378
> 1955.603
> > 100.00 100.00 100.00 67.98 95.25 1139 836.225 104 1119.378
> 1955.603
> > 0.00 100.00 43.37 72.29 95.82 1139 837.636 104 1119.378
> 1957.014
> > 24.55 100.00 100.00 76.41 96.40 1140 837.636 104 1119.378
> 1957.014
> > 93.50 100.00 100.00 80.50 96.40 1140 837.636 104 1119.378
> 1957.014
> > 100.00 0.00 62.48 82.20 96.79 1140 839.089 104 1119.378
> 1958.467
> > 100.00 22.78 100.00 88.93 97.05 1141 839.089 104 1119.378
> 1958.467
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378
> 1959.958
> > 0.00 0.00 53.78 46.66 89.50 1141 840.580 105 1129.520
> 1970.101
> > 0.27 0.00 100.00 46.66 89.75 1142 840.580 105 1129.520
> 1970.101
> > 26.45 0.00 33.61 46.66 89.77 1142 840.924 105 1129.520
> 1970.444
> > 26.45 0.00 84.87 46.66 89.81 1142 840.924 105 1129.520
> 1970.444
> > 0.00 2.68 16.33 50.87 89.83 1143 841.276 105 1129.520
> 1970.796
> > 0.00 2.68 63.77 50.87 89.87 1143 841.276 105 1129.520
> 1970.796
> >
> >
> > Not sure what is happening over here. any leads would be helpful..
> >
> > Regards,
> > Ankit TYagi
> >
>
Re: too many rebalances in my consumer log and fetcher threads are
getting stopped
Posted by Jun Rao <ju...@gmail.com>.
Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
?
Thanks,
Jun
On Fri, Apr 18, 2014 at 3:58 AM, ankit tyagi <an...@gmail.com>wrote:
> Hi,
>
> I am seeing consumer re-balances very frequently and getting socket
> reconnect exception. log is given below for more insights
>
>
> [2014-04-18
>
> 16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
> Disconnecting from kafka_leader_coms07.snapdeal.com:9092
> [2014-04-18
>
> 16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> [ConsumerFetcherManager-1397812122507] *Stopping leader finder thread*
> [2014-04-18
>
> 16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
>
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> Shutting down
> [2014-04-18
>
> 16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
>
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> Shutdown completed
> [2014-04-18
>
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> [ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
> [2014-04-18
>
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> Shutting down
> [2014-04-18
>
> 16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
> Reconnect due to socket error: *
> *java.nio.channels.ClosedByInterruptException*
> * at
>
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> * at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> at kafka.utils.Utils$.read(Utils.scala:394)
> at
>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> at
>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> at
>
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> at
>
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> [2014-04-18
>
> 16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> Stopped
> [2014-04-18
>
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> Shutdown completed*
> [2014-04-18
>
> 16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
> Shutting down
> [2014-04-18
>
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
>
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> Stopped
> [*2014-04-18
>
> 16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
> Reconnect due to socket error: *
> *java.nio.channels.ClosedByInterruptException*
> * at
>
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> * at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> at kafka.utils.Utils$.read(Utils.scala:394)
> at
>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> at
>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> at
>
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> at
>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> at
>
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>
>
> currently we are using 3 node kafka cluster(0.8.0_beta) with 1 topic of 100
> partition. According to kakfa documentation, GC maybe the reason of too may
> rebalances so i monitored my app with jstat, couldn't any issue over there.
>
> [coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
> S0 S1 E O P YGC YGCT FGC FGCT GCT
> 35.09 8.88 100.00 66.89 90.73 1119 820.300 102 1093.934 1914.234
> 0.00 59.06 21.02 70.93 90.89 1119 821.362 102 1093.934 1915.296
> 0.00 59.06 84.61 70.93 91.85 1119 821.362 102 1093.934 1915.296
> 35.08 59.06 100.00 79.29 92.00 1120 821.362 102 1093.934 1915.296
> 53.67 0.00 43.22 80.63 92.55 1120 822.511 102 1093.934 1916.445
> 53.67 0.00 89.50 80.63 93.55 1120 822.511 102 1093.934 1916.445
> 53.67 31.42 100.00 86.79 93.97 1121 822.511 102 1093.934 1916.445
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 40.26 0.00 89.55 93.97 1121 823.580 103 1093.934 1917.514
> 0.00 0.00 65.02 41.18 90.02 1121 823.580 103 1104.702 1928.282
> 36.43 0.00 4.34 41.18 90.58 1122 823.979 103 1104.702 1928.681
> 36.43 0.00 69.49 41.18 91.02 1122 823.979 103 1104.702 1928.681
> 36.43 26.60 100.00 42.34 91.20 1123 823.979 103 1104.702 1928.681
> 0.00 35.07 30.98 46.02 91.41 1123 824.749 103 1104.702 1929.451
> 0.00 35.07 86.93 46.02 91.60 1123 824.749 103 1104.702 1929.451
> 1.60 0.00 14.58 51.85 91.74 1124 825.198 103 1104.702 1929.900
> 1.60 0.00 62.30 51.85 91.98 1124 825.198 103 1104.702 1929.900
> 0.00 9.65 6.49 52.08 92.19 1125 825.360 103 1104.702 1930.063
> 0.00 9.65 58.28 52.08 92.47 1125 825.360 103 1104.702 1930.063
> 19.19 9.65 100.00 53.38 92.63 1126 825.360 103 1104.702 1930.063
> 25.31 0.00 51.89 53.48 92.87 1126 825.784 103 1104.702 1930.487
> 25.31 11.21 100.00 55.24 93.04 1127 825.784 103 1104.702 1930.487
> 0.00 99.97 39.63 57.69 93.22 1127 826.513 103 1104.702 1931.216
> 0.00 99.97 86.22 57.69 93.40 1127 826.513 103 1104.702 1931.216
> 72.36 99.97 100.00 57.69 93.51 1128 826.513 103 1104.702 1931.216
> 85.99 0.00 44.82 57.69 93.76 1128 827.339 103 1104.702 1932.041
> 85.99 0.83 100.00 66.34 94.09 1129 827.339 103 1104.702 1932.041
> 0.00 36.82 12.21 73.88 94.17 1129 828.588 103 1104.702 1933.290
> 0.00 36.82 100.00 73.88 94.94 1130 828.588 103 1104.702 1933.290
> 47.42 0.00 5.23 79.99 95.02 1130 829.497 103 1104.702 1934.199
> 47.42 0.00 69.50 79.99 96.14 1130 829.497 103 1104.702 1934.199
> 47.42 19.36 100.00 80.78 97.13 1131 829.497 103 1104.702 1934.199
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 99.99 0.00 87.63 97.13 1131 830.454 104 1104.702 1935.156
> 0.00 0.00 33.49 59.22 89.53 1131 830.454 104 1119.378 1949.831
> 0.00 0.00 87.06 59.22 90.67 1131 830.454 104 1119.378 1949.831
> 24.49 0.00 19.01 59.22 91.22 1132 830.770 104 1119.378 1950.147
> 24.49 0.00 72.05 59.22 91.46 1132 830.770 104 1119.378 1950.147
> 24.49 39.18 100.00 59.22 91.64 1133 830.770 104 1119.378 1950.147
> 0.00 99.99 44.95 59.22 91.82 1133 831.338 104 1119.378 1950.715
> 0.00 99.99 94.30 59.22 92.17 1133 831.338 104 1119.378 1950.715
> 58.40 0.00 11.63 59.22 92.22 1134 831.993 104 1119.378 1951.370
> 58.40 0.00 58.77 59.22 92.46 1134 831.993 104 1119.378 1951.370
> 58.40 27.41 100.00 59.22 92.55 1135 831.993 104 1119.378 1951.370
> 0.00 63.03 30.83 59.22 92.71 1135 832.675 104 1119.378 1952.053
> 0.00 63.03 81.31 59.22 92.94 1135 832.675 104 1119.378 1952.053
> 55.01 63.03 100.00 59.22 92.98 1136 832.675 104 1119.378 1952.053
> 79.90 0.00 37.84 59.22 93.14 1136 833.582 104 1119.378 1952.960
> 79.90 0.00 92.75 59.22 93.24 1136 833.582 104 1119.378 1952.960
> 79.90 69.59 100.00 59.22 93.30 1137 833.582 104 1119.378 1952.960
> 0.00 100.00 54.86 59.77 93.65 1137 834.765 104 1119.378 1954.143
> 31.63 100.00 98.64 60.31 94.04 1138 834.765 104 1119.378 1954.143
> 100.00 100.00 98.64 63.88 94.04 1138 834.765 104 1119.378 1954.143
> 100.00 0.00 50.43 65.69 94.50 1138 836.225 104 1119.378 1955.603
> 100.00 14.88 100.00 66.31 95.25 1139 836.225 104 1119.378 1955.603
> 100.00 100.00 100.00 67.98 95.25 1139 836.225 104 1119.378 1955.603
> 0.00 100.00 43.37 72.29 95.82 1139 837.636 104 1119.378 1957.014
> 24.55 100.00 100.00 76.41 96.40 1140 837.636 104 1119.378 1957.014
> 93.50 100.00 100.00 80.50 96.40 1140 837.636 104 1119.378 1957.014
> 100.00 0.00 62.48 82.20 96.79 1140 839.089 104 1119.378 1958.467
> 100.00 22.78 100.00 88.93 97.05 1141 839.089 104 1119.378 1958.467
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 96.13 0.00 90.89 97.05 1141 840.580 105 1119.378 1959.958
> 0.00 0.00 53.78 46.66 89.50 1141 840.580 105 1129.520 1970.101
> 0.27 0.00 100.00 46.66 89.75 1142 840.580 105 1129.520 1970.101
> 26.45 0.00 33.61 46.66 89.77 1142 840.924 105 1129.520 1970.444
> 26.45 0.00 84.87 46.66 89.81 1142 840.924 105 1129.520 1970.444
> 0.00 2.68 16.33 50.87 89.83 1143 841.276 105 1129.520 1970.796
> 0.00 2.68 63.77 50.87 89.87 1143 841.276 105 1129.520 1970.796
>
>
> Not sure what is happening over here. any leads would be helpful..
>
> Regards,
> Ankit TYagi
>