You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sameer Yami <sy...@ailive.net> on 2014/11/04 21:47:21 UTC
Re: Cannot connect to Kafka from outside of EC2
Hi Guozhang,
This is the server.log -
[2014-11-04 20:21:57,510] INFO Verifying properties
(kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,545] INFO Property advertised.host.name is overridden
to x.x.x.x (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,545] INFO Property broker.id is overridden to 0
(kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,545] INFO Property controlled.shutdown.enable is
overridden to true (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,545] INFO Property host.name is overridden to
172.31.25.198 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is overridden to
false (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,546] INFO Property log.dirs is overridden to
/tmp/kafka-logs (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,546] INFO Property log.retention.check.interval.ms is
overridden to 60000 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,546] INFO Property log.retention.hours is overridden
to 168 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,546] INFO Property log.segment.bytes is overridden to
536870912 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,546] INFO Property num.io.threads is overridden to 8
(kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,547] INFO Property num.network.threads is overridden
to 2 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,547] INFO Property num.partitions is overridden to 2
(kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,547] INFO Property port is overridden to 9092
(kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,547] INFO Property socket.receive.buffer.bytes is
overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,547] INFO Property socket.request.max.bytes is
overridden to 104857600 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,547] INFO Property socket.send.buffer.bytes is
overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,548] INFO Property zookeeper.connect is overridden to
localhost:2181 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,548] INFO Property zookeeper.connection.timeout.ms is
overridden to 1000000 (kafka.utils.VerifiableProperties)
[2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
(kafka.server.KafkaServer)
[2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to zookeeper on
localhost:2181 (kafka.server.KafkaServer)
[2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs' not found,
creating it. (kafka.log.LogManager)
[2014-11-04 20:21:57,748] INFO Starting log cleanup with a period of 60000
ms. (kafka.log.LogManager)
[2014-11-04 20:21:57,752] INFO Starting log flusher with a default period
of 9223372036854775807 ms. (kafka.log.LogManager)
[2014-11-04 20:21:57,783] INFO Awaiting socket connections on
ip-172-31-25-198.us-west-1.compute.internal:9092. (kafka.network.Acceptor)
[2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0], Started
(kafka.network.SocketServer)
[2014-11-04 20:21:57,852] INFO Will not load MX4J, mx4j-tools.jar is not in
the classpath (kafka.utils.Mx4jLoader$)
[2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
(kafka.server.ZookeeperLeaderElector)
[2014-11-04 20:21:57,967] INFO Registered broker 0 at path /brokers/ids/0
with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
[2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
(kafka.server.KafkaServer)
[2014-11-04 20:21:58,050] INFO New leader is 0
(kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2014-11-04 20:22:01,809] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:22:11,812] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:29,084] INFO Topic creation
{"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
[2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of topic Test
with 2 partitions and replication factor 1 is successful!
(kafka.server.KafkaApis)
[2014-11-04 20:30:29,280] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:29,301] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:30,310] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:30,315] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:31,320] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:31,325] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:32,331] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:32,336] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:33,343] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:33,347] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:34,356] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:34,360] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:35,366] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:35,372] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:36,377] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:36,382] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:37,387] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:37,394] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:38,399] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:38,403] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:39,410] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:39,414] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:40,419] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-04 20:30:41,602] INFO Topic creation
{"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
thanks
On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <wa...@gmail.com> wrote:
> The server.log should be the broker logs.
>
> Guozhang
>
> On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <sy...@ailive.net> wrote:
>
> > I do not see any other logs besides these.
> > I checked the log4j.properties and where the logs are getting stored.
> > The only other logs (apart from the controller logs) in the directory are
> > zookeeper-gc.log,kafkaServer-gc.log, server.log and state-change.log
> >
> > Thanks
> >
> > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Sorry for getting late on this Sameer.
> > >
> > > The logs you showed above are from controller logs, do you see brokers
> > logs
> > > besides that?
> > >
> > > Guozhang
> > >
> > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <sy...@ailive.net> wrote:
> > >
> > > > Hi Guozhang,
> > > >
> > > > Is there any clue in the above data as to what is wrong?
> > > >
> > > > thanks
> > > >
> > > >
> > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <sy...@ailive.net>
> wrote:
> > > >
> > > > > This is the one from the file specified in log4j.properties -
> > > > >
> > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on Controller
> > 0]:
> > > > > Topic change listener fired for path /brokers/topics with children
> > Test
> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener on Controller
> 0]:
> > > New
> > > > > topics: [Set(Test)], deleted topics: [Set()], new partition replica
> > > > > assignment [Map([Test,0] -> List(0), [Test,1] -> List(0))]
> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New topic creation
> > > > callback
> > > > > for [Test,0],[Test,1] (kafka.controller.KafkaController)
> > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New partition
> creation
> > > > > callback for [Test,0],[Test,1] (kafka.controller.KafkaController)
> > > > > [2014-10-28 22:38:36,095] INFO [Partition state machine on
> Controller
> > > 0]:
> > > > > Invoking state change to NewPartition for partitions
> > [Test,0],[Test,1]
> > > > > (kafka.controller.PartitionStateMachine)
> > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine on controller
> > 0]:
> > > > > Invoking state change to NewReplica for replicas
> > > > >
> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > (kafka.controller.ReplicaStateMachine)
> > > > > [2014-10-28 22:38:36,111] INFO [Partition state machine on
> Controller
> > > 0]:
> > > > > Invoking state change to OnlinePartition for partitions
> > > [Test,0],[Test,1]
> > > > > (kafka.controller.PartitionStateMachine)
> > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state machine on
> > Controller
> > > > 0]:
> > > > > Live assigned replicas for partition [Test,0] are: [List(0)]
> > > > > (kafka.controller.PartitionStateMachine)
> > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state machine on
> > Controller
> > > > 0]:
> > > > > Initializing leader and isr for partition [Test,0] to
> > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > (kafka.controller.PartitionStateMachine)
> > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state machine on
> > Controller
> > > > 0]:
> > > > > Live assigned replicas for partition [Test,1] are: [List(0)]
> > > > > (kafka.controller.PartitionStateMachine)
> > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state machine on
> > Controller
> > > > 0]:
> > > > > Initializing leader and isr for partition [Test,1] to
> > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > (kafka.controller.PartitionStateMachine)
> > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine on controller
> > 0]:
> > > > > Invoking state change to OnlineReplica for replicas
> > > > >
> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > (kafka.controller.ReplicaStateMachine)
> > > > > [2014-10-28 22:39:28,160] ERROR
> > [Controller-0-to-broker-0-send-thread],
> > > > > Controller 0's connection to broker id:0,host:x.x.x.x,port:9092 was
> > > > > unsuccessful (kafka.controller.RequestSendThread)
> > > > > java.net.ConnectException: Connection timed out
> > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > at sun.nio.ch.Net.connect(Net.java:449)
> > > > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> > > > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > at
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
> > > > > at
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
> > > > > at
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > at
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > [2014-10-28 22:39:28,460] ERROR
> > [Controller-0-to-broker-0-send-thread],
> > > > > Controller 0 epoch 1 failed to send LeaderAndIsr request with
> > > correlation
> > > > > id 7 to broker id:0,host:x.x.x.x,port:9092. Reconnecting to broker.
> > > > > (kafka.controller.RequestSendThread)
> > > > > java.nio.channels.ClosedChannelException
> > > > > at kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> > > > > at
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> > > > > at
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > at
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > >
> > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <wangguoz@gmail.com
> >
> > > > wrote:
> > > > >
> > > > >> Sameer, the one you sent me is from the producer logs, not the
> > broker
> > > > >> logs.
> > > > >> And /tmp/kafka-logs actually store the message log (i.e. the
> "data")
> > > > >> instead of the server logs. Their locations should be specified in
> > > your
> > > > >> log4j.properties file.
> > > > >>
> > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <sy...@ailive.net>
> > > wrote:
> > > > >>
> > > > >> > Is this what you are looking for -
> > > > >> >
> > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
> > > > >> > kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092
> > for
> > > > >> > producing
> > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
> > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > 172.31.16.31:9092
> > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
> > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata for
> 1
> > > > >> topic(s)
> > > > >> > Set(Test)
> > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
> > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > metadata
> > > > >> > [{TopicMetadata for topic Test ->
> > > > >> > No partition metadata for topic Test due to
> > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> class
> > > > >> > kafka.common.LeaderNotAvailableException
> > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
> > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> > info
> > > > for
> > > > >> > topic Test
> > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
> > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 9 for 1
> > > topic(s)
> > > > >> > Set(Test)
> > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
> > > > >> > kafka.network.BlockingChannel-52: Created socket with
> SO_TIMEOUT =
> > > > 10000
> > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF
> =
> > > > 102400
> > > > >> > (requested 102400).
> > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
> > > > >> > kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092
> > for
> > > > >> > producing
> > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
> > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > 172.31.16.31:9092
> > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
> > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata for
> 1
> > > > >> topic(s)
> > > > >> > Set(Test)
> > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
> > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > metadata
> > > > >> > [{TopicMetadata for topic Test ->
> > > > >> > No partition metadata for topic Test due to
> > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> class
> > > > >> > kafka.common.LeaderNotAvailableException
> > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
> > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > > > messages
> > > > >> by
> > > > >> > topic, partition due to: Failed to fetch topic metadata for
> topic:
> > > > Test
> > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
> > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000
> ms
> > > > before
> > > > >> > retrying send. Remaining retries = 6
> > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
> > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 10 for 1
> > > topic(s)
> > > > >> > Set(Test)
> > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
> > > > >> > kafka.network.BlockingChannel-52: Created socket with
> SO_TIMEOUT =
> > > > 10000
> > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF
> =
> > > > 102400
> > > > >> > (requested 102400).
> > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
> > > > >> > kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092
> > for
> > > > >> > producing
> > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
> > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > 172.31.16.31:9092
> > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata for
> 1
> > > > >> topic(s)
> > > > >> > Set(Test)
> > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
> > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > metadata
> > > > >> > [{TopicMetadata for topic Test ->
> > > > >> > No partition metadata for topic Test due to
> > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> class
> > > > >> > kafka.common.LeaderNotAvailableException
> > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> > info
> > > > for
> > > > >> > topic Test
> > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
> > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 11 for 1
> > > topic(s)
> > > > >> > Set(Test)
> > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
> > > > >> > kafka.network.BlockingChannel-52: Created socket with
> SO_TIMEOUT =
> > > > 10000
> > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF
> =
> > > > 102400
> > > > >> > (requested 102400).
> > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
> > > > >> > kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092
> > for
> > > > >> > producing
> > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
> > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > 172.31.16.31:9092
> > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
> > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata for
> 1
> > > > >> topic(s)
> > > > >> > Set(Test)
> > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
> > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > metadata
> > > > >> > [{TopicMetadata for topic Test ->
> > > > >> > No partition metadata for topic Test due to
> > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> class
> > > > >> > kafka.common.LeaderNotAvailableException
> > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
> > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > > > messages
> > > > >> by
> > > > >> > topic, partition due to: Failed to fetch topic metadata for
> topic:
> > > > Test
> > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
> > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000
> ms
> > > > before
> > > > >> > retrying send. Remaining retries = 5
> > > > >> >
> > > > >> > As for /tmp/kafka-logs, I can see only these files -
> > > > >> > recovery-point-offset-checkpoint replication-offset-checkpoint
> > > > >> >
> > > > >> > I do not see the usual Kafka log files here.
> > > > >> >
> > > > >> > thanks
> > > > >> > Sameer
> > > > >> >
> > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
> > wangguoz@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Sameer, could you check broker logs and see if it has similar
> > > > >> > > LeaderNotAvailable exceptions?
> > > > >> > >
> > > > >> > > Guozhang
> > > > >> > >
> > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
> syami@ailive.net
> > >
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Yes. I can telnet from outside using the public ip of the
> > > instance
> > > > >> and
> > > > >> > > port
> > > > >> > > > of 9092
> > > > >> > > >
> > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <
> > > > >> roger.hoover@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Just a guess but could it be a firewall issue? Did you
> > enable
> > > > >> > > > connections
> > > > >> > > > > to port 9092 from outside EC2 in a security group? Can
> you
> > > > >> telnet to
> > > > >> > > > each
> > > > >> > > > > broker IP and port?
> > > > >> > > > >
> > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami <
> > > syami@ailive.net
> > > > >
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > > There was a typo earlier.
> > > > >> > > > > >
> > > > >> > > > > > This is the output -
> > > > >> > > > > >
> > > > >> > > > > > Topic:Test PartitionCount:2 ReplicationFactor:1
> > > Configs:
> > > > >> > > > > > Topic: Test Partition: 0 Leader: 0
> > > Replicas:
> > > > 0
> > > > >> > > > Isr: 0
> > > > >> > > > > > Topic: Test Partition: 1 Leader: 0
> > > Replicas:
> > > > 0
> > > > >> > > > Isr: 0
> > > > >> > > > > >
> > > > >> > > > > > thanks
> > > > >> > > > > > Sameer
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang Wang <
> > > > >> wangguoz@gmail.com
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Sameer,
> > > > >> > > > > > >
> > > > >> > > > > > > The topic name is not consistent from your describe
> > > outputs.
> > > > >> > > Normally
> > > > >> > > > > > they
> > > > >> > > > > > > should be either
> > > > >> > > > > > >
> > > > >> > > > > > > Topic:*Test* PartitionCount:2
> ReplicationFactor:1
> > > > >> > > Configs:
> > > > >> > > > > > > Topic: *Test* Partition: 0 Leader: 0
> > > > >> Replicas: 0
> > > > >> > > > > > Isr: 0
> > > > >> > > > > > > Topic: *Test* Partition: 1 Leader: 0
> > > > >> Replicas: 0
> > > > >> > > > > > Isr: 0
> > > > >> > > > > > >
> > > > >> > > > > > > Or
> > > > >> > > > > > >
> > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
> > > > ReplicationFactor:1
> > > > >> > > > > Configs:
> > > > >> > > > > > > Topic: *Heartbeat* Partition: 0 Leader:
> 0
> > > > >> > > Replicas:
> > > > >> > > > 0
> > > > >> > > > > > > Isr: 0
> > > > >> > > > > > > Topic: *Heartbeat* Partition: 1 Leader:
> 0
> > > > >> > > Replicas:
> > > > >> > > > 0
> > > > >> > > > > > > Isr: 0
> > > > >> > > > > > >
> > > > >> > > > > > > Did you manually modify the topic name in ZK?
> > > > >> > > > > > >
> > > > >> > > > > > > Guozhang
> > > > >> > > > > > >
> > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer Yami <
> > > > >> syami@ailive.net>
> > > > >> > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi Guozhang,
> > > > >> > > > > > > >
> > > > >> > > > > > > > I was wondering if there is anything wrong in the
> > above
> > > > >> > > --describe
> > > > >> > > > > > data.
> > > > >> > > > > > > >
> > > > >> > > > > > > > thanks
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer Yami <
> > > > >> syami@ailive.net
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > This is what I get -
> > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper localhost:2181
> > > > --describe
> > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
> > > > >> ReplicationFactor:1
> > > > >> > > > > > Configs:
> > > > >> > > > > > > > > Topic: Test Partition: 0 Leader: 0
> > > > >> > Replicas: 0
> > > > >> > > > > > > Isr: 0
> > > > >> > > > > > > > > Topic: Test Partition: 1 Leader: 0
> > > > >> > Replicas: 0
> > > > >> > > > > > > Isr: 0
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > The topics are being created at run-time.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > thanks
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM, Guozhang Wang <
> > > > >> > > > wangguoz@gmail.com
> > > > >> > > > > >
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >> Could you use the topic tools "--describe" to
> check
> > > if
> > > > >> the
> > > > >> > > topic
> > > > >> > > > > > > Test's
> > > > >> > > > > > > > >> partitions leaders are available?
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >>
> > > > >> > >
> http://kafka.apache.org/documentation.html#basic_ops_add_topic
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> Guozhang
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM, Sameer Yami <
> > > > >> > > syami@ailive.net>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> > That is not the case.
> > > > >> > > > > > > > >> > This detailed error is below. Due to this
> error ,
> > > the
> > > > >> > > producer
> > > > >> > > > > is
> > > > >> > > > > > > not
> > > > >> > > > > > > > >> able
> > > > >> > > > > > > > >> > to send messages
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG
> > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-52:
> > > > >> > > > > > > > >> > Getting broker partition info for topic Test
> > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main INFO
> > > > >> > > > > kafka.client.ClientUtils$-68:
> > > > >> > > > > > > > >> Fetching
> > > > >> > > > > > > > >> > metadata from broker
> > > > id:0,host:54.215.200.61,port:9092
> > > > >> > with
> > > > >> > > > > > > > correlation
> > > > >> > > > > > > > >> id
> > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
> > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG
> > > > >> > > > > > > kafka.network.BlockingChannel-52:
> > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT = 10000
> (requested
> > > > >> 10000),
> > > > >> > > > > > SO_RCVBUF
> > > > >> > > > > > > =
> > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF = 102400
> > > (requested
> > > > >> > > 102400).
> > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main INFO
> > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092 for producing
> > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main INFO
> > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > >> > > > > > > > >> > Disconnecting from 54.215.200.61:9092
> > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG
> > > > >> > > > > kafka.client.ClientUtils$-52:
> > > > >> > > > > > > > >> > Successfully fetched metadata for 1 topic(s)
> > > > Set(Test)
> > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main WARN
> > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-83:
> > > > >> > > > > > > > >> > Error while fetching metadata [{TopicMetadata
> for
> > > > topic
> > > > >> > Test
> > > > >> > > > ->
> > > > >> > > > > > > > >> > No partition metadata for topic Test due to
> > > > >> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for
> > > topic
> > > > >> > [Test]:
> > > > >> > > > > class
> > > > >> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR
> > > > >> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97:
> > Failed
> > > > to
> > > > >> > > collate
> > > > >> > > > > > > > messages
> > > > >> > > > > > > > >> by
> > > > >> > > > > > > > >> > topic, partition due to: Failed to fetch topic
> > > > metadata
> > > > >> > for
> > > > >> > > > > topic:
> > > > >> > > > > > > > Test
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM, Guozhang Wang
> <
> > > > >> > > > > > wangguoz@gmail.com>
> > > > >> > > > > > > > >> wrote:
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > > Could you see if this is the case:
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >>
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> > > > >> > > > > > > > >> > > ?
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM, Sameer
> Yami <
> > > > >> > > > > syami@ailive.net
> > > > >> > > > > > >
> > > > >> > > > > > > > >> wrote:
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > > > Hi,
> > > > >> > > > > > > > >> > > >
> > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper test setup on a
> > > > single
> > > > >> EC2
> > > > >> > > > > > machine.
> > > > >> > > > > > > > The
> > > > >> > > > > > > > >> > > > host.name is the private ip address and
> the
> > > > >> > > > > > > advertised.host.name
> > > > >> > > > > > > > is
> > > > >> > > > > > > > >> > the
> > > > >> > > > > > > > >> > > > public address in server.properties.
> > > > >> > > > > > > > >> > > >
> > > > >> > > > > > > > >> > > > We are trying to publish from outside EC2
> > using
> > > > the
> > > > >> > > public
> > > > >> > > > > ip
> > > > >> > > > > > of
> > > > >> > > > > > > > the
> > > > >> > > > > > > > >> > EC2
> > > > >> > > > > > > > >> > > > machine but get this error on the client
> > side -
> > > > >> > > > > > > > >> > > > No partition metadata for topic test due to
> > > > >> > > > > > > > >> > > > kafka.common.LeaderNotAvailableException}]
> > for
> > > > >> topic
> > > > >> > > > [test]:
> > > > >> > > > > > > class
> > > > >> > > > > > > > >> > > > kafka.common.LeaderNotAvailableException
> > > > >> > > > > > > > >> > > >
> > > > >> > > > > > > > >> > > > The zookeeper has the public ip stored in
> > > > >> > > /brokers/ids/0.
> > > > >> > > > > > > > >> > > > We have tried multiple combinations but it
> > > works
> > > > >> ONLY
> > > > >> > > when
> > > > >> > > > > the
> > > > >> > > > > > > > >> > advertised
> > > > >> > > > > > > > >> > > > host name is not used and the client is
> > > internal
> > > > to
> > > > >> > EC2.
> > > > >> > > > > > > > >> > > >
> > > > >> > > > > > > > >> > > > Please let me know what the problem could
> be.
> > > > >> > > > > > > > >> > > >
> > > > >> > > > > > > > >> > > > thanks
> > > > >> > > > > > > > >> > > > Sameer
> > > > >> > > > > > > > >> > > >
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > > --
> > > > >> > > > > > > > >> > > -- Guozhang
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> --
> > > > >> > > > > > > > >> -- Guozhang
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > --
> > > > >> > > > > > > -- Guozhang
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > -- Guozhang
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> -- Guozhang
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Cannot connect to Kafka from outside of EC2
Posted by Sameer Yami <sy...@ailive.net>.
Is it ok, if I send you directly?
On Tue, Nov 11, 2014 at 10:17 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Sameer,
>
> I think apache mailing list has blocked your attachment. If it is too long
> to include in the email body could you paste it somewhere and give me the
> link?
>
> Guozhang
>
> On Tue, Nov 11, 2014 at 10:01 AM, Sameer Yami <sy...@ailive.net> wrote:
>
> > Hi Guozhang,
> >
> > I was wondering if you found anything wrong in the logs/
> >
> > thanks
> >
> >
> > On Fri, Nov 7, 2014 at 4:19 PM, Sameer Yami <sy...@ailive.net> wrote:
> >
> > > Hi Guozhang,
> > >
> > > Attached are the two logs with debug enabled.
> > >
> > > Thanks!
> > >
> > > On Fri, Nov 7, 2014 at 2:09 PM, Sameer Yami <sy...@ailive.net> wrote:
> > >
> > >> The version is kafka_2.10-0.8.1.1. It is not the latest trunk.
> > >> Will try enabling debug version.
> > >>
> > >> thanks
> > >>
> > >>
> > >> On Thu, Nov 6, 2014 at 9:37 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >>
> > >>> Sameer,
> > >>>
> > >>> The server logs do not contain any non-INFO logs, which is a bit
> wired.
> > >>> Did
> > >>> you deploy the current trunk of Kafka? Also could you enable DEBUG
> > level
> > >>> logging on Kafka brokers?
> > >>>
> > >>> Guozhang
> > >>>
> > >>> On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami <sy...@ailive.net>
> wrote:
> > >>>
> > >>> > The server.log was taken separately.
> > >>> > We ran the test again and the server and producer logs are below
> (to
> > >>> get
> > >>> > same timings).
> > >>> >
> > >>> >
> > >>> > Thanks!
> > >>> >
> > >>> >
> > >>> >
> > >>>
> >
> ------------------------------------------------------------------------------------
> > >>> >
> > >>> >
> > >>> >
> > >>> > Producer Logs -
> > >>> >
> > >>> >
> > >>> > 2014-11-05 23:38:58,693
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:00,695
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:02,696
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:02,828 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Verifying properties
> > >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property
> > auto.commit.interval.ms
> > >>> is
> > >>> > overridden to 1000
> > >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property auto.offset.reset is
> > >>> > overridden to smallest
> > >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property consumer.timeout.ms
> is
> > >>> > overridden to 10
> > >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property group.id is
> overridden
> > >>> to
> > >>> > TestCheck
> > >>> > 2014-11-05 23:39:02,830 pool-13-thread-2 WARN
> > >>> > kafka.utils.VerifiableProperties-83: Property serializer.class is
> not
> > >>> valid
> > >>> > 2014-11-05 23:39:02,830 pool-13-thread-2 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property zookeeper.connect is
> > >>> > overridden to 172.31.25.198:2181
> > >>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Connecting to
> > >>> > zookeeper instance at 172.31.25.198:2181
> > >>> > 2014-11-05 23:39:02,831 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkConnection-63: Creating new ZookKeeper
> instance
> > >>> to
> > >>> > connect to 172.31.25.198:2181.
> > >>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> > >>> > org.apache.zookeeper.ZooKeeper-379: Initiating client connection,
> > >>> > connectString=172.31.25.198:2181 sessionTimeout=6000
> > >>> > watcher=org.I0Itec.zkclient.ZkClient@3903b165
> > >>> > 2014-11-05 23:39:02,831 ZkClient-EventThread-29-172.31.25.198:2181
> > >>> INFO
> > >>> > org.I0Itec.zkclient.ZkEventThread-64: Starting ZkClient event
> thread.
> > >>> > 2014-11-05 23:39:02,831 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Verifying properties
> > >>> > 2014-11-05 23:39:02,836 pool-13-thread-2-SendThread() INFO
> > >>> > org.apache.zookeeper.ClientCnxn-1061: Opening socket connection to
> > >>> server /
> > >>> > 172.31.25.198:2181
> > >>> > 2014-11-05 23:39:02,836 pool-13-thread-1 WARN
> > >>> > kafka.utils.VerifiableProperties-83: Property batch.size is not
> valid
> > >>> > 2014-11-05 23:39:02,832 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-878: Awaiting connection to Zookeeper
> > >>> server
> > >>> > 2014-11-05 23:39:02,836 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property
> > message.send.max.retries
> > >>> is
> > >>> > overridden to 10
> > >>> > 2014-11-05 23:39:02,836 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-628: Waiting for keeper state
> > >>> SyncConnected
> > >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property metadata.broker.list
> is
> > >>> > overridden to 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property retry.backoff.ms is
> > >>> > overridden to 1000
> > >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> > >>> > kafka.utils.VerifiableProperties-68: Property serializer.class is
> > >>> > overridden to kafka.serializer.StringEncoder
> > >>> > 2014-11-05 23:39:02,837
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > INFO org.apache.zookeeper.ClientCnxn-950: Socket connection
> > >>> established to
> > >>> > ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
> > >>> initiating
> > >>> > session
> > >>> > 2014-11-05 23:39:02,838
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-999: Session establishment
> > >>> request
> > >>> > sent on ip-172-31-25-198.us-west-1.compute.internal/
> > 172.31.25.198:2181
> > >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 WARN
> > >>> > kafka.utils.VerifiableProperties-83: Property
> > zk.connectiontimeout.ms
> > >>> is
> > >>> > not valid
> > >>> > 2014-11-05 23:39:02,841
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > INFO org.apache.zookeeper.ClientCnxn-739: Session establishment
> > >>> complete on
> > >>> > server ip-172-31-25-198.us-west-1.compute.internal/
> > 172.31.25.198:2181,
> > >>> > sessionid = 0x1498251e8680003, negotiated timeout = 6000
> > >>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> > >>> > state:SyncConnected type:None path:null
> > >>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread INFO
> > >>> > org.I0Itec.zkclient.ZkClient-449: zookeeper state changed
> > >>> (SyncConnected)
> > >>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
> > >>> > 2014-11-05 23:39:02,841 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-638: State is SyncConnected
> > >>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> > >>> > kafka.utils.KafkaScheduler-52: Initializing task scheduler.
> > >>> > 2014-11-05 23:39:02,842 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting auto
> > >>> > committer every 1000 ms
> > >>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> > >>> > kafka.utils.KafkaScheduler-52: Scheduling task
> > >>> kafka-consumer-autocommit
> > >>> > with initial delay 1000 ms and period 1000 ms.
> > >>> > 2014-11-05 23:39:02,843 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], entering
> consume
> > >>> > 2014-11-05 23:39:02,843 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin
> > registering
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> > >>> > 2014-11-05 23:39:02,846
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 1,1 replyHeader:: 1,29,-101 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> > >>> > response::
> > >>> > 2014-11-05 23:39:02,854
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 2,1 replyHeader:: 2,30,-101 request::
> > >>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> > >>> > 2014-11-05 23:39:02,858
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 3,1 replyHeader:: 3,31,0 request::
> > >>> > '/consumers/TestCheck,,v{s{31,s{'world,'anyone}}},0 response::
> > >>> > '/consumers/TestCheck
> > >>> > 2014-11-05 23:39:02,861
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 4,1 replyHeader:: 4,32,0 request::
> > >>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> > >>> > '/consumers/TestCheck/ids
> > >>> > 2014-11-05 23:39:02,866
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 5,1 replyHeader:: 5,33,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> > >>> > response::
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> > >>> > 2014-11-05 23:39:02,867 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end
> registering
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> > >>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Adding
> > >>> topicThreadId
> > >>> > (Test,TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0) and
> queue
> > >>> [] to
> > >>> > topicThreadIdAndQueues data structure
> > >>> > 2014-11-05 23:39:02,868
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting
> watcher
> > >>> > executor thread for consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> > >>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], adding topic
> > Test
> > >>> and
> > >>> > 1 streams to map.
> > >>> > 2014-11-05 23:39:02,870
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 6,3 replyHeader:: 6,33,0 request::
> > >>> > '/consumers/TestCheck/ids,T response::
> > >>> > s{32,32,1415230742859,1415230742859,0,1,0,0,0,1,33}
> > >>> > 2014-11-05 23:39:02,872
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 7,8 replyHeader:: 7,33,0 request::
> > >>> > '/consumers/TestCheck/ids,T response::
> > >>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> > >>> > 2014-11-05 23:39:02,873
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 8,3 replyHeader:: 8,33,-101 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> > 2014-11-05 23:39:02,873 pool-13-thread-2 DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-137: Subscribed data changes for
> > >>> > /brokers/topics/Test
> > >>> > 2014-11-05 23:39:02,873 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin
> > rebalancing
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > >>> > 2014-11-05 23:39:02,875 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.async.DefaultEventHandler-52: Handling 1 events
> > >>> > 2014-11-05 23:39:02,877
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 9,8 replyHeader:: 9,33,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:02,879
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 10,4 replyHeader:: 10,33,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:02,887
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 11,4 replyHeader:: 11,33,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > >>> > response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> > >>> >
> > >>> > 2014-11-05 23:39:02,896
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 12,8 replyHeader:: 12,33,0 request::
> > >>> > '/consumers/TestCheck/ids,T response::
> > >>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> > >>> > 2014-11-05 23:39:02,898
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 13,4 replyHeader:: 13,33,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > >>> > response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> > >>> >
> > >>> > 2014-11-05 23:39:02,908
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 14,8 replyHeader:: 14,33,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:02,910
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 15,4 replyHeader:: 15,33,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:02,917 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 0 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:02,917
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 16,4 replyHeader:: 16,33,-101 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> > 2014-11-05 23:39:02,920 pool-13-thread-2 DEBUG
> > >>> kafka.utils.ZkUtils$-52:
> > >>> > Partition map for /brokers/topics/Test is Map()
> > >>> > 2014-11-05 23:39:02,920 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder
> thread
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] All connections stopped
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
> > >>> relevant
> > >>> > queues for this fetcher
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk
> > for
> > >>> > this consumer iterator
> > >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the
> data
> > >>> > chunks in all the consumer message iterators
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
> > >>> offsets
> > >>> > after clearing the fetcher queues
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing
> > >>> partition
> > >>> > ownership
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
> > >>> following
> > >>> > partitions: ArrayBuffer() for topic Test with consumers:
> > >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 WARN
> > >>> > kafka.consumer.ZookeeperConsumerConnector-83:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], No broker
> > >>> partitions
> > >>> > consumed by consumer thread
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 for topic Test
> > >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the
> > cache
> > >>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
> > >>> topic
> > >>> > cache Map(Test -> ArrayBuffer())
> > >>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per
> > >>> topic
> > >>> > cache Map(Test ->
> > >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> > >>> > 2014-11-05 23:39:02,923 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected
> > partitions :
> > >>> > 2014-11-05 23:39:02,927 pool-13-thread-2 INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end
> rebalancing
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > >>> > 2014-11-05 23:39:02,928
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Starting
> > >>> > 2014-11-05 23:39:02,930 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:02,930 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:02,985
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-783: Got notification
> > >>> > sessionid:0x1498251e8680003
> > >>> > 2014-11-05 23:39:02,986
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-800: Got WatchedEvent
> > >>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test for
> > >>> > sessionid 0x1498251e8680003
> > >>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> > >>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test
> > >>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkEventThread-88: New event: ZkEvent[Data of
> > >>> > /brokers/topics/Test changed sent to
> > >>> >
> > >>> >
> > >>>
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> > >>> > ]
> > >>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> > >>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
> > >>> > 2014-11-05 23:39:02,988 ZkClient-EventThread-29-172.31.25.198:2181
> > >>> DEBUG
> > >>> > org.I0Itec.zkclient.ZkEventThread-69: Delivering event #1
> > ZkEvent[Data
> > >>> of
> > >>> > /brokers/topics/Test changed sent to
> > >>> >
> > >>> >
> > >>>
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> > >>> > ]
> > >>> > 2014-11-05 23:39:02,991
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 17,3 replyHeader:: 17,37,0 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> > s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> > >>> > 2014-11-05 23:39:02,993
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 18,4 replyHeader:: 18,37,0 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> > >>> >
> > >>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Topic info for
> > >>> path
> > >>> > /brokers/topics/Test changed to
> > >>> > {"version":1,"partitions":{"1":[0],"0":[0]}}, triggering rebalance
> > >>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
> > >>> DEBUG
> > >>> > org.I0Itec.zkclient.ZkEventThread-79: Delivering event #1 done
> > >>> > 2014-11-05 23:39:02,995
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin
> > rebalancing
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > >>> > 2014-11-05 23:39:02,996
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 19,8 replyHeader:: 19,37,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:02,998
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 20,4 replyHeader:: 20,37,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:03,008
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 21,4 replyHeader:: 21,37,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > >>> > response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> > >>> >
> > >>> > 2014-11-05 23:39:03,014
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 22,8 replyHeader:: 22,37,0 request::
> > >>> > '/consumers/TestCheck/ids,T response::
> > >>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> > >>> > 2014-11-05 23:39:03,018
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 23,4 replyHeader:: 23,37,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > >>> > response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> > >>> >
> > >>> > 2014-11-05 23:39:03,041
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 24,8 replyHeader:: 24,37,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:03,043
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 25,4 replyHeader:: 25,37,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:03,058
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 26,4 replyHeader:: 26,37,0 request::
> > >>> > '/brokers/topics/Test,T response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> > >>> >
> > >>> > 2014-11-05 23:39:03,074
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.utils.ZkUtils$-52: Partition map for /brokers/topics/Test is
> > >>> Map(1 ->
> > >>> > List(0), 0 -> List(0))
> > >>> > 2014-11-05 23:39:03,075
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder
> thread
> > >>> > 2014-11-05 23:39:03,082
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Shutting down
> > >>> > 2014-11-05 23:39:03,082 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:03,089
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Stopped
> > >>> > 2014-11-05 23:39:03,089
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Shutdown completed
> > >>> > 2014-11-05 23:39:03,089
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> > >>> > 2014-11-05 23:39:03,090
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ConsumerFetcherManager-68:
> > >>> > [ConsumerFetcherManager-1415230742842] All connections stopped
> > >>> > 2014-11-05 23:39:03,090
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
> > >>> relevant
> > >>> > queues for this fetcher
> > >>> > 2014-11-05 23:39:03,090
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk
> > for
> > >>> > this consumer iterator
> > >>> > 2014-11-05 23:39:03,091
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the
> data
> > >>> > chunks in all the consumer message iterators
> > >>> > 2014-11-05 23:39:03,091
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
> > >>> offsets
> > >>> > after clearing the fetcher queues
> > >>> > 2014-11-05 23:39:03,091
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing
> > >>> partition
> > >>> > ownership
> > >>> > 2014-11-05 23:39:03,094 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:03,096
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
> > >>> following
> > >>> > partitions: ArrayBuffer(0, 1) for topic Test with consumers:
> > >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> > >>> > 2014-11-05 23:39:03,098
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to
> > claim
> > >>> > partition 0
> > >>> > 2014-11-05 23:39:03,100
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 27,4 replyHeader:: 27,37,-101 request::
> > >>> > '/consumers/TestCheck/offsets/Test/0,F response::
> > >>> > 2014-11-05 23:39:03,116
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of
> > >>> Test:0:
> > >>> > fetched offset = -1: consumed offset = -1 is -1
> > >>> > 2014-11-05 23:39:03,117
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of
> Test:0:
> > >>> > fetched offset = -1: consumed offset = -1 is -1
> > >>> > 2014-11-05 23:39:03,118
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:0:
> fetched
> > >>> offset
> > >>> > = -1: consumed offset = -1 selected new offset -1
> > >>> > 2014-11-05 23:39:03,169
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to
> > claim
> > >>> > partition 1
> > >>> > 2014-11-05 23:39:03,171
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 28,4 replyHeader:: 28,37,-101 request::
> > >>> > '/consumers/TestCheck/offsets/Test/1,F response::
> > >>> > 2014-11-05 23:39:03,172
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of
> > >>> Test:1:
> > >>> > fetched offset = -1: consumed offset = -1 is -1
> > >>> > 2014-11-05 23:39:03,172
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of
> Test:1:
> > >>> > fetched offset = -1: consumed offset = -1 is -1
> > >>> > 2014-11-05 23:39:03,172
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:1:
> fetched
> > >>> offset
> > >>> > = -1: consumed offset = -1 selected new offset -1
> > >>> > 2014-11-05 23:39:03,175 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:03,175
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 29,1 replyHeader:: 29,38,-101 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> > >>> > response::
> > >>> > 2014-11-05 23:39:03,179
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 30,1 replyHeader:: 30,39,-101 request::
> > >>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
> > >>> > response::
> > >>> > 2014-11-05 23:39:03,180 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:03,181 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 1 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:03,182 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:03,182 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:03,186
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 31,1 replyHeader:: 31,40,0 request::
> > >>> > '/consumers/TestCheck/owners,,v{s{31,s{'world,'anyone}}},0
> > response::
> > >>> > '/consumers/TestCheck/owners
> > >>> > 2014-11-05 23:39:03,189 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:03,190
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 32,1 replyHeader:: 32,41,0 request::
> > >>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
> > >>> response::
> > >>> > '/consumers/TestCheck/owners/Test
> > >>> > 2014-11-05 23:39:03,190 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:03,191 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:03,192 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:03,193
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 33,1 replyHeader:: 33,42,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> > >>> > response:: '/consumers/TestCheck/owners/Test/1
> > >>> > 2014-11-05 23:39:03,193 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 10
> > >>> > 2014-11-05 23:39:03,193
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully
> > owned
> > >>> > partition 1 for topic Test
> > >>> > 2014-11-05 23:39:03,196
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 34,1 replyHeader:: 34,43,0 request::
> > >>> >
> > >>> >
> > >>>
> >
> '/consumers/TestCheck/owners/Test/0,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> > >>> > response:: '/consumers/TestCheck/owners/Test/0
> > >>> > 2014-11-05 23:39:03,196
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully
> > owned
> > >>> > partition 0 for topic Test
> > >>> > 2014-11-05 23:39:03,196
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the
> > cache
> > >>> > 2014-11-05 23:39:03,197
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
> > >>> topic
> > >>> > cache Map(Test -> ArrayBuffer(0, 1))
> > >>> > 2014-11-05 23:39:03,197
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> DEBUG
> > >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per
> > >>> topic
> > >>> > cache Map(Test ->
> > >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> > >>> > 2014-11-05 23:39:03,197
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected
> > partitions :
> > >>> > Test:0: fetched offset = -1: consumed offset = -1,Test:1: fetched
> > >>> offset =
> > >>> > -1: consumed offset = -1
> > >>> > 2014-11-05 23:39:03,199
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > >>> >
> > >>>
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > >>> > Starting
> > >>> > 2014-11-05 23:39:03,199
> > >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> > >>> INFO
> > >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> > >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end
> rebalancing
> > >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > >>> > 2014-11-05 23:39:03,202
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 35,8 replyHeader:: 35,43,0 request::
> > >>> > '/brokers/ids,F response:: v{'0}
> > >>> > 2014-11-05 23:39:03,204
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > >>> > sessionid:0x1498251e8680003, packet:: clientPath:null
> serverPath:null
> > >>> > finished:false header:: 36,4 replyHeader:: 36,43,0 request::
> > >>> > '/brokers/ids/0,F response::
> > >>> >
> > >>> >
> > >>>
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> > >>> >
> > >>> > 2014-11-05 23:39:03,214
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.utils.VerifiableProperties-68: Verifying properties
> > >>> > 2014-11-05 23:39:03,215
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.utils.VerifiableProperties-68: Property client.id is
> > >>> overridden
> > >>> > to TestCheck
> > >>> > 2014-11-05 23:39:03,215
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.utils.VerifiableProperties-68: Property
> > >>> metadata.broker.list is
> > >>> > overridden to x.x.x.x:9092
> > >>> > 2014-11-05 23:39:03,215
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.utils.VerifiableProperties-68: Property
> > request.timeout.ms
> > >>> is
> > >>> > overridden to 30000
> > >>> > 2014-11-05 23:39:03,217
> > >>> >
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > >>> > INFO kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:x.x.x.x,port:9092 with correlation id 0 for 1 topic(s)
> > >>> Set(Test)
> > >>> > 2014-11-05 23:39:04,195 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 2 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:04,196 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:04,196 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:04,199 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:04,199 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:04,200 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:04,200 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 3 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:04,201 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:04,205 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:04,205 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:04,206 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:04,206 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:04,206 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 9
> > >>> > 2014-11-05 23:39:04,698
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:05,205
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680003 after 0ms
> > >>> > 2014-11-05 23:39:05,207 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 4 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:05,208 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:05,208 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:05,211 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:05,211 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:05,212 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:05,212 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:05,212 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 5 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:05,213 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:05,213 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:05,218 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 8
> > >>> > 2014-11-05 23:39:06,218 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 6 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:06,219 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:06,219 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:06,224 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:06,224 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:06,225 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:06,225 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 7 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:06,226 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:06,230 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:06,230 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:06,231 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:06,231 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:06,231 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 7
> > >>> > 2014-11-05 23:39:06,699
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:07,208
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680003 after 0ms
> > >>> > 2014-11-05 23:39:07,232 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 8 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:07,233 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:07,233 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:07,236 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:07,236 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:07,237 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:07,237 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 9 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:07,238 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:07,241 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:07,241 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:07,242 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:07,242 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:07,242 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 6
> > >>> > 2014-11-05 23:39:08,243 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 10 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:08,243 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:08,245 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:08,249 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:08,250 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:08,250 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:08,251 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 11 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:08,252 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:08,255 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:08,256 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:08,256 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:08,257 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:08,257 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 5
> > >>> > 2014-11-05 23:39:08,702
> > >>> >
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680002 after 0ms
> > >>> > 2014-11-05 23:39:09,211
> > >>> >
> > >>> >
> > >>>
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> > >>> sessionid:
> > >>> > 0x1498251e8680003 after 0ms
> > >>> > 2014-11-05 23:39:09,257 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 12 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:09,258 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:09,258 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:09,261 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:09,262 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:09,263 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 13 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:09,264 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:09,264 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:09,268 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:09,268 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:09,269 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:09,269 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:09,269 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 4
> > >>> > 2014-11-05 23:39:10,270 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 14 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:10,271 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:10,271 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:10,275 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:10,275 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:10,275 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:10,276 pool-13-thread-1 DEBUG
> > >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition
> info
> > >>> for
> > >>> > topic Test
> > >>> > 2014-11-05 23:39:10,276 pool-13-thread-1 INFO
> > >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > >>> > id:0,host:172.31.25.198,port:9092 with correlation id 15 for 1
> > topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:10,277 pool-13-thread-1 DEBUG
> > >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> > >>> 102400
> > >>> > (requested 102400).
> > >>> > 2014-11-05 23:39:10,277 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092
> for
> > >>> > producing
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> > >>> > kafka.producer.SyncProducer-68: Disconnecting from
> > 172.31.25.198:9092
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 DEBUG
> > >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> > >>> topic(s)
> > >>> > Set(Test)
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 WARN
> > >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> metadata
> > >>> > [{TopicMetadata for topic Test ->
> > >>> > No partition metadata for topic Test due to
> > >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > >>> > kafka.common.LeaderNotAvailableException
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 ERROR
> > >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > >>> messages by
> > >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> > Test
> > >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> > >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> > >>> before
> > >>> > retrying send. Remaining retries = 3
> > >>> >
> > >>> >
> > >>> > Brokers Logs -
> > >>> >
> > >>> > [2014-11-05 23:39:02,983] INFO Topic creation
> > >>> > {"version":1,"partitions":{"1":[0],"0":[0]}}
> > (kafka.admin.AdminUtils$)
> > >>> > [2014-11-05 23:39:02,988] INFO [KafkaApi-0] Auto creation of topic
> > Test
> > >>> > with 2 partitions and replication factor 1 is successful!
> > >>> > (kafka.server.KafkaApis)
> > >>> > [2014-11-05 23:39:03,092] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:03,191] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:04,200] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:04,205] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:05,211] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:05,218] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:06,224] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:06,230] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:07,236] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> > [2014-11-05 23:39:07,241] INFO Closing socket connection to /
> > >>> 172.31.25.198
> > >>> > .
> > >>> > (kafka.network.Processor)
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Wed, Nov 5, 2014 at 3:05 PM, Guozhang Wang <wa...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Sameer,
> > >>> > >
> > >>> > > Yes, this is the server log. But there seems no abnormal entries
> in
> > >>> it,
> > >>> > and
> > >>> > > it does not cover the same time range as the producer client
> > throwing
> > >>> > > LeaderNotAvailableException
> > >>> > > (it was 10/24, 14:30).
> > >>> > >
> > >>> > > The reason that I want to check the server log at that same
> reason
> > is
> > >>> > > that LeaderNotAvailableException
> > >>> > > is set by the brokers while handling requests, and its server log
> > >>> will
> > >>> > show
> > >>> > > what is the cause it set this error code.
> > >>> > >
> > >>> > > Guozhang
> > >>> > >
> > >>> > > On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net>
> > >>> wrote:
> > >>> > >
> > >>> > > > Hi Guozhang,
> > >>> > > >
> > >>> > > > This is the server.log -
> > >>> > > >
> > >>> > > > [2014-11-04 20:21:57,510] INFO Verifying properties
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property advertised.host.name
> is
> > >>> > > overridden
> > >>> > > > to x.x.x.x (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property broker.id is
> overridden
> > >>> to 0
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property
> > controlled.shutdown.enable
> > >>> is
> > >>> > > > overridden to true (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property host.name is
> overridden
> > to
> > >>> > > > 172.31.25.198 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is
> > >>> > overridden
> > >>> > > to
> > >>> > > > false (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden
> to
> > >>> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property
> > >>> > log.retention.check.interval.ms
> > >>> > > is
> > >>> > > > overridden to 60000 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property log.retention.hours is
> > >>> > overridden
> > >>> > > > to 168 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is
> > >>> overridden
> > >>> > > to
> > >>> > > > 536870912 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,546] INFO Property num.io.threads is
> > >>> overridden
> > >>> > to 8
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property num.network.threads is
> > >>> > overridden
> > >>> > > > to 2 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property num.partitions is
> > >>> overridden
> > >>> > to 2
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property port is overridden to
> > 9092
> > >>> > > > (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property
> > >>> socket.receive.buffer.bytes is
> > >>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property
> socket.request.max.bytes
> > is
> > >>> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,547] INFO Property
> socket.send.buffer.bytes
> > is
> > >>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is
> > >>> overridden
> > >>> > > to
> > >>> > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,548] INFO Property
> > >>> > zookeeper.connection.timeout.ms
> > >>> > > is
> > >>> > > > overridden to 1000000 (kafka.utils.VerifiableProperties)
> > >>> > > > [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
> > >>> > > > (kafka.server.KafkaServer)
> > >>> > > > [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to
> > >>> > zookeeper
> > >>> > > on
> > >>> > > > localhost:2181 (kafka.server.KafkaServer)
> > >>> > > > [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs'
> > not
> > >>> > found,
> > >>> > > > creating it. (kafka.log.LogManager)
> > >>> > > > [2014-11-04 20:21:57,748] INFO Starting log cleanup with a
> period
> > >>> of
> > >>> > > 60000
> > >>> > > > ms. (kafka.log.LogManager)
> > >>> > > > [2014-11-04 20:21:57,752] INFO Starting log flusher with a
> > default
> > >>> > period
> > >>> > > > of 9223372036854775807 ms. (kafka.log.LogManager)
> > >>> > > > [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
> > >>> > > > ip-172-31-25-198.us-west-1.compute.internal:9092.
> > >>> > > (kafka.network.Acceptor)
> > >>> > > > [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0],
> > Started
> > >>> > > > (kafka.network.SocketServer)
> > >>> > > > [2014-11-04 20:21:57,852] INFO Will not load MX4J,
> mx4j-tools.jar
> > >>> is
> > >>> > not
> > >>> > > in
> > >>> > > > the classpath (kafka.utils.Mx4jLoader$)
> > >>> > > > [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
> > >>> > > > (kafka.server.ZookeeperLeaderElector)
> > >>> > > > [2014-11-04 20:21:57,967] INFO Registered broker 0 at path
> > >>> > /brokers/ids/0
> > >>> > > > with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
> > >>> > > > [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
> > >>> > > > (kafka.server.KafkaServer)
> > >>> > > > [2014-11-04 20:21:58,050] INFO New leader is 0
> > >>> > > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > >>> > > > [2014-11-04 20:22:01,809] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:22:11,812] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:29,084] INFO Topic creation
> > >>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
> > >>> (kafka.admin.AdminUtils$)
> > >>> > > > [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of
> > topic
> > >>> Test
> > >>> > > > with 2 partitions and replication factor 1 is successful!
> > >>> > > > (kafka.server.KafkaApis)
> > >>> > > > [2014-11-04 20:30:29,280] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:29,301] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:30,310] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:30,315] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:31,320] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:31,325] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:32,331] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:32,336] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:33,343] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:33,347] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:34,356] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:34,360] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:35,366] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:35,372] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:36,377] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:36,382] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:37,387] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:37,394] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:38,399] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:38,403] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:39,410] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:39,414] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:40,419] INFO Closing socket connection to /
> > >>> > > 172.31.25.198
> > >>> > > > .
> > >>> > > > (kafka.network.Processor)
> > >>> > > > [2014-11-04 20:30:41,602] INFO Topic creation
> > >>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
> > >>> (kafka.admin.AdminUtils$)
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > > thanks
> > >>> > > >
> > >>> > > >
> > >>> > > > On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <
> > wangguoz@gmail.com
> > >>> >
> > >>> > > wrote:
> > >>> > > >
> > >>> > > > > The server.log should be the broker logs.
> > >>> > > > >
> > >>> > > > > Guozhang
> > >>> > > > >
> > >>> > > > > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <
> > syami@ailive.net>
> > >>> > > wrote:
> > >>> > > > >
> > >>> > > > > > I do not see any other logs besides these.
> > >>> > > > > > I checked the log4j.properties and where the logs are
> getting
> > >>> > stored.
> > >>> > > > > > The only other logs (apart from the controller logs) in the
> > >>> > directory
> > >>> > > > are
> > >>> > > > > > zookeeper-gc.log,kafkaServer-gc.log, server.log and
> > >>> > state-change.log
> > >>> > > > > >
> > >>> > > > > > Thanks
> > >>> > > > > >
> > >>> > > > > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <
> > >>> wangguoz@gmail.com
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > Sorry for getting late on this Sameer.
> > >>> > > > > > >
> > >>> > > > > > > The logs you showed above are from controller logs, do
> you
> > >>> see
> > >>> > > > brokers
> > >>> > > > > > logs
> > >>> > > > > > > besides that?
> > >>> > > > > > >
> > >>> > > > > > > Guozhang
> > >>> > > > > > >
> > >>> > > > > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <
> > >>> syami@ailive.net>
> > >>> > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Hi Guozhang,
> > >>> > > > > > > >
> > >>> > > > > > > > Is there any clue in the above data as to what is
> wrong?
> > >>> > > > > > > >
> > >>> > > > > > > > thanks
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <
> > >>> syami@ailive.net
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > This is the one from the file specified in
> > >>> log4j.properties -
> > >>> > > > > > > > >
> > >>> > > > > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener
> on
> > >>> > > > Controller
> > >>> > > > > > 0]:
> > >>> > > > > > > > > Topic change listener fired for path /brokers/topics
> > with
> > >>> > > > children
> > >>> > > > > > Test
> > >>> > > > > > > > >
> > >>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > >>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener
> on
> > >>> > > Controller
> > >>> > > > > 0]:
> > >>> > > > > > > New
> > >>> > > > > > > > > topics: [Set(Test)], deleted topics: [Set()], new
> > >>> partition
> > >>> > > > replica
> > >>> > > > > > > > > assignment [Map([Test,0] -> List(0), [Test,1] ->
> > >>> List(0))]
> > >>> > > > > > > > >
> > >>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > >>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New
> > topic
> > >>> > > creation
> > >>> > > > > > > > callback
> > >>> > > > > > > > > for [Test,0],[Test,1]
> > (kafka.controller.KafkaController)
> > >>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New
> > >>> partition
> > >>> > > > > creation
> > >>> > > > > > > > > callback for [Test,0],[Test,1]
> > >>> > > (kafka.controller.KafkaController)
> > >>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Partition state
> machine
> > >>> on
> > >>> > > > > Controller
> > >>> > > > > > > 0]:
> > >>> > > > > > > > > Invoking state change to NewPartition for partitions
> > >>> > > > > > [Test,0],[Test,1]
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine
> > on
> > >>> > > > controller
> > >>> > > > > > 0]:
> > >>> > > > > > > > > Invoking state change to NewReplica for replicas
> > >>> > > > > > > > >
> > >>> > > > >
> > >>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > >>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,111] INFO [Partition state
> machine
> > >>> on
> > >>> > > > > Controller
> > >>> > > > > > > 0]:
> > >>> > > > > > > > > Invoking state change to OnlinePartition for
> partitions
> > >>> > > > > > > [Test,0],[Test,1]
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state
> > machine
> > >>> on
> > >>> > > > > > Controller
> > >>> > > > > > > > 0]:
> > >>> > > > > > > > > Live assigned replicas for partition [Test,0] are:
> > >>> [List(0)]
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state
> > machine
> > >>> on
> > >>> > > > > > Controller
> > >>> > > > > > > > 0]:
> > >>> > > > > > > > > Initializing leader and isr for partition [Test,0] to
> > >>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state
> > machine
> > >>> on
> > >>> > > > > > Controller
> > >>> > > > > > > > 0]:
> > >>> > > > > > > > > Live assigned replicas for partition [Test,1] are:
> > >>> [List(0)]
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state
> > machine
> > >>> on
> > >>> > > > > > Controller
> > >>> > > > > > > > 0]:
> > >>> > > > > > > > > Initializing leader and isr for partition [Test,1] to
> > >>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine
> > on
> > >>> > > > controller
> > >>> > > > > > 0]:
> > >>> > > > > > > > > Invoking state change to OnlineReplica for replicas
> > >>> > > > > > > > >
> > >>> > > > >
> > >>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > >>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
> > >>> > > > > > > > > [2014-10-28 22:39:28,160] ERROR
> > >>> > > > > > [Controller-0-to-broker-0-send-thread],
> > >>> > > > > > > > > Controller 0's connection to broker
> > >>> > id:0,host:x.x.x.x,port:9092
> > >>> > > > was
> > >>> > > > > > > > > unsuccessful (kafka.controller.RequestSendThread)
> > >>> > > > > > > > > java.net.ConnectException: Connection timed out
> > >>> > > > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > >>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > >>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:449)
> > >>> > > > > > > > > at
> > >>> > > > > >
> > >>> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> > >>> > > > > > > > > at
> > >>> > > > > >
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > >>> > > > > > > > > at
> > >>> > > > > >
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > >>> > > > > > > > > [2014-10-28 22:39:28,460] ERROR
> > >>> > > > > > [Controller-0-to-broker-0-send-thread],
> > >>> > > > > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr
> > request
> > >>> with
> > >>> > > > > > > correlation
> > >>> > > > > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092.
> > Reconnecting
> > >>> to
> > >>> > > > broker.
> > >>> > > > > > > > > (kafka.controller.RequestSendThread)
> > >>> > > > > > > > > java.nio.channels.ClosedChannelException
> > >>> > > > > > > > > at
> > >>> > > > kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> > >>> > > > > > > > > at
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > >>> > > > > > > > > at
> > >>> > > > > >
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
> > >>> > > > wangguoz@gmail.com
> > >>> > > > > >
> > >>> > > > > > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > >> Sameer, the one you sent me is from the producer
> logs,
> > >>> not
> > >>> > the
> > >>> > > > > > broker
> > >>> > > > > > > > >> logs.
> > >>> > > > > > > > >> And /tmp/kafka-logs actually store the message log
> > >>> (i.e. the
> > >>> > > > > "data")
> > >>> > > > > > > > >> instead of the server logs. Their locations should
> be
> > >>> > > specified
> > >>> > > > in
> > >>> > > > > > > your
> > >>> > > > > > > > >> log4j.properties file.
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <
> > >>> > > syami@ailive.net>
> > >>> > > > > > > wrote:
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> > Is this what you are looking for -
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > >>> > > > 172.31.16.31:9092
> > >>> > > > > > for
> > >>> > > > > > > > >> > producing
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > >>> > > > > > > 172.31.16.31:9092
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > >>> > metadata
> > >>> > > > for
> > >>> > > > > 1
> > >>> > > > > > > > >> topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > >>> > fetching
> > >>> > > > > > metadata
> > >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > >>> > > > > > > > >> > No partition metadata for topic Test due to
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for
> topic
> > >>> > [Test]:
> > >>> > > > > class
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting
> > broker
> > >>> > > > partition
> > >>> > > > > > info
> > >>> > > > > > > > for
> > >>> > > > > > > > >> > topic Test
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata
> from
> > >>> > broker
> > >>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation
> > id 9
> > >>> > for 1
> > >>> > > > > > > topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket
> > with
> > >>> > > > > SO_TIMEOUT =
> > >>> > > > > > > > 10000
> > >>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested
> > -1),
> > >>> > > > SO_SNDBUF
> > >>> > > > > =
> > >>> > > > > > > > 102400
> > >>> > > > > > > > >> > (requested 102400).
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > >>> > > > 172.31.16.31:9092
> > >>> > > > > > for
> > >>> > > > > > > > >> > producing
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > >>> > > > > > > 172.31.16.31:9092
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > >>> > metadata
> > >>> > > > for
> > >>> > > > > 1
> > >>> > > > > > > > >> topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > >>> > fetching
> > >>> > > > > > metadata
> > >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > >>> > > > > > > > >> > No partition metadata for topic Test due to
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for
> topic
> > >>> > [Test]:
> > >>> > > > > class
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
> > >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97:
> Failed
> > to
> > >>> > > collate
> > >>> > > > > > > > messages
> > >>> > > > > > > > >> by
> > >>> > > > > > > > >> > topic, partition due to: Failed to fetch topic
> > >>> metadata
> > >>> > for
> > >>> > > > > topic:
> > >>> > > > > > > > Test
> > >>> > > > > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back
> > off
> > >>> for
> > >>> > > 1000
> > >>> > > > > ms
> > >>> > > > > > > > before
> > >>> > > > > > > > >> > retrying send. Remaining retries = 6
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata
> from
> > >>> > broker
> > >>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation
> id
> > >>> 10
> > >>> > for
> > >>> > > 1
> > >>> > > > > > > topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket
> > with
> > >>> > > > > SO_TIMEOUT =
> > >>> > > > > > > > 10000
> > >>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested
> > -1),
> > >>> > > > SO_SNDBUF
> > >>> > > > > =
> > >>> > > > > > > > 102400
> > >>> > > > > > > > >> > (requested 102400).
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > >>> > > > 172.31.16.31:9092
> > >>> > > > > > for
> > >>> > > > > > > > >> > producing
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > >>> > > > > > > 172.31.16.31:9092
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > >>> > metadata
> > >>> > > > for
> > >>> > > > > 1
> > >>> > > > > > > > >> topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > >>> > fetching
> > >>> > > > > > metadata
> > >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > >>> > > > > > > > >> > No partition metadata for topic Test due to
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for
> topic
> > >>> > [Test]:
> > >>> > > > > class
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting
> > broker
> > >>> > > > partition
> > >>> > > > > > info
> > >>> > > > > > > > for
> > >>> > > > > > > > >> > topic Test
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata
> from
> > >>> > broker
> > >>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation
> id
> > >>> 11
> > >>> > for
> > >>> > > 1
> > >>> > > > > > > topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket
> > with
> > >>> > > > > SO_TIMEOUT =
> > >>> > > > > > > > 10000
> > >>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested
> > -1),
> > >>> > > > SO_SNDBUF
> > >>> > > > > =
> > >>> > > > > > > > 102400
> > >>> > > > > > > > >> > (requested 102400).
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > >>> > > > 172.31.16.31:9092
> > >>> > > > > > for
> > >>> > > > > > > > >> > producing
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > >>> > > > > > > 172.31.16.31:9092
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
> > >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > >>> > metadata
> > >>> > > > for
> > >>> > > > > 1
> > >>> > > > > > > > >> topic(s)
> > >>> > > > > > > > >> > Set(Test)
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
> > >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > >>> > fetching
> > >>> > > > > > metadata
> > >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > >>> > > > > > > > >> > No partition metadata for topic Test due to
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for
> topic
> > >>> > [Test]:
> > >>> > > > > class
> > >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
> > >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97:
> Failed
> > to
> > >>> > > collate
> > >>> > > > > > > > messages
> > >>> > > > > > > > >> by
> > >>> > > > > > > > >> > topic, partition due to: Failed to fetch topic
> > >>> metadata
> > >>> > for
> > >>> > > > > topic:
> > >>> > > > > > > > Test
> > >>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
> > >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back
> > off
> > >>> for
> > >>> > > 1000
> > >>> > > > > ms
> > >>> > > > > > > > before
> > >>> > > > > > > > >> > retrying send. Remaining retries = 5
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > As for /tmp/kafka-logs, I can see only these
> files -
> > >>> > > > > > > > >> > recovery-point-offset-checkpoint
> > >>> > > > replication-offset-checkpoint
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > I do not see the usual Kafka log files here.
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > thanks
> > >>> > > > > > > > >> > Sameer
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
> > >>> > > > > > wangguoz@gmail.com>
> > >>> > > > > > > > >> wrote:
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > > Sameer, could you check broker logs and see if
> it
> > >>> has
> > >>> > > > similar
> > >>> > > > > > > > >> > > LeaderNotAvailable exceptions?
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > Guozhang
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
> > >>> > > > > syami@ailive.net
> > >>> > > > > > >
> > >>> > > > > > > > >> wrote:
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > Yes. I can telnet from outside using the
> public
> > >>> ip of
> > >>> > > the
> > >>> > > > > > > instance
> > >>> > > > > > > > >> and
> > >>> > > > > > > > >> > > port
> > >>> > > > > > > > >> > > > of 9092
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger
> Hoover <
> > >>> > > > > > > > >> roger.hoover@gmail.com
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > wrote:
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > Just a guess but could it be a firewall
> issue?
> > >>> Did
> > >>> > > you
> > >>> > > > > > enable
> > >>> > > > > > > > >> > > > connections
> > >>> > > > > > > > >> > > > > to port 9092 from outside EC2 in a security
> > >>> group?
> > >>> > > Can
> > >>> > > > > you
> > >>> > > > > > > > >> telnet to
> > >>> > > > > > > > >> > > > each
> > >>> > > > > > > > >> > > > > broker IP and port?
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer
> Yami
> > <
> > >>> > > > > > > syami@ailive.net
> > >>> > > > > > > > >
> > >>> > > > > > > > >> > > wrote:
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > > There was a typo earlier.
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > This is the output -
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > Topic:Test PartitionCount:2
> > >>> ReplicationFactor:1
> > >>> > > > > > > Configs:
> > >>> > > > > > > > >> > > > > > Topic: Test Partition: 0
> > >>> Leader: 0
> > >>> > > > > > > Replicas:
> > >>> > > > > > > > 0
> > >>> > > > > > > > >> > > > Isr: 0
> > >>> > > > > > > > >> > > > > > Topic: Test Partition: 1
> > >>> Leader: 0
> > >>> > > > > > > Replicas:
> > >>> > > > > > > > 0
> > >>> > > > > > > > >> > > > Isr: 0
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > thanks
> > >>> > > > > > > > >> > > > > > Sameer
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang
> > >>> Wang <
> > >>> > > > > > > > >> wangguoz@gmail.com
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > wrote:
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > > Sameer,
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > The topic name is not consistent from
> your
> > >>> > > describe
> > >>> > > > > > > outputs.
> > >>> > > > > > > > >> > > Normally
> > >>> > > > > > > > >> > > > > > they
> > >>> > > > > > > > >> > > > > > > should be either
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Topic:*Test* PartitionCount:2
> > >>> > > > > ReplicationFactor:1
> > >>> > > > > > > > >> > > Configs:
> > >>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 0
> > >>> > Leader: 0
> > >>> > > > > > > > >> Replicas: 0
> > >>> > > > > > > > >> > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 1
> > >>> > Leader: 0
> > >>> > > > > > > > >> Replicas: 0
> > >>> > > > > > > > >> > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Or
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
> > >>> > > > > > > > ReplicationFactor:1
> > >>> > > > > > > > >> > > > > Configs:
> > >>> > > > > > > > >> > > > > > > Topic: *Heartbeat*
> Partition: 0
> > >>> > > > Leader:
> > >>> > > > > 0
> > >>> > > > > > > > >> > > Replicas:
> > >>> > > > > > > > >> > > > 0
> > >>> > > > > > > > >> > > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > > Topic: *Heartbeat*
> Partition: 1
> > >>> > > > Leader:
> > >>> > > > > 0
> > >>> > > > > > > > >> > > Replicas:
> > >>> > > > > > > > >> > > > 0
> > >>> > > > > > > > >> > > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Did you manually modify the topic name
> in
> > >>> ZK?
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Guozhang
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer
> > >>> Yami <
> > >>> > > > > > > > >> syami@ailive.net>
> > >>> > > > > > > > >> > > > wrote:
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > > Hi Guozhang,
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > I was wondering if there is anything
> > >>> wrong in
> > >>> > > the
> > >>> > > > > > above
> > >>> > > > > > > > >> > > --describe
> > >>> > > > > > > > >> > > > > > data.
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > thanks
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM,
> Sameer
> > >>> Yami <
> > >>> > > > > > > > >> syami@ailive.net
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > wrote:
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > This is what I get -
> > >>> > > > > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper
> > >>> > > localhost:2181
> > >>> > > > > > > > --describe
> > >>> > > > > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
> > >>> > > > > > > > >> ReplicationFactor:1
> > >>> > > > > > > > >> > > > > > Configs:
> > >>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 0
> > >>> > > Leader: 0
> > >>> > > > > > > > >> > Replicas: 0
> > >>> > > > > > > > >> > > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 1
> > >>> > > Leader: 0
> > >>> > > > > > > > >> > Replicas: 0
> > >>> > > > > > > > >> > > > > > > Isr: 0
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > The topics are being created at
> > >>> run-time.
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > thanks
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM,
> > >>> Guozhang
> > >>> > > Wang <
> > >>> > > > > > > > >> > > > wangguoz@gmail.com
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > >> Could you use the topic tools
> > >>> "--describe"
> > >>> > to
> > >>> > > > > check
> > >>> > > > > > > if
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > topic
> > >>> > > > > > > > >> > > > > > > Test's
> > >>> > > > > > > > >> > > > > > > > >> partitions leaders are available?
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > >
> > >>> > > > >
> http://kafka.apache.org/documentation.html#basic_ops_add_topic
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >> Guozhang
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM,
> > Sameer
> > >>> > Yami
> > >>> > > <
> > >>> > > > > > > > >> > > syami@ailive.net>
> > >>> > > > > > > > >> > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >> > That is not the case.
> > >>> > > > > > > > >> > > > > > > > >> > This detailed error is below. Due
> > to
> > >>> this
> > >>> > > > > error ,
> > >>> > > > > > > the
> > >>> > > > > > > > >> > > producer
> > >>> > > > > > > > >> > > > > is
> > >>> > > > > > > > >> > > > > > > not
> > >>> > > > > > > > >> > > > > > > > >> able
> > >>> > > > > > > > >> > > > > > > > >> > to send messages
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main
> DEBUG
> > >>> > > > > > > > >> > > > > > > > >>
> > kafka.producer.BrokerPartitionInfo-52:
> > >>> > > > > > > > >> > > > > > > > >> > Getting broker partition info for
> > >>> topic
> > >>> > > Test
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main
> INFO
> > >>> > > > > > > > >> > > > > kafka.client.ClientUtils$-68:
> > >>> > > > > > > > >> > > > > > > > >> Fetching
> > >>> > > > > > > > >> > > > > > > > >> > metadata from broker
> > >>> > > > > > > > id:0,host:54.215.200.61,port:9092
> > >>> > > > > > > > >> > with
> > >>> > > > > > > > >> > > > > > > > correlation
> > >>> > > > > > > > >> > > > > > > > >> id
> > >>> > > > > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main
> DEBUG
> > >>> > > > > > > > >> > > > > > > kafka.network.BlockingChannel-52:
> > >>> > > > > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT =
> > >>> 10000
> > >>> > > > > (requested
> > >>> > > > > > > > >> 10000),
> > >>> > > > > > > > >> > > > > > SO_RCVBUF
> > >>> > > > > > > > >> > > > > > > =
> > >>> > > > > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF
> =
> > >>> 102400
> > >>> > > > > > > (requested
> > >>> > > > > > > > >> > > 102400).
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main
> INFO
> > >>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > >>> > > > > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092
> > for
> > >>> > > > producing
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main
> INFO
> > >>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > >>> > > > > > > > >> > > > > > > > >> > Disconnecting from
> > >>> 54.215.200.61:9092
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main
> DEBUG
> > >>> > > > > > > > >> > > > > kafka.client.ClientUtils$-52:
> > >>> > > > > > > > >> > > > > > > > >> > Successfully fetched metadata
> for 1
> > >>> > > topic(s)
> > >>> > > > > > > > Set(Test)
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main
> WARN
> > >>> > > > > > > > >> > > > > > > > >>
> > kafka.producer.BrokerPartitionInfo-83:
> > >>> > > > > > > > >> > > > > > > > >> > Error while fetching metadata
> > >>> > > [{TopicMetadata
> > >>> > > > > for
> > >>> > > > > > > > topic
> > >>> > > > > > > > >> > Test
> > >>> > > > > > > > >> > > > ->
> > >>> > > > > > > > >> > > > > > > > >> > No partition metadata for topic
> > Test
> > >>> due
> > >>> > to
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > kafka.common.LeaderNotAvailableException}]
> > >>> > > > for
> > >>> > > > > > > topic
> > >>> > > > > > > > >> > [Test]:
> > >>> > > > > > > > >> > > > > class
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main
> ERROR
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > kafka.producer.async.DefaultEventHandler-97:
> > >>> > > > > > Failed
> > >>> > > > > > > > to
> > >>> > > > > > > > >> > > collate
> > >>> > > > > > > > >> > > > > > > > messages
> > >>> > > > > > > > >> > > > > > > > >> by
> > >>> > > > > > > > >> > > > > > > > >> > topic, partition due to: Failed
> to
> > >>> fetch
> > >>> > > > topic
> > >>> > > > > > > > metadata
> > >>> > > > > > > > >> > for
> > >>> > > > > > > > >> > > > > topic:
> > >>> > > > > > > > >> > > > > > > > Test
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM,
> > >>> Guozhang
> > >>> > > > Wang
> > >>> > > > > <
> > >>> > > > > > > > >> > > > > > wangguoz@gmail.com>
> > >>> > > > > > > > >> > > > > > > > >> wrote:
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >> > > Could you see if this is the
> > case:
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >>
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> > >>> > > > > > > > >> > > > > > > > >> > > ?
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35
> AM,
> > >>> > Sameer
> > >>> > > > > Yami <
> > >>> > > > > > > > >> > > > > syami@ailive.net
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > > >> wrote:
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > > > Hi,
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper
> > test
> > >>> > setup
> > >>> > > > on a
> > >>> > > > > > > > single
> > >>> > > > > > > > >> EC2
> > >>> > > > > > > > >> > > > > > machine.
> > >>> > > > > > > > >> > > > > > > > The
> > >>> > > > > > > > >> > > > > > > > >> > > > host.name is the private ip
> > >>> address
> > >>> > > and
> > >>> > > > > the
> > >>> > > > > > > > >> > > > > > > advertised.host.name
> > >>> > > > > > > > >> > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > >> > the
> > >>> > > > > > > > >> > > > > > > > >> > > > public address in
> > >>> server.properties.
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > We are trying to publish from
> > >>> outside
> > >>> > > EC2
> > >>> > > > > > using
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > > public
> > >>> > > > > > > > >> > > > > ip
> > >>> > > > > > > > >> > > > > > of
> > >>> > > > > > > > >> > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > >> > EC2
> > >>> > > > > > > > >> > > > > > > > >> > > > machine but get this error on
> > the
> > >>> > > client
> > >>> > > > > > side -
> > >>> > > > > > > > >> > > > > > > > >> > > > No partition metadata for
> topic
> > >>> test
> > >>> > > due
> > >>> > > > to
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > kafka.common.LeaderNotAvailableException}]
> > >>> > > > > > for
> > >>> > > > > > > > >> topic
> > >>> > > > > > > > >> > > > [test]:
> > >>> > > > > > > > >> > > > > > > class
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > kafka.common.LeaderNotAvailableException
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > The zookeeper has the public
> ip
> > >>> > stored
> > >>> > > in
> > >>> > > > > > > > >> > > /brokers/ids/0.
> > >>> > > > > > > > >> > > > > > > > >> > > > We have tried multiple
> > >>> combinations
> > >>> > but
> > >>> > > > it
> > >>> > > > > > > works
> > >>> > > > > > > > >> ONLY
> > >>> > > > > > > > >> > > when
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > >> > advertised
> > >>> > > > > > > > >> > > > > > > > >> > > > host name is not used and the
> > >>> client
> > >>> > is
> > >>> > > > > > > internal
> > >>> > > > > > > > to
> > >>> > > > > > > > >> > EC2.
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > Please let me know what the
> > >>> problem
> > >>> > > could
> > >>> > > > > be.
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > > > thanks
> > >>> > > > > > > > >> > > > > > > > >> > > > Sameer
> > >>> > > > > > > > >> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> > > --
> > >>> > > > > > > > >> > > > > > > > >> > > -- Guozhang
> > >>> > > > > > > > >> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > > > > > >> >
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >> --
> > >>> > > > > > > > >> > > > > > > > >> -- Guozhang
> > >>> > > > > > > > >> > > > > > > > >>
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > --
> > >>> > > > > > > > >> > > > > > > -- Guozhang
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > --
> > >>> > > > > > > > >> > > -- Guozhang
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >>
> > >>> > > > > > > > >>
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> --
> > >>> > > > > > > > >> -- Guozhang
> > >>> > > > > > > > >>
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > --
> > >>> > > > > > > -- Guozhang
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > --
> > >>> > > > > -- Guozhang
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > --
> > >>> > > -- Guozhang
> > >>> > >
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>>
> > >>
> > >>
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Cannot connect to Kafka from outside of EC2
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Sameer,
I think apache mailing list has blocked your attachment. If it is too long
to include in the email body could you paste it somewhere and give me the
link?
Guozhang
On Tue, Nov 11, 2014 at 10:01 AM, Sameer Yami <sy...@ailive.net> wrote:
> Hi Guozhang,
>
> I was wondering if you found anything wrong in the logs/
>
> thanks
>
>
> On Fri, Nov 7, 2014 at 4:19 PM, Sameer Yami <sy...@ailive.net> wrote:
>
> > Hi Guozhang,
> >
> > Attached are the two logs with debug enabled.
> >
> > Thanks!
> >
> > On Fri, Nov 7, 2014 at 2:09 PM, Sameer Yami <sy...@ailive.net> wrote:
> >
> >> The version is kafka_2.10-0.8.1.1. It is not the latest trunk.
> >> Will try enabling debug version.
> >>
> >> thanks
> >>
> >>
> >> On Thu, Nov 6, 2014 at 9:37 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >>
> >>> Sameer,
> >>>
> >>> The server logs do not contain any non-INFO logs, which is a bit wired.
> >>> Did
> >>> you deploy the current trunk of Kafka? Also could you enable DEBUG
> level
> >>> logging on Kafka brokers?
> >>>
> >>> Guozhang
> >>>
> >>> On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami <sy...@ailive.net> wrote:
> >>>
> >>> > The server.log was taken separately.
> >>> > We ran the test again and the server and producer logs are below (to
> >>> get
> >>> > same timings).
> >>> >
> >>> >
> >>> > Thanks!
> >>> >
> >>> >
> >>> >
> >>>
> ------------------------------------------------------------------------------------
> >>> >
> >>> >
> >>> >
> >>> > Producer Logs -
> >>> >
> >>> >
> >>> > 2014-11-05 23:38:58,693
> >>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> >>> sessionid:
> >>> > 0x1498251e8680002 after 0ms
> >>> > 2014-11-05 23:39:00,695
> >>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> >>> sessionid:
> >>> > 0x1498251e8680002 after 0ms
> >>> > 2014-11-05 23:39:02,696
> >>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> >>> sessionid:
> >>> > 0x1498251e8680002 after 0ms
> >>> > 2014-11-05 23:39:02,828 pool-13-thread-2 INFO
> >>> > kafka.utils.VerifiableProperties-68: Verifying properties
> >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> >>> > kafka.utils.VerifiableProperties-68: Property
> auto.commit.interval.ms
> >>> is
> >>> > overridden to 1000
> >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> >>> > kafka.utils.VerifiableProperties-68: Property auto.offset.reset is
> >>> > overridden to smallest
> >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> >>> > kafka.utils.VerifiableProperties-68: Property consumer.timeout.ms is
> >>> > overridden to 10
> >>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> >>> > kafka.utils.VerifiableProperties-68: Property group.id is overridden
> >>> to
> >>> > TestCheck
> >>> > 2014-11-05 23:39:02,830 pool-13-thread-2 WARN
> >>> > kafka.utils.VerifiableProperties-83: Property serializer.class is not
> >>> valid
> >>> > 2014-11-05 23:39:02,830 pool-13-thread-2 INFO
> >>> > kafka.utils.VerifiableProperties-68: Property zookeeper.connect is
> >>> > overridden to 172.31.25.198:2181
> >>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Connecting to
> >>> > zookeeper instance at 172.31.25.198:2181
> >>> > 2014-11-05 23:39:02,831 pool-13-thread-2 DEBUG
> >>> > org.I0Itec.zkclient.ZkConnection-63: Creating new ZookKeeper instance
> >>> to
> >>> > connect to 172.31.25.198:2181.
> >>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> >>> > org.apache.zookeeper.ZooKeeper-379: Initiating client connection,
> >>> > connectString=172.31.25.198:2181 sessionTimeout=6000
> >>> > watcher=org.I0Itec.zkclient.ZkClient@3903b165
> >>> > 2014-11-05 23:39:02,831 ZkClient-EventThread-29-172.31.25.198:2181
> >>> INFO
> >>> > org.I0Itec.zkclient.ZkEventThread-64: Starting ZkClient event thread.
> >>> > 2014-11-05 23:39:02,831 pool-13-thread-1 INFO
> >>> > kafka.utils.VerifiableProperties-68: Verifying properties
> >>> > 2014-11-05 23:39:02,836 pool-13-thread-2-SendThread() INFO
> >>> > org.apache.zookeeper.ClientCnxn-1061: Opening socket connection to
> >>> server /
> >>> > 172.31.25.198:2181
> >>> > 2014-11-05 23:39:02,836 pool-13-thread-1 WARN
> >>> > kafka.utils.VerifiableProperties-83: Property batch.size is not valid
> >>> > 2014-11-05 23:39:02,832 pool-13-thread-2 DEBUG
> >>> > org.I0Itec.zkclient.ZkClient-878: Awaiting connection to Zookeeper
> >>> server
> >>> > 2014-11-05 23:39:02,836 pool-13-thread-1 INFO
> >>> > kafka.utils.VerifiableProperties-68: Property
> message.send.max.retries
> >>> is
> >>> > overridden to 10
> >>> > 2014-11-05 23:39:02,836 pool-13-thread-2 DEBUG
> >>> > org.I0Itec.zkclient.ZkClient-628: Waiting for keeper state
> >>> SyncConnected
> >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> >>> > kafka.utils.VerifiableProperties-68: Property metadata.broker.list is
> >>> > overridden to 172.31.25.198:9092
> >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> >>> > kafka.utils.VerifiableProperties-68: Property retry.backoff.ms is
> >>> > overridden to 1000
> >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> >>> > kafka.utils.VerifiableProperties-68: Property serializer.class is
> >>> > overridden to kafka.serializer.StringEncoder
> >>> > 2014-11-05 23:39:02,837
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > INFO org.apache.zookeeper.ClientCnxn-950: Socket connection
> >>> established to
> >>> > ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
> >>> initiating
> >>> > session
> >>> > 2014-11-05 23:39:02,838
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-999: Session establishment
> >>> request
> >>> > sent on ip-172-31-25-198.us-west-1.compute.internal/
> 172.31.25.198:2181
> >>> > 2014-11-05 23:39:02,837 pool-13-thread-1 WARN
> >>> > kafka.utils.VerifiableProperties-83: Property
> zk.connectiontimeout.ms
> >>> is
> >>> > not valid
> >>> > 2014-11-05 23:39:02,841
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > INFO org.apache.zookeeper.ClientCnxn-739: Session establishment
> >>> complete on
> >>> > server ip-172-31-25-198.us-west-1.compute.internal/
> 172.31.25.198:2181,
> >>> > sessionid = 0x1498251e8680003, negotiated timeout = 6000
> >>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> >>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> >>> > state:SyncConnected type:None path:null
> >>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread INFO
> >>> > org.I0Itec.zkclient.ZkClient-449: zookeeper state changed
> >>> (SyncConnected)
> >>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> >>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
> >>> > 2014-11-05 23:39:02,841 pool-13-thread-2 DEBUG
> >>> > org.I0Itec.zkclient.ZkClient-638: State is SyncConnected
> >>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> >>> > kafka.utils.KafkaScheduler-52: Initializing task scheduler.
> >>> > 2014-11-05 23:39:02,842 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting auto
> >>> > committer every 1000 ms
> >>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> >>> > kafka.utils.KafkaScheduler-52: Scheduling task
> >>> kafka-consumer-autocommit
> >>> > with initial delay 1000 ms and period 1000 ms.
> >>> > 2014-11-05 23:39:02,843 pool-13-thread-2 DEBUG
> >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], entering consume
> >>> > 2014-11-05 23:39:02,843 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin
> registering
> >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> >>> > 2014-11-05 23:39:02,846
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 1,1 replyHeader:: 1,29,-101 request::
> >>> >
> >>> >
> >>>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> >>> > response::
> >>> > 2014-11-05 23:39:02,854
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 2,1 replyHeader:: 2,30,-101 request::
> >>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> >>> > 2014-11-05 23:39:02,858
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 3,1 replyHeader:: 3,31,0 request::
> >>> > '/consumers/TestCheck,,v{s{31,s{'world,'anyone}}},0 response::
> >>> > '/consumers/TestCheck
> >>> > 2014-11-05 23:39:02,861
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 4,1 replyHeader:: 4,32,0 request::
> >>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> >>> > '/consumers/TestCheck/ids
> >>> > 2014-11-05 23:39:02,866
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 5,1 replyHeader:: 5,33,0 request::
> >>> >
> >>> >
> >>>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> >>> > response::
> >>> >
> >>>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> >>> > 2014-11-05 23:39:02,867 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end registering
> >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> >>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Adding
> >>> topicThreadId
> >>> > (Test,TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0) and queue
> >>> [] to
> >>> > topicThreadIdAndQueues data structure
> >>> > 2014-11-05 23:39:02,868
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting watcher
> >>> > executor thread for consumer
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> >>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], adding topic
> Test
> >>> and
> >>> > 1 streams to map.
> >>> > 2014-11-05 23:39:02,870
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 6,3 replyHeader:: 6,33,0 request::
> >>> > '/consumers/TestCheck/ids,T response::
> >>> > s{32,32,1415230742859,1415230742859,0,1,0,0,0,1,33}
> >>> > 2014-11-05 23:39:02,872
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 7,8 replyHeader:: 7,33,0 request::
> >>> > '/consumers/TestCheck/ids,T response::
> >>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> >>> > 2014-11-05 23:39:02,873
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 8,3 replyHeader:: 8,33,-101 request::
> >>> > '/brokers/topics/Test,T response::
> >>> > 2014-11-05 23:39:02,873 pool-13-thread-2 DEBUG
> >>> > org.I0Itec.zkclient.ZkClient-137: Subscribed data changes for
> >>> > /brokers/topics/Test
> >>> > 2014-11-05 23:39:02,873 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin
> rebalancing
> >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> >>> > 2014-11-05 23:39:02,875 pool-13-thread-1 DEBUG
> >>> > kafka.producer.async.DefaultEventHandler-52: Handling 1 events
> >>> > 2014-11-05 23:39:02,877
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 9,8 replyHeader:: 9,33,0 request::
> >>> > '/brokers/ids,F response:: v{'0}
> >>> > 2014-11-05 23:39:02,879
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 10,4 replyHeader:: 10,33,0 request::
> >>> > '/brokers/ids/0,F response::
> >>> >
> >>> >
> >>>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >>> >
> >>> > 2014-11-05 23:39:02,887
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 11,4 replyHeader:: 11,33,0 request::
> >>> >
> >>> >
> >>>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> >>> > response::
> >>> >
> >>> >
> >>>
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> >>> >
> >>> > 2014-11-05 23:39:02,896
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 12,8 replyHeader:: 12,33,0 request::
> >>> > '/consumers/TestCheck/ids,T response::
> >>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> >>> > 2014-11-05 23:39:02,898
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 13,4 replyHeader:: 13,33,0 request::
> >>> >
> >>> >
> >>>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> >>> > response::
> >>> >
> >>> >
> >>>
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> >>> >
> >>> > 2014-11-05 23:39:02,908
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 14,8 replyHeader:: 14,33,0 request::
> >>> > '/brokers/ids,F response:: v{'0}
> >>> > 2014-11-05 23:39:02,910
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 15,4 replyHeader:: 15,33,0 request::
> >>> > '/brokers/ids/0,F response::
> >>> >
> >>> >
> >>>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >>> >
> >>> > 2014-11-05 23:39:02,917 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 0 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:02,917
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 16,4 replyHeader:: 16,33,-101 request::
> >>> > '/brokers/topics/Test,T response::
> >>> > 2014-11-05 23:39:02,920 pool-13-thread-2 DEBUG
> >>> kafka.utils.ZkUtils$-52:
> >>> > Partition map for /brokers/topics/Test is Map()
> >>> > 2014-11-05 23:39:02,920 pool-13-thread-2 INFO
> >>> > kafka.consumer.ConsumerFetcherManager-68:
> >>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
> >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> >>> > kafka.consumer.ConsumerFetcherManager-68:
> >>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> >>> > kafka.consumer.ConsumerFetcherManager-68:
> >>> > [ConsumerFetcherManager-1415230742842] All connections stopped
> >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
> >>> relevant
> >>> > queues for this fetcher
> >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 DEBUG
> >>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk
> for
> >>> > this consumer iterator
> >>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
> >>> > chunks in all the consumer message iterators
> >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
> >>> offsets
> >>> > after clearing the fetcher queues
> >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing
> >>> partition
> >>> > ownership
> >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
> >>> following
> >>> > partitions: ArrayBuffer() for topic Test with consumers:
> >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 WARN
> >>> > kafka.consumer.ZookeeperConsumerConnector-83:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], No broker
> >>> partitions
> >>> > consumed by consumer thread
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 for topic Test
> >>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the
> cache
> >>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
> >>> topic
> >>> > cache Map(Test -> ArrayBuffer())
> >>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per
> >>> topic
> >>> > cache Map(Test ->
> >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> >>> > 2014-11-05 23:39:02,923 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected
> partitions :
> >>> > 2014-11-05 23:39:02,927 pool-13-thread-2 INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
> >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> >>> > 2014-11-05 23:39:02,928
> >>> >
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> >>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> >>> >
> >>>
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> >>> > Starting
> >>> > 2014-11-05 23:39:02,930 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:02,930 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:02,985
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-783: Got notification
> >>> > sessionid:0x1498251e8680003
> >>> > 2014-11-05 23:39:02,986
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-800: Got WatchedEvent
> >>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test for
> >>> > sessionid 0x1498251e8680003
> >>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> >>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> >>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test
> >>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> >>> > org.I0Itec.zkclient.ZkEventThread-88: New event: ZkEvent[Data of
> >>> > /brokers/topics/Test changed sent to
> >>> >
> >>> >
> >>>
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> >>> > ]
> >>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> >>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
> >>> > 2014-11-05 23:39:02,988 ZkClient-EventThread-29-172.31.25.198:2181
> >>> DEBUG
> >>> > org.I0Itec.zkclient.ZkEventThread-69: Delivering event #1
> ZkEvent[Data
> >>> of
> >>> > /brokers/topics/Test changed sent to
> >>> >
> >>> >
> >>>
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> >>> > ]
> >>> > 2014-11-05 23:39:02,991
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 17,3 replyHeader:: 17,37,0 request::
> >>> > '/brokers/topics/Test,T response::
> >>> > s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> >>> > 2014-11-05 23:39:02,993
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 18,4 replyHeader:: 18,37,0 request::
> >>> > '/brokers/topics/Test,T response::
> >>> >
> >>> >
> >>>
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> >>> >
> >>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Topic info for
> >>> path
> >>> > /brokers/topics/Test changed to
> >>> > {"version":1,"partitions":{"1":[0],"0":[0]}}, triggering rebalance
> >>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
> >>> DEBUG
> >>> > org.I0Itec.zkclient.ZkEventThread-79: Delivering event #1 done
> >>> > 2014-11-05 23:39:02,995
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin
> rebalancing
> >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> >>> > 2014-11-05 23:39:02,996
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 19,8 replyHeader:: 19,37,0 request::
> >>> > '/brokers/ids,F response:: v{'0}
> >>> > 2014-11-05 23:39:02,998
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 20,4 replyHeader:: 20,37,0 request::
> >>> > '/brokers/ids/0,F response::
> >>> >
> >>> >
> >>>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >>> >
> >>> > 2014-11-05 23:39:03,008
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 21,4 replyHeader:: 21,37,0 request::
> >>> >
> >>> >
> >>>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> >>> > response::
> >>> >
> >>> >
> >>>
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> >>> >
> >>> > 2014-11-05 23:39:03,014
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 22,8 replyHeader:: 22,37,0 request::
> >>> > '/consumers/TestCheck/ids,T response::
> >>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> >>> > 2014-11-05 23:39:03,018
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 23,4 replyHeader:: 23,37,0 request::
> >>> >
> >>> >
> >>>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> >>> > response::
> >>> >
> >>> >
> >>>
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> >>> >
> >>> > 2014-11-05 23:39:03,041
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 24,8 replyHeader:: 24,37,0 request::
> >>> > '/brokers/ids,F response:: v{'0}
> >>> > 2014-11-05 23:39:03,043
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 25,4 replyHeader:: 25,37,0 request::
> >>> > '/brokers/ids/0,F response::
> >>> >
> >>> >
> >>>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >>> >
> >>> > 2014-11-05 23:39:03,058
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 26,4 replyHeader:: 26,37,0 request::
> >>> > '/brokers/topics/Test,T response::
> >>> >
> >>> >
> >>>
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> >>> >
> >>> > 2014-11-05 23:39:03,074
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.utils.ZkUtils$-52: Partition map for /brokers/topics/Test is
> >>> Map(1 ->
> >>> > List(0), 0 -> List(0))
> >>> > 2014-11-05 23:39:03,075
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ConsumerFetcherManager-68:
> >>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
> >>> > 2014-11-05 23:39:03,082
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> >>> >
> >>>
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> >>> > Shutting down
> >>> > 2014-11-05 23:39:03,082 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:03,089
> >>> >
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> >>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> >>> >
> >>>
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> >>> > Stopped
> >>> > 2014-11-05 23:39:03,089
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> >>> >
> >>>
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> >>> > Shutdown completed
> >>> > 2014-11-05 23:39:03,089
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ConsumerFetcherManager-68:
> >>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> >>> > 2014-11-05 23:39:03,090
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ConsumerFetcherManager-68:
> >>> > [ConsumerFetcherManager-1415230742842] All connections stopped
> >>> > 2014-11-05 23:39:03,090
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
> >>> relevant
> >>> > queues for this fetcher
> >>> > 2014-11-05 23:39:03,090
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk
> for
> >>> > this consumer iterator
> >>> > 2014-11-05 23:39:03,091
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
> >>> > chunks in all the consumer message iterators
> >>> > 2014-11-05 23:39:03,091
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
> >>> offsets
> >>> > after clearing the fetcher queues
> >>> > 2014-11-05 23:39:03,091
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing
> >>> partition
> >>> > ownership
> >>> > 2014-11-05 23:39:03,094 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:03,096
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
> >>> following
> >>> > partitions: ArrayBuffer(0, 1) for topic Test with consumers:
> >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> >>> > 2014-11-05 23:39:03,098
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to
> claim
> >>> > partition 0
> >>> > 2014-11-05 23:39:03,100
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 27,4 replyHeader:: 27,37,-101 request::
> >>> > '/consumers/TestCheck/offsets/Test/0,F response::
> >>> > 2014-11-05 23:39:03,116
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of
> >>> Test:0:
> >>> > fetched offset = -1: consumed offset = -1 is -1
> >>> > 2014-11-05 23:39:03,117
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:0:
> >>> > fetched offset = -1: consumed offset = -1 is -1
> >>> > 2014-11-05 23:39:03,118
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:0: fetched
> >>> offset
> >>> > = -1: consumed offset = -1 selected new offset -1
> >>> > 2014-11-05 23:39:03,169
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to
> claim
> >>> > partition 1
> >>> > 2014-11-05 23:39:03,171
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 28,4 replyHeader:: 28,37,-101 request::
> >>> > '/consumers/TestCheck/offsets/Test/1,F response::
> >>> > 2014-11-05 23:39:03,172
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of
> >>> Test:1:
> >>> > fetched offset = -1: consumed offset = -1 is -1
> >>> > 2014-11-05 23:39:03,172
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:1:
> >>> > fetched offset = -1: consumed offset = -1 is -1
> >>> > 2014-11-05 23:39:03,172
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:1: fetched
> >>> offset
> >>> > = -1: consumed offset = -1 selected new offset -1
> >>> > 2014-11-05 23:39:03,175 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:03,175
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 29,1 replyHeader:: 29,38,-101 request::
> >>> >
> >>> >
> >>>
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> >>> > response::
> >>> > 2014-11-05 23:39:03,179
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 30,1 replyHeader:: 30,39,-101 request::
> >>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
> >>> > response::
> >>> > 2014-11-05 23:39:03,180 pool-13-thread-1 DEBUG
> >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
> >>> for
> >>> > topic Test
> >>> > 2014-11-05 23:39:03,181 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 1 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:03,182 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:03,182 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:03,186
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 31,1 replyHeader:: 31,40,0 request::
> >>> > '/consumers/TestCheck/owners,,v{s{31,s{'world,'anyone}}},0
> response::
> >>> > '/consumers/TestCheck/owners
> >>> > 2014-11-05 23:39:03,189 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:03,190
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 32,1 replyHeader:: 32,41,0 request::
> >>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
> >>> response::
> >>> > '/consumers/TestCheck/owners/Test
> >>> > 2014-11-05 23:39:03,190 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:03,191 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:03,192 pool-13-thread-1 ERROR
> >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> >>> messages by
> >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> Test
> >>> > 2014-11-05 23:39:03,193
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 33,1 replyHeader:: 33,42,0 request::
> >>> >
> >>> >
> >>>
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> >>> > response:: '/consumers/TestCheck/owners/Test/1
> >>> > 2014-11-05 23:39:03,193 pool-13-thread-1 INFO
> >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> >>> before
> >>> > retrying send. Remaining retries = 10
> >>> > 2014-11-05 23:39:03,193
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully
> owned
> >>> > partition 1 for topic Test
> >>> > 2014-11-05 23:39:03,196
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 34,1 replyHeader:: 34,43,0 request::
> >>> >
> >>> >
> >>>
> '/consumers/TestCheck/owners/Test/0,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> >>> > response:: '/consumers/TestCheck/owners/Test/0
> >>> > 2014-11-05 23:39:03,196
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully
> owned
> >>> > partition 0 for topic Test
> >>> > 2014-11-05 23:39:03,196
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the
> cache
> >>> > 2014-11-05 23:39:03,197
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
> >>> topic
> >>> > cache Map(Test -> ArrayBuffer(0, 1))
> >>> > 2014-11-05 23:39:03,197
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> DEBUG
> >>> > kafka.consumer.ZookeeperConsumerConnector-52:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per
> >>> topic
> >>> > cache Map(Test ->
> >>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> >>> > 2014-11-05 23:39:03,197
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected
> partitions :
> >>> > Test:0: fetched offset = -1: consumed offset = -1,Test:1: fetched
> >>> offset =
> >>> > -1: consumed offset = -1
> >>> > 2014-11-05 23:39:03,199
> >>> >
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> >>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> >>> >
> >>>
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> >>> > Starting
> >>> > 2014-11-05 23:39:03,199
> >>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
> >>> INFO
> >>> > kafka.consumer.ZookeeperConsumerConnector-68:
> >>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
> >>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> >>> > 2014-11-05 23:39:03,202
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 35,8 replyHeader:: 35,43,0 request::
> >>> > '/brokers/ids,F response:: v{'0}
> >>> > 2014-11-05 23:39:03,204
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> >>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> >>> > finished:false header:: 36,4 replyHeader:: 36,43,0 request::
> >>> > '/brokers/ids/0,F response::
> >>> >
> >>> >
> >>>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >>> >
> >>> > 2014-11-05 23:39:03,214
> >>> >
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> >>> > INFO kafka.utils.VerifiableProperties-68: Verifying properties
> >>> > 2014-11-05 23:39:03,215
> >>> >
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> >>> > INFO kafka.utils.VerifiableProperties-68: Property client.id is
> >>> overridden
> >>> > to TestCheck
> >>> > 2014-11-05 23:39:03,215
> >>> >
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> >>> > INFO kafka.utils.VerifiableProperties-68: Property
> >>> metadata.broker.list is
> >>> > overridden to x.x.x.x:9092
> >>> > 2014-11-05 23:39:03,215
> >>> >
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> >>> > INFO kafka.utils.VerifiableProperties-68: Property
> request.timeout.ms
> >>> is
> >>> > overridden to 30000
> >>> > 2014-11-05 23:39:03,217
> >>> >
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> >>> > INFO kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:x.x.x.x,port:9092 with correlation id 0 for 1 topic(s)
> >>> Set(Test)
> >>> > 2014-11-05 23:39:04,195 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 2 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:04,196 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:04,196 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:04,199 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:04,199 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:04,200 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:04,200 pool-13-thread-1 DEBUG
> >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
> >>> for
> >>> > topic Test
> >>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 3 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:04,201 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:04,205 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:04,205 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:04,206 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:04,206 pool-13-thread-1 ERROR
> >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> >>> messages by
> >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> Test
> >>> > 2014-11-05 23:39:04,206 pool-13-thread-1 INFO
> >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> >>> before
> >>> > retrying send. Remaining retries = 9
> >>> > 2014-11-05 23:39:04,698
> >>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> >>> sessionid:
> >>> > 0x1498251e8680002 after 0ms
> >>> > 2014-11-05 23:39:05,205
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> >>> sessionid:
> >>> > 0x1498251e8680003 after 0ms
> >>> > 2014-11-05 23:39:05,207 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 4 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:05,208 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:05,208 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:05,211 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:05,211 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:05,212 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:05,212 pool-13-thread-1 DEBUG
> >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
> >>> for
> >>> > topic Test
> >>> > 2014-11-05 23:39:05,212 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 5 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:05,213 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:05,213 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:05,217 pool-13-thread-1 ERROR
> >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> >>> messages by
> >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> Test
> >>> > 2014-11-05 23:39:05,218 pool-13-thread-1 INFO
> >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> >>> before
> >>> > retrying send. Remaining retries = 8
> >>> > 2014-11-05 23:39:06,218 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 6 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:06,219 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:06,219 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:06,224 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:06,224 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:06,225 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:06,225 pool-13-thread-1 DEBUG
> >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
> >>> for
> >>> > topic Test
> >>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 7 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:06,226 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:06,230 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:06,230 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:06,231 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:06,231 pool-13-thread-1 ERROR
> >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> >>> messages by
> >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> Test
> >>> > 2014-11-05 23:39:06,231 pool-13-thread-1 INFO
> >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> >>> before
> >>> > retrying send. Remaining retries = 7
> >>> > 2014-11-05 23:39:06,699
> >>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> >>> sessionid:
> >>> > 0x1498251e8680002 after 0ms
> >>> > 2014-11-05 23:39:07,208
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> >>> sessionid:
> >>> > 0x1498251e8680003 after 0ms
> >>> > 2014-11-05 23:39:07,232 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 8 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:07,233 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:07,233 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:07,236 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:07,236 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:07,237 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:07,237 pool-13-thread-1 DEBUG
> >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
> >>> for
> >>> > topic Test
> >>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 9 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:07,238 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:07,241 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:07,241 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:07,242 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:07,242 pool-13-thread-1 ERROR
> >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> >>> messages by
> >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> Test
> >>> > 2014-11-05 23:39:07,242 pool-13-thread-1 INFO
> >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> >>> before
> >>> > retrying send. Remaining retries = 6
> >>> > 2014-11-05 23:39:08,243 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 10 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:08,243 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:08,245 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:08,249 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:08,250 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:08,250 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
> >>> for
> >>> > topic Test
> >>> > 2014-11-05 23:39:08,251 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 11 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:08,252 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:08,255 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:08,256 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:08,256 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:08,257 pool-13-thread-1 ERROR
> >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> >>> messages by
> >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> Test
> >>> > 2014-11-05 23:39:08,257 pool-13-thread-1 INFO
> >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> >>> before
> >>> > retrying send. Remaining retries = 5
> >>> > 2014-11-05 23:39:08,702
> >>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> >>> sessionid:
> >>> > 0x1498251e8680002 after 0ms
> >>> > 2014-11-05 23:39:09,211
> >>> >
> >>> >
> >>>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> >>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> >>> sessionid:
> >>> > 0x1498251e8680003 after 0ms
> >>> > 2014-11-05 23:39:09,257 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 12 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:09,258 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:09,258 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:09,261 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:09,262 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
> >>> for
> >>> > topic Test
> >>> > 2014-11-05 23:39:09,263 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 13 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:09,264 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:09,264 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:09,268 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:09,268 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:09,269 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:09,269 pool-13-thread-1 ERROR
> >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> >>> messages by
> >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> Test
> >>> > 2014-11-05 23:39:09,269 pool-13-thread-1 INFO
> >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> >>> before
> >>> > retrying send. Remaining retries = 4
> >>> > 2014-11-05 23:39:10,270 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 14 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:10,271 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:10,271 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:10,275 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:10,275 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:10,275 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:10,276 pool-13-thread-1 DEBUG
> >>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
> >>> for
> >>> > topic Test
> >>> > 2014-11-05 23:39:10,276 pool-13-thread-1 INFO
> >>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> >>> > id:0,host:172.31.25.198,port:9092 with correlation id 15 for 1
> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:10,277 pool-13-thread-1 DEBUG
> >>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > (requested 102400).
> >>> > 2014-11-05 23:39:10,277 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> >>> > producing
> >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> >>> > kafka.producer.SyncProducer-68: Disconnecting from
> 172.31.25.198:9092
> >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 DEBUG
> >>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> >>> topic(s)
> >>> > Set(Test)
> >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 WARN
> >>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> >>> > [{TopicMetadata for topic Test ->
> >>> > No partition metadata for topic Test due to
> >>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> >>> > kafka.common.LeaderNotAvailableException
> >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 ERROR
> >>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> >>> messages by
> >>> > topic, partition due to: Failed to fetch topic metadata for topic:
> Test
> >>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> >>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
> >>> before
> >>> > retrying send. Remaining retries = 3
> >>> >
> >>> >
> >>> > Brokers Logs -
> >>> >
> >>> > [2014-11-05 23:39:02,983] INFO Topic creation
> >>> > {"version":1,"partitions":{"1":[0],"0":[0]}}
> (kafka.admin.AdminUtils$)
> >>> > [2014-11-05 23:39:02,988] INFO [KafkaApi-0] Auto creation of topic
> Test
> >>> > with 2 partitions and replication factor 1 is successful!
> >>> > (kafka.server.KafkaApis)
> >>> > [2014-11-05 23:39:03,092] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> > [2014-11-05 23:39:03,191] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> > [2014-11-05 23:39:04,200] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> > [2014-11-05 23:39:04,205] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> > [2014-11-05 23:39:05,211] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> > [2014-11-05 23:39:05,218] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> > [2014-11-05 23:39:06,224] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> > [2014-11-05 23:39:06,230] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> > [2014-11-05 23:39:07,236] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> > [2014-11-05 23:39:07,241] INFO Closing socket connection to /
> >>> 172.31.25.198
> >>> > .
> >>> > (kafka.network.Processor)
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Nov 5, 2014 at 3:05 PM, Guozhang Wang <wa...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Sameer,
> >>> > >
> >>> > > Yes, this is the server log. But there seems no abnormal entries in
> >>> it,
> >>> > and
> >>> > > it does not cover the same time range as the producer client
> throwing
> >>> > > LeaderNotAvailableException
> >>> > > (it was 10/24, 14:30).
> >>> > >
> >>> > > The reason that I want to check the server log at that same reason
> is
> >>> > > that LeaderNotAvailableException
> >>> > > is set by the brokers while handling requests, and its server log
> >>> will
> >>> > show
> >>> > > what is the cause it set this error code.
> >>> > >
> >>> > > Guozhang
> >>> > >
> >>> > > On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net>
> >>> wrote:
> >>> > >
> >>> > > > Hi Guozhang,
> >>> > > >
> >>> > > > This is the server.log -
> >>> > > >
> >>> > > > [2014-11-04 20:21:57,510] INFO Verifying properties
> >>> > > > (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,545] INFO Property advertised.host.name is
> >>> > > overridden
> >>> > > > to x.x.x.x (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,545] INFO Property broker.id is overridden
> >>> to 0
> >>> > > > (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,545] INFO Property
> controlled.shutdown.enable
> >>> is
> >>> > > > overridden to true (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,545] INFO Property host.name is overridden
> to
> >>> > > > 172.31.25.198 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is
> >>> > overridden
> >>> > > to
> >>> > > > false (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden to
> >>> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,546] INFO Property
> >>> > log.retention.check.interval.ms
> >>> > > is
> >>> > > > overridden to 60000 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,546] INFO Property log.retention.hours is
> >>> > overridden
> >>> > > > to 168 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is
> >>> overridden
> >>> > > to
> >>> > > > 536870912 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,546] INFO Property num.io.threads is
> >>> overridden
> >>> > to 8
> >>> > > > (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,547] INFO Property num.network.threads is
> >>> > overridden
> >>> > > > to 2 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,547] INFO Property num.partitions is
> >>> overridden
> >>> > to 2
> >>> > > > (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,547] INFO Property port is overridden to
> 9092
> >>> > > > (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,547] INFO Property
> >>> socket.receive.buffer.bytes is
> >>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,547] INFO Property socket.request.max.bytes
> is
> >>> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,547] INFO Property socket.send.buffer.bytes
> is
> >>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is
> >>> overridden
> >>> > > to
> >>> > > > localhost:2181 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,548] INFO Property
> >>> > zookeeper.connection.timeout.ms
> >>> > > is
> >>> > > > overridden to 1000000 (kafka.utils.VerifiableProperties)
> >>> > > > [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
> >>> > > > (kafka.server.KafkaServer)
> >>> > > > [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to
> >>> > zookeeper
> >>> > > on
> >>> > > > localhost:2181 (kafka.server.KafkaServer)
> >>> > > > [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs'
> not
> >>> > found,
> >>> > > > creating it. (kafka.log.LogManager)
> >>> > > > [2014-11-04 20:21:57,748] INFO Starting log cleanup with a period
> >>> of
> >>> > > 60000
> >>> > > > ms. (kafka.log.LogManager)
> >>> > > > [2014-11-04 20:21:57,752] INFO Starting log flusher with a
> default
> >>> > period
> >>> > > > of 9223372036854775807 ms. (kafka.log.LogManager)
> >>> > > > [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
> >>> > > > ip-172-31-25-198.us-west-1.compute.internal:9092.
> >>> > > (kafka.network.Acceptor)
> >>> > > > [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0],
> Started
> >>> > > > (kafka.network.SocketServer)
> >>> > > > [2014-11-04 20:21:57,852] INFO Will not load MX4J, mx4j-tools.jar
> >>> is
> >>> > not
> >>> > > in
> >>> > > > the classpath (kafka.utils.Mx4jLoader$)
> >>> > > > [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
> >>> > > > (kafka.server.ZookeeperLeaderElector)
> >>> > > > [2014-11-04 20:21:57,967] INFO Registered broker 0 at path
> >>> > /brokers/ids/0
> >>> > > > with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
> >>> > > > [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
> >>> > > > (kafka.server.KafkaServer)
> >>> > > > [2014-11-04 20:21:58,050] INFO New leader is 0
> >>> > > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> >>> > > > [2014-11-04 20:22:01,809] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:22:11,812] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:29,084] INFO Topic creation
> >>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
> >>> (kafka.admin.AdminUtils$)
> >>> > > > [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of
> topic
> >>> Test
> >>> > > > with 2 partitions and replication factor 1 is successful!
> >>> > > > (kafka.server.KafkaApis)
> >>> > > > [2014-11-04 20:30:29,280] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:29,301] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:30,310] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:30,315] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:31,320] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:31,325] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:32,331] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:32,336] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:33,343] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:33,347] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:34,356] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:34,360] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:35,366] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:35,372] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:36,377] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:36,382] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:37,387] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:37,394] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:38,399] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:38,403] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:39,410] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:39,414] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:40,419] INFO Closing socket connection to /
> >>> > > 172.31.25.198
> >>> > > > .
> >>> > > > (kafka.network.Processor)
> >>> > > > [2014-11-04 20:30:41,602] INFO Topic creation
> >>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
> >>> (kafka.admin.AdminUtils$)
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > thanks
> >>> > > >
> >>> > > >
> >>> > > > On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <
> wangguoz@gmail.com
> >>> >
> >>> > > wrote:
> >>> > > >
> >>> > > > > The server.log should be the broker logs.
> >>> > > > >
> >>> > > > > Guozhang
> >>> > > > >
> >>> > > > > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <
> syami@ailive.net>
> >>> > > wrote:
> >>> > > > >
> >>> > > > > > I do not see any other logs besides these.
> >>> > > > > > I checked the log4j.properties and where the logs are getting
> >>> > stored.
> >>> > > > > > The only other logs (apart from the controller logs) in the
> >>> > directory
> >>> > > > are
> >>> > > > > > zookeeper-gc.log,kafkaServer-gc.log, server.log and
> >>> > state-change.log
> >>> > > > > >
> >>> > > > > > Thanks
> >>> > > > > >
> >>> > > > > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <
> >>> wangguoz@gmail.com
> >>> > >
> >>> > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Sorry for getting late on this Sameer.
> >>> > > > > > >
> >>> > > > > > > The logs you showed above are from controller logs, do you
> >>> see
> >>> > > > brokers
> >>> > > > > > logs
> >>> > > > > > > besides that?
> >>> > > > > > >
> >>> > > > > > > Guozhang
> >>> > > > > > >
> >>> > > > > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <
> >>> syami@ailive.net>
> >>> > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Hi Guozhang,
> >>> > > > > > > >
> >>> > > > > > > > Is there any clue in the above data as to what is wrong?
> >>> > > > > > > >
> >>> > > > > > > > thanks
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <
> >>> syami@ailive.net
> >>> > >
> >>> > > > > wrote:
> >>> > > > > > > >
> >>> > > > > > > > > This is the one from the file specified in
> >>> log4j.properties -
> >>> > > > > > > > >
> >>> > > > > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on
> >>> > > > Controller
> >>> > > > > > 0]:
> >>> > > > > > > > > Topic change listener fired for path /brokers/topics
> with
> >>> > > > children
> >>> > > > > > Test
> >>> > > > > > > > >
> >>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> >>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener on
> >>> > > Controller
> >>> > > > > 0]:
> >>> > > > > > > New
> >>> > > > > > > > > topics: [Set(Test)], deleted topics: [Set()], new
> >>> partition
> >>> > > > replica
> >>> > > > > > > > > assignment [Map([Test,0] -> List(0), [Test,1] ->
> >>> List(0))]
> >>> > > > > > > > >
> >>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> >>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New
> topic
> >>> > > creation
> >>> > > > > > > > callback
> >>> > > > > > > > > for [Test,0],[Test,1]
> (kafka.controller.KafkaController)
> >>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New
> >>> partition
> >>> > > > > creation
> >>> > > > > > > > > callback for [Test,0],[Test,1]
> >>> > > (kafka.controller.KafkaController)
> >>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Partition state machine
> >>> on
> >>> > > > > Controller
> >>> > > > > > > 0]:
> >>> > > > > > > > > Invoking state change to NewPartition for partitions
> >>> > > > > > [Test,0],[Test,1]
> >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> >>> > > > > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine
> on
> >>> > > > controller
> >>> > > > > > 0]:
> >>> > > > > > > > > Invoking state change to NewReplica for replicas
> >>> > > > > > > > >
> >>> > > > >
> >>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> >>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
> >>> > > > > > > > > [2014-10-28 22:38:36,111] INFO [Partition state machine
> >>> on
> >>> > > > > Controller
> >>> > > > > > > 0]:
> >>> > > > > > > > > Invoking state change to OnlinePartition for partitions
> >>> > > > > > > [Test,0],[Test,1]
> >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> >>> > > > > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state
> machine
> >>> on
> >>> > > > > > Controller
> >>> > > > > > > > 0]:
> >>> > > > > > > > > Live assigned replicas for partition [Test,0] are:
> >>> [List(0)]
> >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> >>> > > > > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state
> machine
> >>> on
> >>> > > > > > Controller
> >>> > > > > > > > 0]:
> >>> > > > > > > > > Initializing leader and isr for partition [Test,0] to
> >>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> >>> > > > > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state
> machine
> >>> on
> >>> > > > > > Controller
> >>> > > > > > > > 0]:
> >>> > > > > > > > > Live assigned replicas for partition [Test,1] are:
> >>> [List(0)]
> >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> >>> > > > > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state
> machine
> >>> on
> >>> > > > > > Controller
> >>> > > > > > > > 0]:
> >>> > > > > > > > > Initializing leader and isr for partition [Test,1] to
> >>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> >>> > > > > > > > > (kafka.controller.PartitionStateMachine)
> >>> > > > > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine
> on
> >>> > > > controller
> >>> > > > > > 0]:
> >>> > > > > > > > > Invoking state change to OnlineReplica for replicas
> >>> > > > > > > > >
> >>> > > > >
> >>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> >>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
> >>> > > > > > > > > [2014-10-28 22:39:28,160] ERROR
> >>> > > > > > [Controller-0-to-broker-0-send-thread],
> >>> > > > > > > > > Controller 0's connection to broker
> >>> > id:0,host:x.x.x.x,port:9092
> >>> > > > was
> >>> > > > > > > > > unsuccessful (kafka.controller.RequestSendThread)
> >>> > > > > > > > > java.net.ConnectException: Connection timed out
> >>> > > > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> >>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> >>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:449)
> >>> > > > > > > > > at
> >>> > > > > >
> >>> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> >>> > > > > > > > > at
> >>> > > > > >
> kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> >>> > > > > > > > > at
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
> >>> > > > > > > > > at
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
> >>> > > > > > > > > at
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> >>> > > > > > > > > at
> >>> > > > > >
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >>> > > > > > > > > [2014-10-28 22:39:28,460] ERROR
> >>> > > > > > [Controller-0-to-broker-0-send-thread],
> >>> > > > > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr
> request
> >>> with
> >>> > > > > > > correlation
> >>> > > > > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092.
> Reconnecting
> >>> to
> >>> > > > broker.
> >>> > > > > > > > > (kafka.controller.RequestSendThread)
> >>> > > > > > > > > java.nio.channels.ClosedChannelException
> >>> > > > > > > > > at
> >>> > > > kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> >>> > > > > > > > > at
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> >>> > > > > > > > > at
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> >>> > > > > > > > > at
> >>> > > > > >
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >>> > > > > > > > >
> >>> > > > > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
> >>> > > > wangguoz@gmail.com
> >>> > > > > >
> >>> > > > > > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > >> Sameer, the one you sent me is from the producer logs,
> >>> not
> >>> > the
> >>> > > > > > broker
> >>> > > > > > > > >> logs.
> >>> > > > > > > > >> And /tmp/kafka-logs actually store the message log
> >>> (i.e. the
> >>> > > > > "data")
> >>> > > > > > > > >> instead of the server logs. Their locations should be
> >>> > > specified
> >>> > > > in
> >>> > > > > > > your
> >>> > > > > > > > >> log4j.properties file.
> >>> > > > > > > > >>
> >>> > > > > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <
> >>> > > syami@ailive.net>
> >>> > > > > > > wrote:
> >>> > > > > > > > >>
> >>> > > > > > > > >> > Is this what you are looking for -
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> >>> > > > 172.31.16.31:9092
> >>> > > > > > for
> >>> > > > > > > > >> > producing
> >>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> >>> > > > > > > 172.31.16.31:9092
> >>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
> >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> >>> > metadata
> >>> > > > for
> >>> > > > > 1
> >>> > > > > > > > >> topic(s)
> >>> > > > > > > > >> > Set(Test)
> >>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
> >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> >>> > fetching
> >>> > > > > > metadata
> >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> >>> > > > > > > > >> > No partition metadata for topic Test due to
> >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> >>> > [Test]:
> >>> > > > > class
> >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> >>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
> >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting
> broker
> >>> > > > partition
> >>> > > > > > info
> >>> > > > > > > > for
> >>> > > > > > > > >> > topic Test
> >>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
> >>> > broker
> >>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation
> id 9
> >>> > for 1
> >>> > > > > > > topic(s)
> >>> > > > > > > > >> > Set(Test)
> >>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
> >>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket
> with
> >>> > > > > SO_TIMEOUT =
> >>> > > > > > > > 10000
> >>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested
> -1),
> >>> > > > SO_SNDBUF
> >>> > > > > =
> >>> > > > > > > > 102400
> >>> > > > > > > > >> > (requested 102400).
> >>> > > > > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> >>> > > > 172.31.16.31:9092
> >>> > > > > > for
> >>> > > > > > > > >> > producing
> >>> > > > > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> >>> > > > > > > 172.31.16.31:9092
> >>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
> >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> >>> > metadata
> >>> > > > for
> >>> > > > > 1
> >>> > > > > > > > >> topic(s)
> >>> > > > > > > > >> > Set(Test)
> >>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
> >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> >>> > fetching
> >>> > > > > > metadata
> >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> >>> > > > > > > > >> > No partition metadata for topic Test due to
> >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> >>> > [Test]:
> >>> > > > > class
> >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> >>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
> >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed
> to
> >>> > > collate
> >>> > > > > > > > messages
> >>> > > > > > > > >> by
> >>> > > > > > > > >> > topic, partition due to: Failed to fetch topic
> >>> metadata
> >>> > for
> >>> > > > > topic:
> >>> > > > > > > > Test
> >>> > > > > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back
> off
> >>> for
> >>> > > 1000
> >>> > > > > ms
> >>> > > > > > > > before
> >>> > > > > > > > >> > retrying send. Remaining retries = 6
> >>> > > > > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
> >>> > broker
> >>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id
> >>> 10
> >>> > for
> >>> > > 1
> >>> > > > > > > topic(s)
> >>> > > > > > > > >> > Set(Test)
> >>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
> >>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket
> with
> >>> > > > > SO_TIMEOUT =
> >>> > > > > > > > 10000
> >>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested
> -1),
> >>> > > > SO_SNDBUF
> >>> > > > > =
> >>> > > > > > > > 102400
> >>> > > > > > > > >> > (requested 102400).
> >>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> >>> > > > 172.31.16.31:9092
> >>> > > > > > for
> >>> > > > > > > > >> > producing
> >>> > > > > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> >>> > > > > > > 172.31.16.31:9092
> >>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> >>> > metadata
> >>> > > > for
> >>> > > > > 1
> >>> > > > > > > > >> topic(s)
> >>> > > > > > > > >> > Set(Test)
> >>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
> >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> >>> > fetching
> >>> > > > > > metadata
> >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> >>> > > > > > > > >> > No partition metadata for topic Test due to
> >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> >>> > [Test]:
> >>> > > > > class
> >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> >>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting
> broker
> >>> > > > partition
> >>> > > > > > info
> >>> > > > > > > > for
> >>> > > > > > > > >> > topic Test
> >>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
> >>> > broker
> >>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id
> >>> 11
> >>> > for
> >>> > > 1
> >>> > > > > > > topic(s)
> >>> > > > > > > > >> > Set(Test)
> >>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
> >>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket
> with
> >>> > > > > SO_TIMEOUT =
> >>> > > > > > > > 10000
> >>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested
> -1),
> >>> > > > SO_SNDBUF
> >>> > > > > =
> >>> > > > > > > > 102400
> >>> > > > > > > > >> > (requested 102400).
> >>> > > > > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> >>> > > > 172.31.16.31:9092
> >>> > > > > > for
> >>> > > > > > > > >> > producing
> >>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> >>> > > > > > > 172.31.16.31:9092
> >>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
> >>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> >>> > metadata
> >>> > > > for
> >>> > > > > 1
> >>> > > > > > > > >> topic(s)
> >>> > > > > > > > >> > Set(Test)
> >>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
> >>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> >>> > fetching
> >>> > > > > > metadata
> >>> > > > > > > > >> > [{TopicMetadata for topic Test ->
> >>> > > > > > > > >> > No partition metadata for topic Test due to
> >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> >>> > [Test]:
> >>> > > > > class
> >>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> >>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
> >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed
> to
> >>> > > collate
> >>> > > > > > > > messages
> >>> > > > > > > > >> by
> >>> > > > > > > > >> > topic, partition due to: Failed to fetch topic
> >>> metadata
> >>> > for
> >>> > > > > topic:
> >>> > > > > > > > Test
> >>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
> >>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back
> off
> >>> for
> >>> > > 1000
> >>> > > > > ms
> >>> > > > > > > > before
> >>> > > > > > > > >> > retrying send. Remaining retries = 5
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > As for /tmp/kafka-logs, I can see only these files -
> >>> > > > > > > > >> > recovery-point-offset-checkpoint
> >>> > > > replication-offset-checkpoint
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > I do not see the usual Kafka log files here.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > thanks
> >>> > > > > > > > >> > Sameer
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
> >>> > > > > > wangguoz@gmail.com>
> >>> > > > > > > > >> wrote:
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > > Sameer, could you check broker logs and see if it
> >>> has
> >>> > > > similar
> >>> > > > > > > > >> > > LeaderNotAvailable exceptions?
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > Guozhang
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
> >>> > > > > syami@ailive.net
> >>> > > > > > >
> >>> > > > > > > > >> wrote:
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > > Yes. I can telnet from outside using the public
> >>> ip of
> >>> > > the
> >>> > > > > > > instance
> >>> > > > > > > > >> and
> >>> > > > > > > > >> > > port
> >>> > > > > > > > >> > > > of 9092
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <
> >>> > > > > > > > >> roger.hoover@gmail.com
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > > wrote:
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > > Just a guess but could it be a firewall issue?
> >>> Did
> >>> > > you
> >>> > > > > > enable
> >>> > > > > > > > >> > > > connections
> >>> > > > > > > > >> > > > > to port 9092 from outside EC2 in a security
> >>> group?
> >>> > > Can
> >>> > > > > you
> >>> > > > > > > > >> telnet to
> >>> > > > > > > > >> > > > each
> >>> > > > > > > > >> > > > > broker IP and port?
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami
> <
> >>> > > > > > > syami@ailive.net
> >>> > > > > > > > >
> >>> > > > > > > > >> > > wrote:
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > > There was a typo earlier.
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > This is the output -
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > Topic:Test PartitionCount:2
> >>> ReplicationFactor:1
> >>> > > > > > > Configs:
> >>> > > > > > > > >> > > > > > Topic: Test Partition: 0
> >>> Leader: 0
> >>> > > > > > > Replicas:
> >>> > > > > > > > 0
> >>> > > > > > > > >> > > > Isr: 0
> >>> > > > > > > > >> > > > > > Topic: Test Partition: 1
> >>> Leader: 0
> >>> > > > > > > Replicas:
> >>> > > > > > > > 0
> >>> > > > > > > > >> > > > Isr: 0
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > thanks
> >>> > > > > > > > >> > > > > > Sameer
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang
> >>> Wang <
> >>> > > > > > > > >> wangguoz@gmail.com
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > wrote:
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > > Sameer,
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > The topic name is not consistent from your
> >>> > > describe
> >>> > > > > > > outputs.
> >>> > > > > > > > >> > > Normally
> >>> > > > > > > > >> > > > > > they
> >>> > > > > > > > >> > > > > > > should be either
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > Topic:*Test* PartitionCount:2
> >>> > > > > ReplicationFactor:1
> >>> > > > > > > > >> > > Configs:
> >>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 0
> >>> > Leader: 0
> >>> > > > > > > > >> Replicas: 0
> >>> > > > > > > > >> > > > > > Isr: 0
> >>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 1
> >>> > Leader: 0
> >>> > > > > > > > >> Replicas: 0
> >>> > > > > > > > >> > > > > > Isr: 0
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > Or
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
> >>> > > > > > > > ReplicationFactor:1
> >>> > > > > > > > >> > > > > Configs:
> >>> > > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 0
> >>> > > > Leader:
> >>> > > > > 0
> >>> > > > > > > > >> > > Replicas:
> >>> > > > > > > > >> > > > 0
> >>> > > > > > > > >> > > > > > > Isr: 0
> >>> > > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 1
> >>> > > > Leader:
> >>> > > > > 0
> >>> > > > > > > > >> > > Replicas:
> >>> > > > > > > > >> > > > 0
> >>> > > > > > > > >> > > > > > > Isr: 0
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > Did you manually modify the topic name in
> >>> ZK?
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > Guozhang
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer
> >>> Yami <
> >>> > > > > > > > >> syami@ailive.net>
> >>> > > > > > > > >> > > > wrote:
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > > Hi Guozhang,
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > > I was wondering if there is anything
> >>> wrong in
> >>> > > the
> >>> > > > > > above
> >>> > > > > > > > >> > > --describe
> >>> > > > > > > > >> > > > > > data.
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > > thanks
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer
> >>> Yami <
> >>> > > > > > > > >> syami@ailive.net
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > wrote:
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > > > This is what I get -
> >>> > > > > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper
> >>> > > localhost:2181
> >>> > > > > > > > --describe
> >>> > > > > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
> >>> > > > > > > > >> ReplicationFactor:1
> >>> > > > > > > > >> > > > > > Configs:
> >>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 0
> >>> > > Leader: 0
> >>> > > > > > > > >> > Replicas: 0
> >>> > > > > > > > >> > > > > > > Isr: 0
> >>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 1
> >>> > > Leader: 0
> >>> > > > > > > > >> > Replicas: 0
> >>> > > > > > > > >> > > > > > > Isr: 0
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > The topics are being created at
> >>> run-time.
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > thanks
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM,
> >>> Guozhang
> >>> > > Wang <
> >>> > > > > > > > >> > > > wangguoz@gmail.com
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > >> Could you use the topic tools
> >>> "--describe"
> >>> > to
> >>> > > > > check
> >>> > > > > > > if
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > topic
> >>> > > > > > > > >> > > > > > > Test's
> >>> > > > > > > > >> > > > > > > > >> partitions leaders are available?
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > >
> >>> > > > > http://kafka.apache.org/documentation.html#basic_ops_add_topic
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > > > > > > > >> Guozhang
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM,
> Sameer
> >>> > Yami
> >>> > > <
> >>> > > > > > > > >> > > syami@ailive.net>
> >>> > > > > > > > >> > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > > > > > > > >> > That is not the case.
> >>> > > > > > > > >> > > > > > > > >> > This detailed error is below. Due
> to
> >>> this
> >>> > > > > error ,
> >>> > > > > > > the
> >>> > > > > > > > >> > > producer
> >>> > > > > > > > >> > > > > is
> >>> > > > > > > > >> > > > > > > not
> >>> > > > > > > > >> > > > > > > > >> able
> >>> > > > > > > > >> > > > > > > > >> > to send messages
> >>> > > > > > > > >> > > > > > > > >> >
> >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG
> >>> > > > > > > > >> > > > > > > > >>
> kafka.producer.BrokerPartitionInfo-52:
> >>> > > > > > > > >> > > > > > > > >> > Getting broker partition info for
> >>> topic
> >>> > > Test
> >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main INFO
> >>> > > > > > > > >> > > > > kafka.client.ClientUtils$-68:
> >>> > > > > > > > >> > > > > > > > >> Fetching
> >>> > > > > > > > >> > > > > > > > >> > metadata from broker
> >>> > > > > > > > id:0,host:54.215.200.61,port:9092
> >>> > > > > > > > >> > with
> >>> > > > > > > > >> > > > > > > > correlation
> >>> > > > > > > > >> > > > > > > > >> id
> >>> > > > > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
> >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG
> >>> > > > > > > > >> > > > > > > kafka.network.BlockingChannel-52:
> >>> > > > > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT =
> >>> 10000
> >>> > > > > (requested
> >>> > > > > > > > >> 10000),
> >>> > > > > > > > >> > > > > > SO_RCVBUF
> >>> > > > > > > > >> > > > > > > =
> >>> > > > > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF =
> >>> 102400
> >>> > > > > > > (requested
> >>> > > > > > > > >> > > 102400).
> >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main INFO
> >>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> >>> > > > > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092
> for
> >>> > > > producing
> >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main INFO
> >>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> >>> > > > > > > > >> > > > > > > > >> > Disconnecting from
> >>> 54.215.200.61:9092
> >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG
> >>> > > > > > > > >> > > > > kafka.client.ClientUtils$-52:
> >>> > > > > > > > >> > > > > > > > >> > Successfully fetched metadata for 1
> >>> > > topic(s)
> >>> > > > > > > > Set(Test)
> >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main WARN
> >>> > > > > > > > >> > > > > > > > >>
> kafka.producer.BrokerPartitionInfo-83:
> >>> > > > > > > > >> > > > > > > > >> > Error while fetching metadata
> >>> > > [{TopicMetadata
> >>> > > > > for
> >>> > > > > > > > topic
> >>> > > > > > > > >> > Test
> >>> > > > > > > > >> > > > ->
> >>> > > > > > > > >> > > > > > > > >> > No partition metadata for topic
> Test
> >>> due
> >>> > to
> >>> > > > > > > > >> > > > > > > > >> >
> >>> > kafka.common.LeaderNotAvailableException}]
> >>> > > > for
> >>> > > > > > > topic
> >>> > > > > > > > >> > [Test]:
> >>> > > > > > > > >> > > > > class
> >>> > > > > > > > >> > > > > > > > >> >
> >>> kafka.common.LeaderNotAvailableException
> >>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR
> >>> > > > > > > > >> > > > > > > > >> >
> >>> > > kafka.producer.async.DefaultEventHandler-97:
> >>> > > > > > Failed
> >>> > > > > > > > to
> >>> > > > > > > > >> > > collate
> >>> > > > > > > > >> > > > > > > > messages
> >>> > > > > > > > >> > > > > > > > >> by
> >>> > > > > > > > >> > > > > > > > >> > topic, partition due to: Failed to
> >>> fetch
> >>> > > > topic
> >>> > > > > > > > metadata
> >>> > > > > > > > >> > for
> >>> > > > > > > > >> > > > > topic:
> >>> > > > > > > > >> > > > > > > > Test
> >>> > > > > > > > >> > > > > > > > >> >
> >>> > > > > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM,
> >>> Guozhang
> >>> > > > Wang
> >>> > > > > <
> >>> > > > > > > > >> > > > > > wangguoz@gmail.com>
> >>> > > > > > > > >> > > > > > > > >> wrote:
> >>> > > > > > > > >> > > > > > > > >> >
> >>> > > > > > > > >> > > > > > > > >> > > Could you see if this is the
> case:
> >>> > > > > > > > >> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > > > > >> >
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> >
> >>> > > > > > > > >>
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> >>> > > > > > > > >> > > > > > > > >> > > ?
> >>> > > > > > > > >> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM,
> >>> > Sameer
> >>> > > > > Yami <
> >>> > > > > > > > >> > > > > syami@ailive.net
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > > >> wrote:
> >>> > > > > > > > >> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > > > > >> > > > Hi,
> >>> > > > > > > > >> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper
> test
> >>> > setup
> >>> > > > on a
> >>> > > > > > > > single
> >>> > > > > > > > >> EC2
> >>> > > > > > > > >> > > > > > machine.
> >>> > > > > > > > >> > > > > > > > The
> >>> > > > > > > > >> > > > > > > > >> > > > host.name is the private ip
> >>> address
> >>> > > and
> >>> > > > > the
> >>> > > > > > > > >> > > > > > > advertised.host.name
> >>> > > > > > > > >> > > > > > > > is
> >>> > > > > > > > >> > > > > > > > >> > the
> >>> > > > > > > > >> > > > > > > > >> > > > public address in
> >>> server.properties.
> >>> > > > > > > > >> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > > > > > >> > > > We are trying to publish from
> >>> outside
> >>> > > EC2
> >>> > > > > > using
> >>> > > > > > > > the
> >>> > > > > > > > >> > > public
> >>> > > > > > > > >> > > > > ip
> >>> > > > > > > > >> > > > > > of
> >>> > > > > > > > >> > > > > > > > the
> >>> > > > > > > > >> > > > > > > > >> > EC2
> >>> > > > > > > > >> > > > > > > > >> > > > machine but get this error on
> the
> >>> > > client
> >>> > > > > > side -
> >>> > > > > > > > >> > > > > > > > >> > > > No partition metadata for topic
> >>> test
> >>> > > due
> >>> > > > to
> >>> > > > > > > > >> > > > > > > > >> > > >
> >>> > > > kafka.common.LeaderNotAvailableException}]
> >>> > > > > > for
> >>> > > > > > > > >> topic
> >>> > > > > > > > >> > > > [test]:
> >>> > > > > > > > >> > > > > > > class
> >>> > > > > > > > >> > > > > > > > >> > > >
> >>> > > kafka.common.LeaderNotAvailableException
> >>> > > > > > > > >> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > > > > > >> > > > The zookeeper has the public ip
> >>> > stored
> >>> > > in
> >>> > > > > > > > >> > > /brokers/ids/0.
> >>> > > > > > > > >> > > > > > > > >> > > > We have tried multiple
> >>> combinations
> >>> > but
> >>> > > > it
> >>> > > > > > > works
> >>> > > > > > > > >> ONLY
> >>> > > > > > > > >> > > when
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > > >> > advertised
> >>> > > > > > > > >> > > > > > > > >> > > > host name is not used and the
> >>> client
> >>> > is
> >>> > > > > > > internal
> >>> > > > > > > > to
> >>> > > > > > > > >> > EC2.
> >>> > > > > > > > >> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > > > > > >> > > > Please let me know what the
> >>> problem
> >>> > > could
> >>> > > > > be.
> >>> > > > > > > > >> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > > > > > >> > > > thanks
> >>> > > > > > > > >> > > > > > > > >> > > > Sameer
> >>> > > > > > > > >> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > > > > >> > > --
> >>> > > > > > > > >> > > > > > > > >> > > -- Guozhang
> >>> > > > > > > > >> > > > > > > > >> > >
> >>> > > > > > > > >> > > > > > > > >> >
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > > > > > > > >> --
> >>> > > > > > > > >> > > > > > > > >> -- Guozhang
> >>> > > > > > > > >> > > > > > > > >>
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > --
> >>> > > > > > > > >> > > > > > > -- Guozhang
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > --
> >>> > > > > > > > >> > > -- Guozhang
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> >
> >>> > > > > > > > >>
> >>> > > > > > > > >>
> >>> > > > > > > > >>
> >>> > > > > > > > >> --
> >>> > > > > > > > >> -- Guozhang
> >>> > > > > > > > >>
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > --
> >>> > > > > > > -- Guozhang
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > --
> >>> > > > > -- Guozhang
> >>> > > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > -- Guozhang
> >>> > >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >>
> >
>
--
-- Guozhang
Re: Cannot connect to Kafka from outside of EC2
Posted by Sameer Yami <sy...@ailive.net>.
Hi Guozhang,
I was wondering if you found anything wrong in the logs/
thanks
On Fri, Nov 7, 2014 at 4:19 PM, Sameer Yami <sy...@ailive.net> wrote:
> Hi Guozhang,
>
> Attached are the two logs with debug enabled.
>
> Thanks!
>
> On Fri, Nov 7, 2014 at 2:09 PM, Sameer Yami <sy...@ailive.net> wrote:
>
>> The version is kafka_2.10-0.8.1.1. It is not the latest trunk.
>> Will try enabling debug version.
>>
>> thanks
>>
>>
>> On Thu, Nov 6, 2014 at 9:37 PM, Guozhang Wang <wa...@gmail.com> wrote:
>>
>>> Sameer,
>>>
>>> The server logs do not contain any non-INFO logs, which is a bit wired.
>>> Did
>>> you deploy the current trunk of Kafka? Also could you enable DEBUG level
>>> logging on Kafka brokers?
>>>
>>> Guozhang
>>>
>>> On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami <sy...@ailive.net> wrote:
>>>
>>> > The server.log was taken separately.
>>> > We ran the test again and the server and producer logs are below (to
>>> get
>>> > same timings).
>>> >
>>> >
>>> > Thanks!
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------------------
>>> >
>>> >
>>> >
>>> > Producer Logs -
>>> >
>>> >
>>> > 2014-11-05 23:38:58,693
>>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>>> sessionid:
>>> > 0x1498251e8680002 after 0ms
>>> > 2014-11-05 23:39:00,695
>>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>>> sessionid:
>>> > 0x1498251e8680002 after 0ms
>>> > 2014-11-05 23:39:02,696
>>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>>> sessionid:
>>> > 0x1498251e8680002 after 0ms
>>> > 2014-11-05 23:39:02,828 pool-13-thread-2 INFO
>>> > kafka.utils.VerifiableProperties-68: Verifying properties
>>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
>>> > kafka.utils.VerifiableProperties-68: Property auto.commit.interval.ms
>>> is
>>> > overridden to 1000
>>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
>>> > kafka.utils.VerifiableProperties-68: Property auto.offset.reset is
>>> > overridden to smallest
>>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
>>> > kafka.utils.VerifiableProperties-68: Property consumer.timeout.ms is
>>> > overridden to 10
>>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
>>> > kafka.utils.VerifiableProperties-68: Property group.id is overridden
>>> to
>>> > TestCheck
>>> > 2014-11-05 23:39:02,830 pool-13-thread-2 WARN
>>> > kafka.utils.VerifiableProperties-83: Property serializer.class is not
>>> valid
>>> > 2014-11-05 23:39:02,830 pool-13-thread-2 INFO
>>> > kafka.utils.VerifiableProperties-68: Property zookeeper.connect is
>>> > overridden to 172.31.25.198:2181
>>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Connecting to
>>> > zookeeper instance at 172.31.25.198:2181
>>> > 2014-11-05 23:39:02,831 pool-13-thread-2 DEBUG
>>> > org.I0Itec.zkclient.ZkConnection-63: Creating new ZookKeeper instance
>>> to
>>> > connect to 172.31.25.198:2181.
>>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
>>> > org.apache.zookeeper.ZooKeeper-379: Initiating client connection,
>>> > connectString=172.31.25.198:2181 sessionTimeout=6000
>>> > watcher=org.I0Itec.zkclient.ZkClient@3903b165
>>> > 2014-11-05 23:39:02,831 ZkClient-EventThread-29-172.31.25.198:2181
>>> INFO
>>> > org.I0Itec.zkclient.ZkEventThread-64: Starting ZkClient event thread.
>>> > 2014-11-05 23:39:02,831 pool-13-thread-1 INFO
>>> > kafka.utils.VerifiableProperties-68: Verifying properties
>>> > 2014-11-05 23:39:02,836 pool-13-thread-2-SendThread() INFO
>>> > org.apache.zookeeper.ClientCnxn-1061: Opening socket connection to
>>> server /
>>> > 172.31.25.198:2181
>>> > 2014-11-05 23:39:02,836 pool-13-thread-1 WARN
>>> > kafka.utils.VerifiableProperties-83: Property batch.size is not valid
>>> > 2014-11-05 23:39:02,832 pool-13-thread-2 DEBUG
>>> > org.I0Itec.zkclient.ZkClient-878: Awaiting connection to Zookeeper
>>> server
>>> > 2014-11-05 23:39:02,836 pool-13-thread-1 INFO
>>> > kafka.utils.VerifiableProperties-68: Property message.send.max.retries
>>> is
>>> > overridden to 10
>>> > 2014-11-05 23:39:02,836 pool-13-thread-2 DEBUG
>>> > org.I0Itec.zkclient.ZkClient-628: Waiting for keeper state
>>> SyncConnected
>>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
>>> > kafka.utils.VerifiableProperties-68: Property metadata.broker.list is
>>> > overridden to 172.31.25.198:9092
>>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
>>> > kafka.utils.VerifiableProperties-68: Property retry.backoff.ms is
>>> > overridden to 1000
>>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
>>> > kafka.utils.VerifiableProperties-68: Property serializer.class is
>>> > overridden to kafka.serializer.StringEncoder
>>> > 2014-11-05 23:39:02,837
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > INFO org.apache.zookeeper.ClientCnxn-950: Socket connection
>>> established to
>>> > ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
>>> initiating
>>> > session
>>> > 2014-11-05 23:39:02,838
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-999: Session establishment
>>> request
>>> > sent on ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181
>>> > 2014-11-05 23:39:02,837 pool-13-thread-1 WARN
>>> > kafka.utils.VerifiableProperties-83: Property zk.connectiontimeout.ms
>>> is
>>> > not valid
>>> > 2014-11-05 23:39:02,841
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > INFO org.apache.zookeeper.ClientCnxn-739: Session establishment
>>> complete on
>>> > server ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
>>> > sessionid = 0x1498251e8680003, negotiated timeout = 6000
>>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
>>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
>>> > state:SyncConnected type:None path:null
>>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread INFO
>>> > org.I0Itec.zkclient.ZkClient-449: zookeeper state changed
>>> (SyncConnected)
>>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
>>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
>>> > 2014-11-05 23:39:02,841 pool-13-thread-2 DEBUG
>>> > org.I0Itec.zkclient.ZkClient-638: State is SyncConnected
>>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
>>> > kafka.utils.KafkaScheduler-52: Initializing task scheduler.
>>> > 2014-11-05 23:39:02,842 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting auto
>>> > committer every 1000 ms
>>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
>>> > kafka.utils.KafkaScheduler-52: Scheduling task
>>> kafka-consumer-autocommit
>>> > with initial delay 1000 ms and period 1000 ms.
>>> > 2014-11-05 23:39:02,843 pool-13-thread-2 DEBUG
>>> > kafka.consumer.ZookeeperConsumerConnector-52:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], entering consume
>>> > 2014-11-05 23:39:02,843 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin registering
>>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
>>> > 2014-11-05 23:39:02,846
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 1,1 replyHeader:: 1,29,-101 request::
>>> >
>>> >
>>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
>>> > response::
>>> > 2014-11-05 23:39:02,854
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 2,1 replyHeader:: 2,30,-101 request::
>>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
>>> > 2014-11-05 23:39:02,858
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 3,1 replyHeader:: 3,31,0 request::
>>> > '/consumers/TestCheck,,v{s{31,s{'world,'anyone}}},0 response::
>>> > '/consumers/TestCheck
>>> > 2014-11-05 23:39:02,861
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 4,1 replyHeader:: 4,32,0 request::
>>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
>>> > '/consumers/TestCheck/ids
>>> > 2014-11-05 23:39:02,866
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 5,1 replyHeader:: 5,33,0 request::
>>> >
>>> >
>>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
>>> > response::
>>> >
>>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
>>> > 2014-11-05 23:39:02,867 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end registering
>>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
>>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
>>> > kafka.consumer.ZookeeperConsumerConnector-52:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Adding
>>> topicThreadId
>>> > (Test,TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0) and queue
>>> [] to
>>> > topicThreadIdAndQueues data structure
>>> > 2014-11-05 23:39:02,868
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting watcher
>>> > executor thread for consumer
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
>>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
>>> > kafka.consumer.ZookeeperConsumerConnector-52:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], adding topic Test
>>> and
>>> > 1 streams to map.
>>> > 2014-11-05 23:39:02,870
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 6,3 replyHeader:: 6,33,0 request::
>>> > '/consumers/TestCheck/ids,T response::
>>> > s{32,32,1415230742859,1415230742859,0,1,0,0,0,1,33}
>>> > 2014-11-05 23:39:02,872
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 7,8 replyHeader:: 7,33,0 request::
>>> > '/consumers/TestCheck/ids,T response::
>>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
>>> > 2014-11-05 23:39:02,873
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 8,3 replyHeader:: 8,33,-101 request::
>>> > '/brokers/topics/Test,T response::
>>> > 2014-11-05 23:39:02,873 pool-13-thread-2 DEBUG
>>> > org.I0Itec.zkclient.ZkClient-137: Subscribed data changes for
>>> > /brokers/topics/Test
>>> > 2014-11-05 23:39:02,873 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
>>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
>>> > 2014-11-05 23:39:02,875 pool-13-thread-1 DEBUG
>>> > kafka.producer.async.DefaultEventHandler-52: Handling 1 events
>>> > 2014-11-05 23:39:02,877
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 9,8 replyHeader:: 9,33,0 request::
>>> > '/brokers/ids,F response:: v{'0}
>>> > 2014-11-05 23:39:02,879
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 10,4 replyHeader:: 10,33,0 request::
>>> > '/brokers/ids/0,F response::
>>> >
>>> >
>>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>>> >
>>> > 2014-11-05 23:39:02,887
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 11,4 replyHeader:: 11,33,0 request::
>>> >
>>> >
>>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
>>> > response::
>>> >
>>> >
>>> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>>> >
>>> > 2014-11-05 23:39:02,896
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 12,8 replyHeader:: 12,33,0 request::
>>> > '/consumers/TestCheck/ids,T response::
>>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
>>> > 2014-11-05 23:39:02,898
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 13,4 replyHeader:: 13,33,0 request::
>>> >
>>> >
>>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
>>> > response::
>>> >
>>> >
>>> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>>> >
>>> > 2014-11-05 23:39:02,908
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 14,8 replyHeader:: 14,33,0 request::
>>> > '/brokers/ids,F response:: v{'0}
>>> > 2014-11-05 23:39:02,910
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 15,4 replyHeader:: 15,33,0 request::
>>> > '/brokers/ids/0,F response::
>>> >
>>> >
>>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>>> >
>>> > 2014-11-05 23:39:02,917 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 0 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:02,917
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 16,4 replyHeader:: 16,33,-101 request::
>>> > '/brokers/topics/Test,T response::
>>> > 2014-11-05 23:39:02,920 pool-13-thread-2 DEBUG
>>> kafka.utils.ZkUtils$-52:
>>> > Partition map for /brokers/topics/Test is Map()
>>> > 2014-11-05 23:39:02,920 pool-13-thread-2 INFO
>>> > kafka.consumer.ConsumerFetcherManager-68:
>>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
>>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
>>> > kafka.consumer.ConsumerFetcherManager-68:
>>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
>>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
>>> > kafka.consumer.ConsumerFetcherManager-68:
>>> > [ConsumerFetcherManager-1415230742842] All connections stopped
>>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
>>> relevant
>>> > queues for this fetcher
>>> > 2014-11-05 23:39:02,921 pool-13-thread-2 DEBUG
>>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
>>> > this consumer iterator
>>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
>>> > chunks in all the consumer message iterators
>>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
>>> offsets
>>> > after clearing the fetcher queues
>>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing
>>> partition
>>> > ownership
>>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
>>> following
>>> > partitions: ArrayBuffer() for topic Test with consumers:
>>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
>>> > 2014-11-05 23:39:02,922 pool-13-thread-2 WARN
>>> > kafka.consumer.ZookeeperConsumerConnector-83:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], No broker
>>> partitions
>>> > consumed by consumer thread
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 for topic Test
>>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
>>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
>>> > kafka.consumer.ZookeeperConsumerConnector-52:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
>>> topic
>>> > cache Map(Test -> ArrayBuffer())
>>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
>>> > kafka.consumer.ZookeeperConsumerConnector-52:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per
>>> topic
>>> > cache Map(Test ->
>>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
>>> > 2014-11-05 23:39:02,923 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
>>> > 2014-11-05 23:39:02,927 pool-13-thread-2 INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
>>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
>>> > 2014-11-05 23:39:02,928
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>>> >
>>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>>> > Starting
>>> > 2014-11-05 23:39:02,930 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:02,930 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:02,985
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-783: Got notification
>>> > sessionid:0x1498251e8680003
>>> > 2014-11-05 23:39:02,986
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-800: Got WatchedEvent
>>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test for
>>> > sessionid 0x1498251e8680003
>>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
>>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
>>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test
>>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
>>> > org.I0Itec.zkclient.ZkEventThread-88: New event: ZkEvent[Data of
>>> > /brokers/topics/Test changed sent to
>>> >
>>> >
>>> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
>>> > ]
>>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
>>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
>>> > 2014-11-05 23:39:02,988 ZkClient-EventThread-29-172.31.25.198:2181
>>> DEBUG
>>> > org.I0Itec.zkclient.ZkEventThread-69: Delivering event #1 ZkEvent[Data
>>> of
>>> > /brokers/topics/Test changed sent to
>>> >
>>> >
>>> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
>>> > ]
>>> > 2014-11-05 23:39:02,991
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 17,3 replyHeader:: 17,37,0 request::
>>> > '/brokers/topics/Test,T response::
>>> > s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
>>> > 2014-11-05 23:39:02,993
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 18,4 replyHeader:: 18,37,0 request::
>>> > '/brokers/topics/Test,T response::
>>> >
>>> >
>>> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
>>> >
>>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Topic info for
>>> path
>>> > /brokers/topics/Test changed to
>>> > {"version":1,"partitions":{"1":[0],"0":[0]}}, triggering rebalance
>>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
>>> DEBUG
>>> > org.I0Itec.zkclient.ZkEventThread-79: Delivering event #1 done
>>> > 2014-11-05 23:39:02,995
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
>>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
>>> > 2014-11-05 23:39:02,996
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 19,8 replyHeader:: 19,37,0 request::
>>> > '/brokers/ids,F response:: v{'0}
>>> > 2014-11-05 23:39:02,998
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 20,4 replyHeader:: 20,37,0 request::
>>> > '/brokers/ids/0,F response::
>>> >
>>> >
>>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>>> >
>>> > 2014-11-05 23:39:03,008
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 21,4 replyHeader:: 21,37,0 request::
>>> >
>>> >
>>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
>>> > response::
>>> >
>>> >
>>> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>>> >
>>> > 2014-11-05 23:39:03,014
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 22,8 replyHeader:: 22,37,0 request::
>>> > '/consumers/TestCheck/ids,T response::
>>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
>>> > 2014-11-05 23:39:03,018
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 23,4 replyHeader:: 23,37,0 request::
>>> >
>>> >
>>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
>>> > response::
>>> >
>>> >
>>> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>>> >
>>> > 2014-11-05 23:39:03,041
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 24,8 replyHeader:: 24,37,0 request::
>>> > '/brokers/ids,F response:: v{'0}
>>> > 2014-11-05 23:39:03,043
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 25,4 replyHeader:: 25,37,0 request::
>>> > '/brokers/ids/0,F response::
>>> >
>>> >
>>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>>> >
>>> > 2014-11-05 23:39:03,058
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 26,4 replyHeader:: 26,37,0 request::
>>> > '/brokers/topics/Test,T response::
>>> >
>>> >
>>> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
>>> >
>>> > 2014-11-05 23:39:03,074
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.utils.ZkUtils$-52: Partition map for /brokers/topics/Test is
>>> Map(1 ->
>>> > List(0), 0 -> List(0))
>>> > 2014-11-05 23:39:03,075
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ConsumerFetcherManager-68:
>>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
>>> > 2014-11-05 23:39:03,082
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>>> >
>>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>>> > Shutting down
>>> > 2014-11-05 23:39:03,082 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:03,089
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>>> >
>>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>>> > Stopped
>>> > 2014-11-05 23:39:03,089
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>>> >
>>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>>> > Shutdown completed
>>> > 2014-11-05 23:39:03,089
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ConsumerFetcherManager-68:
>>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
>>> > 2014-11-05 23:39:03,090
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ConsumerFetcherManager-68:
>>> > [ConsumerFetcherManager-1415230742842] All connections stopped
>>> > 2014-11-05 23:39:03,090
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
>>> relevant
>>> > queues for this fetcher
>>> > 2014-11-05 23:39:03,090
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
>>> > this consumer iterator
>>> > 2014-11-05 23:39:03,091
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
>>> > chunks in all the consumer message iterators
>>> > 2014-11-05 23:39:03,091
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
>>> offsets
>>> > after clearing the fetcher queues
>>> > 2014-11-05 23:39:03,091
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing
>>> partition
>>> > ownership
>>> > 2014-11-05 23:39:03,094 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:03,096
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
>>> following
>>> > partitions: ArrayBuffer(0, 1) for topic Test with consumers:
>>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
>>> > 2014-11-05 23:39:03,098
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
>>> > partition 0
>>> > 2014-11-05 23:39:03,100
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 27,4 replyHeader:: 27,37,-101 request::
>>> > '/consumers/TestCheck/offsets/Test/0,F response::
>>> > 2014-11-05 23:39:03,116
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of
>>> Test:0:
>>> > fetched offset = -1: consumed offset = -1 is -1
>>> > 2014-11-05 23:39:03,117
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:0:
>>> > fetched offset = -1: consumed offset = -1 is -1
>>> > 2014-11-05 23:39:03,118
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.consumer.ZookeeperConsumerConnector-52:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:0: fetched
>>> offset
>>> > = -1: consumed offset = -1 selected new offset -1
>>> > 2014-11-05 23:39:03,169
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
>>> > partition 1
>>> > 2014-11-05 23:39:03,171
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 28,4 replyHeader:: 28,37,-101 request::
>>> > '/consumers/TestCheck/offsets/Test/1,F response::
>>> > 2014-11-05 23:39:03,172
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of
>>> Test:1:
>>> > fetched offset = -1: consumed offset = -1 is -1
>>> > 2014-11-05 23:39:03,172
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:1:
>>> > fetched offset = -1: consumed offset = -1 is -1
>>> > 2014-11-05 23:39:03,172
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.consumer.ZookeeperConsumerConnector-52:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:1: fetched
>>> offset
>>> > = -1: consumed offset = -1 selected new offset -1
>>> > 2014-11-05 23:39:03,175 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:03,175
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 29,1 replyHeader:: 29,38,-101 request::
>>> >
>>> >
>>> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
>>> > response::
>>> > 2014-11-05 23:39:03,179
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 30,1 replyHeader:: 30,39,-101 request::
>>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
>>> > response::
>>> > 2014-11-05 23:39:03,180 pool-13-thread-1 DEBUG
>>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
>>> for
>>> > topic Test
>>> > 2014-11-05 23:39:03,181 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 1 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:03,182 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:03,182 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:03,186
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 31,1 replyHeader:: 31,40,0 request::
>>> > '/consumers/TestCheck/owners,,v{s{31,s{'world,'anyone}}},0 response::
>>> > '/consumers/TestCheck/owners
>>> > 2014-11-05 23:39:03,189 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:03,190
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 32,1 replyHeader:: 32,41,0 request::
>>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
>>> response::
>>> > '/consumers/TestCheck/owners/Test
>>> > 2014-11-05 23:39:03,190 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:03,191 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:03,192 pool-13-thread-1 ERROR
>>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
>>> messages by
>>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>>> > 2014-11-05 23:39:03,193
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 33,1 replyHeader:: 33,42,0 request::
>>> >
>>> >
>>> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
>>> > response:: '/consumers/TestCheck/owners/Test/1
>>> > 2014-11-05 23:39:03,193 pool-13-thread-1 INFO
>>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
>>> before
>>> > retrying send. Remaining retries = 10
>>> > 2014-11-05 23:39:03,193
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
>>> > partition 1 for topic Test
>>> > 2014-11-05 23:39:03,196
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 34,1 replyHeader:: 34,43,0 request::
>>> >
>>> >
>>> '/consumers/TestCheck/owners/Test/0,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
>>> > response:: '/consumers/TestCheck/owners/Test/0
>>> > 2014-11-05 23:39:03,196
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
>>> > partition 0 for topic Test
>>> > 2014-11-05 23:39:03,196
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
>>> > 2014-11-05 23:39:03,197
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.consumer.ZookeeperConsumerConnector-52:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
>>> topic
>>> > cache Map(Test -> ArrayBuffer(0, 1))
>>> > 2014-11-05 23:39:03,197
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> DEBUG
>>> > kafka.consumer.ZookeeperConsumerConnector-52:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per
>>> topic
>>> > cache Map(Test ->
>>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
>>> > 2014-11-05 23:39:03,197
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
>>> > Test:0: fetched offset = -1: consumed offset = -1,Test:1: fetched
>>> offset =
>>> > -1: consumed offset = -1
>>> > 2014-11-05 23:39:03,199
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>>> >
>>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>>> > Starting
>>> > 2014-11-05 23:39:03,199
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>>> INFO
>>> > kafka.consumer.ZookeeperConsumerConnector-68:
>>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
>>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
>>> > 2014-11-05 23:39:03,202
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 35,8 replyHeader:: 35,43,0 request::
>>> > '/brokers/ids,F response:: v{'0}
>>> > 2014-11-05 23:39:03,204
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>>> > finished:false header:: 36,4 replyHeader:: 36,43,0 request::
>>> > '/brokers/ids/0,F response::
>>> >
>>> >
>>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>>> >
>>> > 2014-11-05 23:39:03,214
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>>> > INFO kafka.utils.VerifiableProperties-68: Verifying properties
>>> > 2014-11-05 23:39:03,215
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>>> > INFO kafka.utils.VerifiableProperties-68: Property client.id is
>>> overridden
>>> > to TestCheck
>>> > 2014-11-05 23:39:03,215
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>>> > INFO kafka.utils.VerifiableProperties-68: Property
>>> metadata.broker.list is
>>> > overridden to x.x.x.x:9092
>>> > 2014-11-05 23:39:03,215
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>>> > INFO kafka.utils.VerifiableProperties-68: Property request.timeout.ms
>>> is
>>> > overridden to 30000
>>> > 2014-11-05 23:39:03,217
>>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>>> > INFO kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:x.x.x.x,port:9092 with correlation id 0 for 1 topic(s)
>>> Set(Test)
>>> > 2014-11-05 23:39:04,195 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 2 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:04,196 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:04,196 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:04,199 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:04,199 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:04,200 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:04,200 pool-13-thread-1 DEBUG
>>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
>>> for
>>> > topic Test
>>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 3 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:04,201 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:04,205 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:04,205 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:04,206 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:04,206 pool-13-thread-1 ERROR
>>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
>>> messages by
>>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>>> > 2014-11-05 23:39:04,206 pool-13-thread-1 INFO
>>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
>>> before
>>> > retrying send. Remaining retries = 9
>>> > 2014-11-05 23:39:04,698
>>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>>> sessionid:
>>> > 0x1498251e8680002 after 0ms
>>> > 2014-11-05 23:39:05,205
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>>> sessionid:
>>> > 0x1498251e8680003 after 0ms
>>> > 2014-11-05 23:39:05,207 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 4 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:05,208 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:05,208 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:05,211 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:05,211 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:05,212 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:05,212 pool-13-thread-1 DEBUG
>>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
>>> for
>>> > topic Test
>>> > 2014-11-05 23:39:05,212 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 5 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:05,213 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:05,213 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:05,217 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:05,217 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:05,217 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:05,217 pool-13-thread-1 ERROR
>>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
>>> messages by
>>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>>> > 2014-11-05 23:39:05,218 pool-13-thread-1 INFO
>>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
>>> before
>>> > retrying send. Remaining retries = 8
>>> > 2014-11-05 23:39:06,218 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 6 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:06,219 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:06,219 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:06,224 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:06,224 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:06,225 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:06,225 pool-13-thread-1 DEBUG
>>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
>>> for
>>> > topic Test
>>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 7 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:06,226 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:06,230 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:06,230 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:06,231 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:06,231 pool-13-thread-1 ERROR
>>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
>>> messages by
>>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>>> > 2014-11-05 23:39:06,231 pool-13-thread-1 INFO
>>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
>>> before
>>> > retrying send. Remaining retries = 7
>>> > 2014-11-05 23:39:06,699
>>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>>> sessionid:
>>> > 0x1498251e8680002 after 0ms
>>> > 2014-11-05 23:39:07,208
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>>> sessionid:
>>> > 0x1498251e8680003 after 0ms
>>> > 2014-11-05 23:39:07,232 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 8 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:07,233 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:07,233 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:07,236 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:07,236 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:07,237 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:07,237 pool-13-thread-1 DEBUG
>>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
>>> for
>>> > topic Test
>>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 9 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:07,238 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:07,241 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:07,241 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:07,242 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:07,242 pool-13-thread-1 ERROR
>>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
>>> messages by
>>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>>> > 2014-11-05 23:39:07,242 pool-13-thread-1 INFO
>>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
>>> before
>>> > retrying send. Remaining retries = 6
>>> > 2014-11-05 23:39:08,243 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 10 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:08,243 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:08,245 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:08,249 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:08,250 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:08,250 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
>>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
>>> for
>>> > topic Test
>>> > 2014-11-05 23:39:08,251 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 11 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:08,252 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:08,255 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:08,256 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:08,256 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:08,257 pool-13-thread-1 ERROR
>>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
>>> messages by
>>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>>> > 2014-11-05 23:39:08,257 pool-13-thread-1 INFO
>>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
>>> before
>>> > retrying send. Remaining retries = 5
>>> > 2014-11-05 23:39:08,702
>>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>>> sessionid:
>>> > 0x1498251e8680002 after 0ms
>>> > 2014-11-05 23:39:09,211
>>> >
>>> >
>>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>>> sessionid:
>>> > 0x1498251e8680003 after 0ms
>>> > 2014-11-05 23:39:09,257 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 12 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:09,258 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:09,258 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:09,261 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:09,262 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
>>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
>>> for
>>> > topic Test
>>> > 2014-11-05 23:39:09,263 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 13 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:09,264 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:09,264 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:09,268 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:09,268 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:09,269 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:09,269 pool-13-thread-1 ERROR
>>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
>>> messages by
>>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>>> > 2014-11-05 23:39:09,269 pool-13-thread-1 INFO
>>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
>>> before
>>> > retrying send. Remaining retries = 4
>>> > 2014-11-05 23:39:10,270 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 14 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:10,271 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:10,271 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:10,275 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:10,275 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:10,275 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:10,276 pool-13-thread-1 DEBUG
>>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info
>>> for
>>> > topic Test
>>> > 2014-11-05 23:39:10,276 pool-13-thread-1 INFO
>>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>>> > id:0,host:172.31.25.198,port:9092 with correlation id 15 for 1 topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:10,277 pool-13-thread-1 DEBUG
>>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT =
>>> 10000
>>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF =
>>> 102400
>>> > (requested 102400).
>>> > 2014-11-05 23:39:10,277 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>>> > producing
>>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
>>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>>> > 2014-11-05 23:39:10,280 pool-13-thread-1 DEBUG
>>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>>> topic(s)
>>> > Set(Test)
>>> > 2014-11-05 23:39:10,280 pool-13-thread-1 WARN
>>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>>> > [{TopicMetadata for topic Test ->
>>> > No partition metadata for topic Test due to
>>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>>> > kafka.common.LeaderNotAvailableException
>>> > 2014-11-05 23:39:10,280 pool-13-thread-1 ERROR
>>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
>>> messages by
>>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
>>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms
>>> before
>>> > retrying send. Remaining retries = 3
>>> >
>>> >
>>> > Brokers Logs -
>>> >
>>> > [2014-11-05 23:39:02,983] INFO Topic creation
>>> > {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
>>> > [2014-11-05 23:39:02,988] INFO [KafkaApi-0] Auto creation of topic Test
>>> > with 2 partitions and replication factor 1 is successful!
>>> > (kafka.server.KafkaApis)
>>> > [2014-11-05 23:39:03,092] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> > [2014-11-05 23:39:03,191] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> > [2014-11-05 23:39:04,200] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> > [2014-11-05 23:39:04,205] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> > [2014-11-05 23:39:05,211] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> > [2014-11-05 23:39:05,218] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> > [2014-11-05 23:39:06,224] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> > [2014-11-05 23:39:06,230] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> > [2014-11-05 23:39:07,236] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> > [2014-11-05 23:39:07,241] INFO Closing socket connection to /
>>> 172.31.25.198
>>> > .
>>> > (kafka.network.Processor)
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Nov 5, 2014 at 3:05 PM, Guozhang Wang <wa...@gmail.com>
>>> wrote:
>>> >
>>> > > Sameer,
>>> > >
>>> > > Yes, this is the server log. But there seems no abnormal entries in
>>> it,
>>> > and
>>> > > it does not cover the same time range as the producer client throwing
>>> > > LeaderNotAvailableException
>>> > > (it was 10/24, 14:30).
>>> > >
>>> > > The reason that I want to check the server log at that same reason is
>>> > > that LeaderNotAvailableException
>>> > > is set by the brokers while handling requests, and its server log
>>> will
>>> > show
>>> > > what is the cause it set this error code.
>>> > >
>>> > > Guozhang
>>> > >
>>> > > On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net>
>>> wrote:
>>> > >
>>> > > > Hi Guozhang,
>>> > > >
>>> > > > This is the server.log -
>>> > > >
>>> > > > [2014-11-04 20:21:57,510] INFO Verifying properties
>>> > > > (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,545] INFO Property advertised.host.name is
>>> > > overridden
>>> > > > to x.x.x.x (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,545] INFO Property broker.id is overridden
>>> to 0
>>> > > > (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,545] INFO Property controlled.shutdown.enable
>>> is
>>> > > > overridden to true (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,545] INFO Property host.name is overridden to
>>> > > > 172.31.25.198 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is
>>> > overridden
>>> > > to
>>> > > > false (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden to
>>> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,546] INFO Property
>>> > log.retention.check.interval.ms
>>> > > is
>>> > > > overridden to 60000 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,546] INFO Property log.retention.hours is
>>> > overridden
>>> > > > to 168 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is
>>> overridden
>>> > > to
>>> > > > 536870912 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,546] INFO Property num.io.threads is
>>> overridden
>>> > to 8
>>> > > > (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,547] INFO Property num.network.threads is
>>> > overridden
>>> > > > to 2 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,547] INFO Property num.partitions is
>>> overridden
>>> > to 2
>>> > > > (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,547] INFO Property port is overridden to 9092
>>> > > > (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,547] INFO Property
>>> socket.receive.buffer.bytes is
>>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,547] INFO Property socket.request.max.bytes is
>>> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,547] INFO Property socket.send.buffer.bytes is
>>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is
>>> overridden
>>> > > to
>>> > > > localhost:2181 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,548] INFO Property
>>> > zookeeper.connection.timeout.ms
>>> > > is
>>> > > > overridden to 1000000 (kafka.utils.VerifiableProperties)
>>> > > > [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
>>> > > > (kafka.server.KafkaServer)
>>> > > > [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to
>>> > zookeeper
>>> > > on
>>> > > > localhost:2181 (kafka.server.KafkaServer)
>>> > > > [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs' not
>>> > found,
>>> > > > creating it. (kafka.log.LogManager)
>>> > > > [2014-11-04 20:21:57,748] INFO Starting log cleanup with a period
>>> of
>>> > > 60000
>>> > > > ms. (kafka.log.LogManager)
>>> > > > [2014-11-04 20:21:57,752] INFO Starting log flusher with a default
>>> > period
>>> > > > of 9223372036854775807 ms. (kafka.log.LogManager)
>>> > > > [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
>>> > > > ip-172-31-25-198.us-west-1.compute.internal:9092.
>>> > > (kafka.network.Acceptor)
>>> > > > [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0], Started
>>> > > > (kafka.network.SocketServer)
>>> > > > [2014-11-04 20:21:57,852] INFO Will not load MX4J, mx4j-tools.jar
>>> is
>>> > not
>>> > > in
>>> > > > the classpath (kafka.utils.Mx4jLoader$)
>>> > > > [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
>>> > > > (kafka.server.ZookeeperLeaderElector)
>>> > > > [2014-11-04 20:21:57,967] INFO Registered broker 0 at path
>>> > /brokers/ids/0
>>> > > > with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
>>> > > > [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
>>> > > > (kafka.server.KafkaServer)
>>> > > > [2014-11-04 20:21:58,050] INFO New leader is 0
>>> > > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
>>> > > > [2014-11-04 20:22:01,809] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:22:11,812] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:29,084] INFO Topic creation
>>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
>>> (kafka.admin.AdminUtils$)
>>> > > > [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of topic
>>> Test
>>> > > > with 2 partitions and replication factor 1 is successful!
>>> > > > (kafka.server.KafkaApis)
>>> > > > [2014-11-04 20:30:29,280] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:29,301] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:30,310] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:30,315] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:31,320] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:31,325] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:32,331] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:32,336] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:33,343] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:33,347] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:34,356] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:34,360] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:35,366] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:35,372] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:36,377] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:36,382] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:37,387] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:37,394] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:38,399] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:38,403] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:39,410] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:39,414] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:40,419] INFO Closing socket connection to /
>>> > > 172.31.25.198
>>> > > > .
>>> > > > (kafka.network.Processor)
>>> > > > [2014-11-04 20:30:41,602] INFO Topic creation
>>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
>>> (kafka.admin.AdminUtils$)
>>> > > >
>>> > > >
>>> > > >
>>> > > > thanks
>>> > > >
>>> > > >
>>> > > > On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <wangguoz@gmail.com
>>> >
>>> > > wrote:
>>> > > >
>>> > > > > The server.log should be the broker logs.
>>> > > > >
>>> > > > > Guozhang
>>> > > > >
>>> > > > > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <sy...@ailive.net>
>>> > > wrote:
>>> > > > >
>>> > > > > > I do not see any other logs besides these.
>>> > > > > > I checked the log4j.properties and where the logs are getting
>>> > stored.
>>> > > > > > The only other logs (apart from the controller logs) in the
>>> > directory
>>> > > > are
>>> > > > > > zookeeper-gc.log,kafkaServer-gc.log, server.log and
>>> > state-change.log
>>> > > > > >
>>> > > > > > Thanks
>>> > > > > >
>>> > > > > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <
>>> wangguoz@gmail.com
>>> > >
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > > Sorry for getting late on this Sameer.
>>> > > > > > >
>>> > > > > > > The logs you showed above are from controller logs, do you
>>> see
>>> > > > brokers
>>> > > > > > logs
>>> > > > > > > besides that?
>>> > > > > > >
>>> > > > > > > Guozhang
>>> > > > > > >
>>> > > > > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <
>>> syami@ailive.net>
>>> > > > wrote:
>>> > > > > > >
>>> > > > > > > > Hi Guozhang,
>>> > > > > > > >
>>> > > > > > > > Is there any clue in the above data as to what is wrong?
>>> > > > > > > >
>>> > > > > > > > thanks
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <
>>> syami@ailive.net
>>> > >
>>> > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > This is the one from the file specified in
>>> log4j.properties -
>>> > > > > > > > >
>>> > > > > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on
>>> > > > Controller
>>> > > > > > 0]:
>>> > > > > > > > > Topic change listener fired for path /brokers/topics with
>>> > > > children
>>> > > > > > Test
>>> > > > > > > > >
>>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
>>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener on
>>> > > Controller
>>> > > > > 0]:
>>> > > > > > > New
>>> > > > > > > > > topics: [Set(Test)], deleted topics: [Set()], new
>>> partition
>>> > > > replica
>>> > > > > > > > > assignment [Map([Test,0] -> List(0), [Test,1] ->
>>> List(0))]
>>> > > > > > > > >
>>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
>>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New topic
>>> > > creation
>>> > > > > > > > callback
>>> > > > > > > > > for [Test,0],[Test,1] (kafka.controller.KafkaController)
>>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New
>>> partition
>>> > > > > creation
>>> > > > > > > > > callback for [Test,0],[Test,1]
>>> > > (kafka.controller.KafkaController)
>>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Partition state machine
>>> on
>>> > > > > Controller
>>> > > > > > > 0]:
>>> > > > > > > > > Invoking state change to NewPartition for partitions
>>> > > > > > [Test,0],[Test,1]
>>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>>> > > > > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine on
>>> > > > controller
>>> > > > > > 0]:
>>> > > > > > > > > Invoking state change to NewReplica for replicas
>>> > > > > > > > >
>>> > > > >
>>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
>>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
>>> > > > > > > > > [2014-10-28 22:38:36,111] INFO [Partition state machine
>>> on
>>> > > > > Controller
>>> > > > > > > 0]:
>>> > > > > > > > > Invoking state change to OnlinePartition for partitions
>>> > > > > > > [Test,0],[Test,1]
>>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>>> > > > > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state machine
>>> on
>>> > > > > > Controller
>>> > > > > > > > 0]:
>>> > > > > > > > > Live assigned replicas for partition [Test,0] are:
>>> [List(0)]
>>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>>> > > > > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state machine
>>> on
>>> > > > > > Controller
>>> > > > > > > > 0]:
>>> > > > > > > > > Initializing leader and isr for partition [Test,0] to
>>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
>>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>>> > > > > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state machine
>>> on
>>> > > > > > Controller
>>> > > > > > > > 0]:
>>> > > > > > > > > Live assigned replicas for partition [Test,1] are:
>>> [List(0)]
>>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>>> > > > > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state machine
>>> on
>>> > > > > > Controller
>>> > > > > > > > 0]:
>>> > > > > > > > > Initializing leader and isr for partition [Test,1] to
>>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
>>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>>> > > > > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine on
>>> > > > controller
>>> > > > > > 0]:
>>> > > > > > > > > Invoking state change to OnlineReplica for replicas
>>> > > > > > > > >
>>> > > > >
>>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
>>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
>>> > > > > > > > > [2014-10-28 22:39:28,160] ERROR
>>> > > > > > [Controller-0-to-broker-0-send-thread],
>>> > > > > > > > > Controller 0's connection to broker
>>> > id:0,host:x.x.x.x,port:9092
>>> > > > was
>>> > > > > > > > > unsuccessful (kafka.controller.RequestSendThread)
>>> > > > > > > > > java.net.ConnectException: Connection timed out
>>> > > > > > > > > at sun.nio.ch.Net.connect0(Native Method)
>>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
>>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:449)
>>> > > > > > > > > at
>>> > > > > >
>>> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
>>> > > > > > > > > at
>>> > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>>> > > > > > > > > at
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
>>> > > > > > > > > at
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
>>> > > > > > > > > at
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
>>> > > > > > > > > at
>>> > > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>>> > > > > > > > > [2014-10-28 22:39:28,460] ERROR
>>> > > > > > [Controller-0-to-broker-0-send-thread],
>>> > > > > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr request
>>> with
>>> > > > > > > correlation
>>> > > > > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092. Reconnecting
>>> to
>>> > > > broker.
>>> > > > > > > > > (kafka.controller.RequestSendThread)
>>> > > > > > > > > java.nio.channels.ClosedChannelException
>>> > > > > > > > > at
>>> > > > kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
>>> > > > > > > > > at
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
>>> > > > > > > > > at
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
>>> > > > > > > > > at
>>> > > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>>> > > > > > > > >
>>> > > > > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
>>> > > > wangguoz@gmail.com
>>> > > > > >
>>> > > > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > >> Sameer, the one you sent me is from the producer logs,
>>> not
>>> > the
>>> > > > > > broker
>>> > > > > > > > >> logs.
>>> > > > > > > > >> And /tmp/kafka-logs actually store the message log
>>> (i.e. the
>>> > > > > "data")
>>> > > > > > > > >> instead of the server logs. Their locations should be
>>> > > specified
>>> > > > in
>>> > > > > > > your
>>> > > > > > > > >> log4j.properties file.
>>> > > > > > > > >>
>>> > > > > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <
>>> > > syami@ailive.net>
>>> > > > > > > wrote:
>>> > > > > > > > >>
>>> > > > > > > > >> > Is this what you are looking for -
>>> > > > > > > > >> >
>>> > > > > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
>>> > > > 172.31.16.31:9092
>>> > > > > > for
>>> > > > > > > > >> > producing
>>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
>>> > > > > > > 172.31.16.31:9092
>>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
>>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
>>> > metadata
>>> > > > for
>>> > > > > 1
>>> > > > > > > > >> topic(s)
>>> > > > > > > > >> > Set(Test)
>>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
>>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
>>> > fetching
>>> > > > > > metadata
>>> > > > > > > > >> > [{TopicMetadata for topic Test ->
>>> > > > > > > > >> > No partition metadata for topic Test due to
>>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
>>> > [Test]:
>>> > > > > class
>>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
>>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
>>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
>>> > > > partition
>>> > > > > > info
>>> > > > > > > > for
>>> > > > > > > > >> > topic Test
>>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
>>> > broker
>>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 9
>>> > for 1
>>> > > > > > > topic(s)
>>> > > > > > > > >> > Set(Test)
>>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
>>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
>>> > > > > SO_TIMEOUT =
>>> > > > > > > > 10000
>>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
>>> > > > SO_SNDBUF
>>> > > > > =
>>> > > > > > > > 102400
>>> > > > > > > > >> > (requested 102400).
>>> > > > > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
>>> > > > 172.31.16.31:9092
>>> > > > > > for
>>> > > > > > > > >> > producing
>>> > > > > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
>>> > > > > > > 172.31.16.31:9092
>>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
>>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
>>> > metadata
>>> > > > for
>>> > > > > 1
>>> > > > > > > > >> topic(s)
>>> > > > > > > > >> > Set(Test)
>>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
>>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
>>> > fetching
>>> > > > > > metadata
>>> > > > > > > > >> > [{TopicMetadata for topic Test ->
>>> > > > > > > > >> > No partition metadata for topic Test due to
>>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
>>> > [Test]:
>>> > > > > class
>>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
>>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
>>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
>>> > > collate
>>> > > > > > > > messages
>>> > > > > > > > >> by
>>> > > > > > > > >> > topic, partition due to: Failed to fetch topic
>>> metadata
>>> > for
>>> > > > > topic:
>>> > > > > > > > Test
>>> > > > > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off
>>> for
>>> > > 1000
>>> > > > > ms
>>> > > > > > > > before
>>> > > > > > > > >> > retrying send. Remaining retries = 6
>>> > > > > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
>>> > broker
>>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id
>>> 10
>>> > for
>>> > > 1
>>> > > > > > > topic(s)
>>> > > > > > > > >> > Set(Test)
>>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
>>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
>>> > > > > SO_TIMEOUT =
>>> > > > > > > > 10000
>>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
>>> > > > SO_SNDBUF
>>> > > > > =
>>> > > > > > > > 102400
>>> > > > > > > > >> > (requested 102400).
>>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
>>> > > > 172.31.16.31:9092
>>> > > > > > for
>>> > > > > > > > >> > producing
>>> > > > > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
>>> > > > > > > 172.31.16.31:9092
>>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
>>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
>>> > metadata
>>> > > > for
>>> > > > > 1
>>> > > > > > > > >> topic(s)
>>> > > > > > > > >> > Set(Test)
>>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
>>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
>>> > fetching
>>> > > > > > metadata
>>> > > > > > > > >> > [{TopicMetadata for topic Test ->
>>> > > > > > > > >> > No partition metadata for topic Test due to
>>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
>>> > [Test]:
>>> > > > > class
>>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
>>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
>>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
>>> > > > partition
>>> > > > > > info
>>> > > > > > > > for
>>> > > > > > > > >> > topic Test
>>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
>>> > broker
>>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id
>>> 11
>>> > for
>>> > > 1
>>> > > > > > > topic(s)
>>> > > > > > > > >> > Set(Test)
>>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
>>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
>>> > > > > SO_TIMEOUT =
>>> > > > > > > > 10000
>>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
>>> > > > SO_SNDBUF
>>> > > > > =
>>> > > > > > > > 102400
>>> > > > > > > > >> > (requested 102400).
>>> > > > > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
>>> > > > 172.31.16.31:9092
>>> > > > > > for
>>> > > > > > > > >> > producing
>>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
>>> > > > > > > 172.31.16.31:9092
>>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
>>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
>>> > metadata
>>> > > > for
>>> > > > > 1
>>> > > > > > > > >> topic(s)
>>> > > > > > > > >> > Set(Test)
>>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
>>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
>>> > fetching
>>> > > > > > metadata
>>> > > > > > > > >> > [{TopicMetadata for topic Test ->
>>> > > > > > > > >> > No partition metadata for topic Test due to
>>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
>>> > [Test]:
>>> > > > > class
>>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
>>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
>>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
>>> > > collate
>>> > > > > > > > messages
>>> > > > > > > > >> by
>>> > > > > > > > >> > topic, partition due to: Failed to fetch topic
>>> metadata
>>> > for
>>> > > > > topic:
>>> > > > > > > > Test
>>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
>>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off
>>> for
>>> > > 1000
>>> > > > > ms
>>> > > > > > > > before
>>> > > > > > > > >> > retrying send. Remaining retries = 5
>>> > > > > > > > >> >
>>> > > > > > > > >> > As for /tmp/kafka-logs, I can see only these files -
>>> > > > > > > > >> > recovery-point-offset-checkpoint
>>> > > > replication-offset-checkpoint
>>> > > > > > > > >> >
>>> > > > > > > > >> > I do not see the usual Kafka log files here.
>>> > > > > > > > >> >
>>> > > > > > > > >> > thanks
>>> > > > > > > > >> > Sameer
>>> > > > > > > > >> >
>>> > > > > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
>>> > > > > > wangguoz@gmail.com>
>>> > > > > > > > >> wrote:
>>> > > > > > > > >> >
>>> > > > > > > > >> > > Sameer, could you check broker logs and see if it
>>> has
>>> > > > similar
>>> > > > > > > > >> > > LeaderNotAvailable exceptions?
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > Guozhang
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
>>> > > > > syami@ailive.net
>>> > > > > > >
>>> > > > > > > > >> wrote:
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > > Yes. I can telnet from outside using the public
>>> ip of
>>> > > the
>>> > > > > > > instance
>>> > > > > > > > >> and
>>> > > > > > > > >> > > port
>>> > > > > > > > >> > > > of 9092
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <
>>> > > > > > > > >> roger.hoover@gmail.com
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > > wrote:
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > > > > Just a guess but could it be a firewall issue?
>>> Did
>>> > > you
>>> > > > > > enable
>>> > > > > > > > >> > > > connections
>>> > > > > > > > >> > > > > to port 9092 from outside EC2 in a security
>>> group?
>>> > > Can
>>> > > > > you
>>> > > > > > > > >> telnet to
>>> > > > > > > > >> > > > each
>>> > > > > > > > >> > > > > broker IP and port?
>>> > > > > > > > >> > > > >
>>> > > > > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami <
>>> > > > > > > syami@ailive.net
>>> > > > > > > > >
>>> > > > > > > > >> > > wrote:
>>> > > > > > > > >> > > > >
>>> > > > > > > > >> > > > > > There was a typo earlier.
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > > > This is the output -
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > > > Topic:Test PartitionCount:2
>>> ReplicationFactor:1
>>> > > > > > > Configs:
>>> > > > > > > > >> > > > > > Topic: Test Partition: 0
>>> Leader: 0
>>> > > > > > > Replicas:
>>> > > > > > > > 0
>>> > > > > > > > >> > > > Isr: 0
>>> > > > > > > > >> > > > > > Topic: Test Partition: 1
>>> Leader: 0
>>> > > > > > > Replicas:
>>> > > > > > > > 0
>>> > > > > > > > >> > > > Isr: 0
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > > > thanks
>>> > > > > > > > >> > > > > > Sameer
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang
>>> Wang <
>>> > > > > > > > >> wangguoz@gmail.com
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > > > wrote:
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > > > > Sameer,
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > The topic name is not consistent from your
>>> > > describe
>>> > > > > > > outputs.
>>> > > > > > > > >> > > Normally
>>> > > > > > > > >> > > > > > they
>>> > > > > > > > >> > > > > > > should be either
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > Topic:*Test* PartitionCount:2
>>> > > > > ReplicationFactor:1
>>> > > > > > > > >> > > Configs:
>>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 0
>>> > Leader: 0
>>> > > > > > > > >> Replicas: 0
>>> > > > > > > > >> > > > > > Isr: 0
>>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 1
>>> > Leader: 0
>>> > > > > > > > >> Replicas: 0
>>> > > > > > > > >> > > > > > Isr: 0
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > Or
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
>>> > > > > > > > ReplicationFactor:1
>>> > > > > > > > >> > > > > Configs:
>>> > > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 0
>>> > > > Leader:
>>> > > > > 0
>>> > > > > > > > >> > > Replicas:
>>> > > > > > > > >> > > > 0
>>> > > > > > > > >> > > > > > > Isr: 0
>>> > > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 1
>>> > > > Leader:
>>> > > > > 0
>>> > > > > > > > >> > > Replicas:
>>> > > > > > > > >> > > > 0
>>> > > > > > > > >> > > > > > > Isr: 0
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > Did you manually modify the topic name in
>>> ZK?
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > Guozhang
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer
>>> Yami <
>>> > > > > > > > >> syami@ailive.net>
>>> > > > > > > > >> > > > wrote:
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > > Hi Guozhang,
>>> > > > > > > > >> > > > > > > >
>>> > > > > > > > >> > > > > > > > I was wondering if there is anything
>>> wrong in
>>> > > the
>>> > > > > > above
>>> > > > > > > > >> > > --describe
>>> > > > > > > > >> > > > > > data.
>>> > > > > > > > >> > > > > > > >
>>> > > > > > > > >> > > > > > > > thanks
>>> > > > > > > > >> > > > > > > >
>>> > > > > > > > >> > > > > > > >
>>> > > > > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer
>>> Yami <
>>> > > > > > > > >> syami@ailive.net
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > > > wrote:
>>> > > > > > > > >> > > > > > > >
>>> > > > > > > > >> > > > > > > > > This is what I get -
>>> > > > > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper
>>> > > localhost:2181
>>> > > > > > > > --describe
>>> > > > > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
>>> > > > > > > > >> ReplicationFactor:1
>>> > > > > > > > >> > > > > > Configs:
>>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 0
>>> > > Leader: 0
>>> > > > > > > > >> > Replicas: 0
>>> > > > > > > > >> > > > > > > Isr: 0
>>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 1
>>> > > Leader: 0
>>> > > > > > > > >> > Replicas: 0
>>> > > > > > > > >> > > > > > > Isr: 0
>>> > > > > > > > >> > > > > > > > >
>>> > > > > > > > >> > > > > > > > > The topics are being created at
>>> run-time.
>>> > > > > > > > >> > > > > > > > >
>>> > > > > > > > >> > > > > > > > > thanks
>>> > > > > > > > >> > > > > > > > >
>>> > > > > > > > >> > > > > > > > >
>>> > > > > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM,
>>> Guozhang
>>> > > Wang <
>>> > > > > > > > >> > > > wangguoz@gmail.com
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > > > > > wrote:
>>> > > > > > > > >> > > > > > > > >
>>> > > > > > > > >> > > > > > > > >> Could you use the topic tools
>>> "--describe"
>>> > to
>>> > > > > check
>>> > > > > > > if
>>> > > > > > > > >> the
>>> > > > > > > > >> > > topic
>>> > > > > > > > >> > > > > > > Test's
>>> > > > > > > > >> > > > > > > > >> partitions leaders are available?
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > >
>>> > > > > http://kafka.apache.org/documentation.html#basic_ops_add_topic
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > > > > > > > >> Guozhang
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM, Sameer
>>> > Yami
>>> > > <
>>> > > > > > > > >> > > syami@ailive.net>
>>> > > > > > > > >> > > > > > > wrote:
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > > > > > > > >> > That is not the case.
>>> > > > > > > > >> > > > > > > > >> > This detailed error is below. Due to
>>> this
>>> > > > > error ,
>>> > > > > > > the
>>> > > > > > > > >> > > producer
>>> > > > > > > > >> > > > > is
>>> > > > > > > > >> > > > > > > not
>>> > > > > > > > >> > > > > > > > >> able
>>> > > > > > > > >> > > > > > > > >> > to send messages
>>> > > > > > > > >> > > > > > > > >> >
>>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG
>>> > > > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-52:
>>> > > > > > > > >> > > > > > > > >> > Getting broker partition info for
>>> topic
>>> > > Test
>>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main INFO
>>> > > > > > > > >> > > > > kafka.client.ClientUtils$-68:
>>> > > > > > > > >> > > > > > > > >> Fetching
>>> > > > > > > > >> > > > > > > > >> > metadata from broker
>>> > > > > > > > id:0,host:54.215.200.61,port:9092
>>> > > > > > > > >> > with
>>> > > > > > > > >> > > > > > > > correlation
>>> > > > > > > > >> > > > > > > > >> id
>>> > > > > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
>>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG
>>> > > > > > > > >> > > > > > > kafka.network.BlockingChannel-52:
>>> > > > > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT =
>>> 10000
>>> > > > > (requested
>>> > > > > > > > >> 10000),
>>> > > > > > > > >> > > > > > SO_RCVBUF
>>> > > > > > > > >> > > > > > > =
>>> > > > > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF =
>>> 102400
>>> > > > > > > (requested
>>> > > > > > > > >> > > 102400).
>>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main INFO
>>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
>>> > > > > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092 for
>>> > > > producing
>>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main INFO
>>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
>>> > > > > > > > >> > > > > > > > >> > Disconnecting from
>>> 54.215.200.61:9092
>>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG
>>> > > > > > > > >> > > > > kafka.client.ClientUtils$-52:
>>> > > > > > > > >> > > > > > > > >> > Successfully fetched metadata for 1
>>> > > topic(s)
>>> > > > > > > > Set(Test)
>>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main WARN
>>> > > > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-83:
>>> > > > > > > > >> > > > > > > > >> > Error while fetching metadata
>>> > > [{TopicMetadata
>>> > > > > for
>>> > > > > > > > topic
>>> > > > > > > > >> > Test
>>> > > > > > > > >> > > > ->
>>> > > > > > > > >> > > > > > > > >> > No partition metadata for topic Test
>>> due
>>> > to
>>> > > > > > > > >> > > > > > > > >> >
>>> > kafka.common.LeaderNotAvailableException}]
>>> > > > for
>>> > > > > > > topic
>>> > > > > > > > >> > [Test]:
>>> > > > > > > > >> > > > > class
>>> > > > > > > > >> > > > > > > > >> >
>>> kafka.common.LeaderNotAvailableException
>>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR
>>> > > > > > > > >> > > > > > > > >> >
>>> > > kafka.producer.async.DefaultEventHandler-97:
>>> > > > > > Failed
>>> > > > > > > > to
>>> > > > > > > > >> > > collate
>>> > > > > > > > >> > > > > > > > messages
>>> > > > > > > > >> > > > > > > > >> by
>>> > > > > > > > >> > > > > > > > >> > topic, partition due to: Failed to
>>> fetch
>>> > > > topic
>>> > > > > > > > metadata
>>> > > > > > > > >> > for
>>> > > > > > > > >> > > > > topic:
>>> > > > > > > > >> > > > > > > > Test
>>> > > > > > > > >> > > > > > > > >> >
>>> > > > > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM,
>>> Guozhang
>>> > > > Wang
>>> > > > > <
>>> > > > > > > > >> > > > > > wangguoz@gmail.com>
>>> > > > > > > > >> > > > > > > > >> wrote:
>>> > > > > > > > >> > > > > > > > >> >
>>> > > > > > > > >> > > > > > > > >> > > Could you see if this is the case:
>>> > > > > > > > >> > > > > > > > >> > >
>>> > > > > > > > >> > > > > > > > >> > >
>>> > > > > > > > >> > > > > > > > >> > >
>>> > > > > > > > >> > > > > > > > >> >
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > > > > > > >
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > >
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
>>> > > > > > > > >> > > > > > > > >> > > ?
>>> > > > > > > > >> > > > > > > > >> > >
>>> > > > > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM,
>>> > Sameer
>>> > > > > Yami <
>>> > > > > > > > >> > > > > syami@ailive.net
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > > >> wrote:
>>> > > > > > > > >> > > > > > > > >> > >
>>> > > > > > > > >> > > > > > > > >> > > > Hi,
>>> > > > > > > > >> > > > > > > > >> > > >
>>> > > > > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper test
>>> > setup
>>> > > > on a
>>> > > > > > > > single
>>> > > > > > > > >> EC2
>>> > > > > > > > >> > > > > > machine.
>>> > > > > > > > >> > > > > > > > The
>>> > > > > > > > >> > > > > > > > >> > > > host.name is the private ip
>>> address
>>> > > and
>>> > > > > the
>>> > > > > > > > >> > > > > > > advertised.host.name
>>> > > > > > > > >> > > > > > > > is
>>> > > > > > > > >> > > > > > > > >> > the
>>> > > > > > > > >> > > > > > > > >> > > > public address in
>>> server.properties.
>>> > > > > > > > >> > > > > > > > >> > > >
>>> > > > > > > > >> > > > > > > > >> > > > We are trying to publish from
>>> outside
>>> > > EC2
>>> > > > > > using
>>> > > > > > > > the
>>> > > > > > > > >> > > public
>>> > > > > > > > >> > > > > ip
>>> > > > > > > > >> > > > > > of
>>> > > > > > > > >> > > > > > > > the
>>> > > > > > > > >> > > > > > > > >> > EC2
>>> > > > > > > > >> > > > > > > > >> > > > machine but get this error on the
>>> > > client
>>> > > > > > side -
>>> > > > > > > > >> > > > > > > > >> > > > No partition metadata for topic
>>> test
>>> > > due
>>> > > > to
>>> > > > > > > > >> > > > > > > > >> > > >
>>> > > > kafka.common.LeaderNotAvailableException}]
>>> > > > > > for
>>> > > > > > > > >> topic
>>> > > > > > > > >> > > > [test]:
>>> > > > > > > > >> > > > > > > class
>>> > > > > > > > >> > > > > > > > >> > > >
>>> > > kafka.common.LeaderNotAvailableException
>>> > > > > > > > >> > > > > > > > >> > > >
>>> > > > > > > > >> > > > > > > > >> > > > The zookeeper has the public ip
>>> > stored
>>> > > in
>>> > > > > > > > >> > > /brokers/ids/0.
>>> > > > > > > > >> > > > > > > > >> > > > We have tried multiple
>>> combinations
>>> > but
>>> > > > it
>>> > > > > > > works
>>> > > > > > > > >> ONLY
>>> > > > > > > > >> > > when
>>> > > > > > > > >> > > > > the
>>> > > > > > > > >> > > > > > > > >> > advertised
>>> > > > > > > > >> > > > > > > > >> > > > host name is not used and the
>>> client
>>> > is
>>> > > > > > > internal
>>> > > > > > > > to
>>> > > > > > > > >> > EC2.
>>> > > > > > > > >> > > > > > > > >> > > >
>>> > > > > > > > >> > > > > > > > >> > > > Please let me know what the
>>> problem
>>> > > could
>>> > > > > be.
>>> > > > > > > > >> > > > > > > > >> > > >
>>> > > > > > > > >> > > > > > > > >> > > > thanks
>>> > > > > > > > >> > > > > > > > >> > > > Sameer
>>> > > > > > > > >> > > > > > > > >> > > >
>>> > > > > > > > >> > > > > > > > >> > >
>>> > > > > > > > >> > > > > > > > >> > >
>>> > > > > > > > >> > > > > > > > >> > >
>>> > > > > > > > >> > > > > > > > >> > > --
>>> > > > > > > > >> > > > > > > > >> > > -- Guozhang
>>> > > > > > > > >> > > > > > > > >> > >
>>> > > > > > > > >> > > > > > > > >> >
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > > > > > > > >> --
>>> > > > > > > > >> > > > > > > > >> -- Guozhang
>>> > > > > > > > >> > > > > > > > >>
>>> > > > > > > > >> > > > > > > > >
>>> > > > > > > > >> > > > > > > > >
>>> > > > > > > > >> > > > > > > >
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > > > --
>>> > > > > > > > >> > > > > > > -- Guozhang
>>> > > > > > > > >> > > > > > >
>>> > > > > > > > >> > > > > >
>>> > > > > > > > >> > > > >
>>> > > > > > > > >> > > >
>>> > > > > > > > >> > >
>>> > > > > > > > >> > >
>>> > > > > > > > >> > >
>>> > > > > > > > >> > > --
>>> > > > > > > > >> > > -- Guozhang
>>> > > > > > > > >> > >
>>> > > > > > > > >> >
>>> > > > > > > > >>
>>> > > > > > > > >>
>>> > > > > > > > >>
>>> > > > > > > > >> --
>>> > > > > > > > >> -- Guozhang
>>> > > > > > > > >>
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > --
>>> > > > > > > -- Guozhang
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > -- Guozhang
>>> > > > >
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > -- Guozhang
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>>
>>
>>
>
Re: Cannot connect to Kafka from outside of EC2
Posted by Sameer Yami <sy...@ailive.net>.
Hi Guozhang,
Attached are the two logs with debug enabled.
Thanks!
On Fri, Nov 7, 2014 at 2:09 PM, Sameer Yami <sy...@ailive.net> wrote:
> The version is kafka_2.10-0.8.1.1. It is not the latest trunk.
> Will try enabling debug version.
>
> thanks
>
>
> On Thu, Nov 6, 2014 at 9:37 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> Sameer,
>>
>> The server logs do not contain any non-INFO logs, which is a bit wired.
>> Did
>> you deploy the current trunk of Kafka? Also could you enable DEBUG level
>> logging on Kafka brokers?
>>
>> Guozhang
>>
>> On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami <sy...@ailive.net> wrote:
>>
>> > The server.log was taken separately.
>> > We ran the test again and the server and producer logs are below (to get
>> > same timings).
>> >
>> >
>> > Thanks!
>> >
>> >
>> >
>> ------------------------------------------------------------------------------------
>> >
>> >
>> >
>> > Producer Logs -
>> >
>> >
>> > 2014-11-05 23:38:58,693
>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>> sessionid:
>> > 0x1498251e8680002 after 0ms
>> > 2014-11-05 23:39:00,695
>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>> sessionid:
>> > 0x1498251e8680002 after 0ms
>> > 2014-11-05 23:39:02,696
>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>> sessionid:
>> > 0x1498251e8680002 after 0ms
>> > 2014-11-05 23:39:02,828 pool-13-thread-2 INFO
>> > kafka.utils.VerifiableProperties-68: Verifying properties
>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
>> > kafka.utils.VerifiableProperties-68: Property auto.commit.interval.ms
>> is
>> > overridden to 1000
>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
>> > kafka.utils.VerifiableProperties-68: Property auto.offset.reset is
>> > overridden to smallest
>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
>> > kafka.utils.VerifiableProperties-68: Property consumer.timeout.ms is
>> > overridden to 10
>> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
>> > kafka.utils.VerifiableProperties-68: Property group.id is overridden to
>> > TestCheck
>> > 2014-11-05 23:39:02,830 pool-13-thread-2 WARN
>> > kafka.utils.VerifiableProperties-83: Property serializer.class is not
>> valid
>> > 2014-11-05 23:39:02,830 pool-13-thread-2 INFO
>> > kafka.utils.VerifiableProperties-68: Property zookeeper.connect is
>> > overridden to 172.31.25.198:2181
>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Connecting to
>> > zookeeper instance at 172.31.25.198:2181
>> > 2014-11-05 23:39:02,831 pool-13-thread-2 DEBUG
>> > org.I0Itec.zkclient.ZkConnection-63: Creating new ZookKeeper instance to
>> > connect to 172.31.25.198:2181.
>> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
>> > org.apache.zookeeper.ZooKeeper-379: Initiating client connection,
>> > connectString=172.31.25.198:2181 sessionTimeout=6000
>> > watcher=org.I0Itec.zkclient.ZkClient@3903b165
>> > 2014-11-05 23:39:02,831 ZkClient-EventThread-29-172.31.25.198:2181
>> INFO
>> > org.I0Itec.zkclient.ZkEventThread-64: Starting ZkClient event thread.
>> > 2014-11-05 23:39:02,831 pool-13-thread-1 INFO
>> > kafka.utils.VerifiableProperties-68: Verifying properties
>> > 2014-11-05 23:39:02,836 pool-13-thread-2-SendThread() INFO
>> > org.apache.zookeeper.ClientCnxn-1061: Opening socket connection to
>> server /
>> > 172.31.25.198:2181
>> > 2014-11-05 23:39:02,836 pool-13-thread-1 WARN
>> > kafka.utils.VerifiableProperties-83: Property batch.size is not valid
>> > 2014-11-05 23:39:02,832 pool-13-thread-2 DEBUG
>> > org.I0Itec.zkclient.ZkClient-878: Awaiting connection to Zookeeper
>> server
>> > 2014-11-05 23:39:02,836 pool-13-thread-1 INFO
>> > kafka.utils.VerifiableProperties-68: Property message.send.max.retries
>> is
>> > overridden to 10
>> > 2014-11-05 23:39:02,836 pool-13-thread-2 DEBUG
>> > org.I0Itec.zkclient.ZkClient-628: Waiting for keeper state SyncConnected
>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
>> > kafka.utils.VerifiableProperties-68: Property metadata.broker.list is
>> > overridden to 172.31.25.198:9092
>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
>> > kafka.utils.VerifiableProperties-68: Property retry.backoff.ms is
>> > overridden to 1000
>> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
>> > kafka.utils.VerifiableProperties-68: Property serializer.class is
>> > overridden to kafka.serializer.StringEncoder
>> > 2014-11-05 23:39:02,837
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > INFO org.apache.zookeeper.ClientCnxn-950: Socket connection established
>> to
>> > ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
>> initiating
>> > session
>> > 2014-11-05 23:39:02,838
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-999: Session establishment request
>> > sent on ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181
>> > 2014-11-05 23:39:02,837 pool-13-thread-1 WARN
>> > kafka.utils.VerifiableProperties-83: Property zk.connectiontimeout.ms
>> is
>> > not valid
>> > 2014-11-05 23:39:02,841
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > INFO org.apache.zookeeper.ClientCnxn-739: Session establishment
>> complete on
>> > server ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
>> > sessionid = 0x1498251e8680003, negotiated timeout = 6000
>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
>> > state:SyncConnected type:None path:null
>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread INFO
>> > org.I0Itec.zkclient.ZkClient-449: zookeeper state changed
>> (SyncConnected)
>> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
>> > 2014-11-05 23:39:02,841 pool-13-thread-2 DEBUG
>> > org.I0Itec.zkclient.ZkClient-638: State is SyncConnected
>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
>> > kafka.utils.KafkaScheduler-52: Initializing task scheduler.
>> > 2014-11-05 23:39:02,842 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting auto
>> > committer every 1000 ms
>> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
>> > kafka.utils.KafkaScheduler-52: Scheduling task kafka-consumer-autocommit
>> > with initial delay 1000 ms and period 1000 ms.
>> > 2014-11-05 23:39:02,843 pool-13-thread-2 DEBUG
>> > kafka.consumer.ZookeeperConsumerConnector-52:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], entering consume
>> > 2014-11-05 23:39:02,843 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin registering
>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
>> > 2014-11-05 23:39:02,846
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 1,1 replyHeader:: 1,29,-101 request::
>> >
>> >
>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
>> > response::
>> > 2014-11-05 23:39:02,854
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 2,1 replyHeader:: 2,30,-101 request::
>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
>> > 2014-11-05 23:39:02,858
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 3,1 replyHeader:: 3,31,0 request::
>> > '/consumers/TestCheck,,v{s{31,s{'world,'anyone}}},0 response::
>> > '/consumers/TestCheck
>> > 2014-11-05 23:39:02,861
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 4,1 replyHeader:: 4,32,0 request::
>> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
>> > '/consumers/TestCheck/ids
>> > 2014-11-05 23:39:02,866
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 5,1 replyHeader:: 5,33,0 request::
>> >
>> >
>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
>> > response::
>> >
>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
>> > 2014-11-05 23:39:02,867 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end registering
>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
>> > kafka.consumer.ZookeeperConsumerConnector-52:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Adding
>> topicThreadId
>> > (Test,TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0) and queue []
>> to
>> > topicThreadIdAndQueues data structure
>> > 2014-11-05 23:39:02,868
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting watcher
>> > executor thread for consumer
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
>> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
>> > kafka.consumer.ZookeeperConsumerConnector-52:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], adding topic Test
>> and
>> > 1 streams to map.
>> > 2014-11-05 23:39:02,870
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 6,3 replyHeader:: 6,33,0 request::
>> > '/consumers/TestCheck/ids,T response::
>> > s{32,32,1415230742859,1415230742859,0,1,0,0,0,1,33}
>> > 2014-11-05 23:39:02,872
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 7,8 replyHeader:: 7,33,0 request::
>> > '/consumers/TestCheck/ids,T response::
>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
>> > 2014-11-05 23:39:02,873
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 8,3 replyHeader:: 8,33,-101 request::
>> > '/brokers/topics/Test,T response::
>> > 2014-11-05 23:39:02,873 pool-13-thread-2 DEBUG
>> > org.I0Itec.zkclient.ZkClient-137: Subscribed data changes for
>> > /brokers/topics/Test
>> > 2014-11-05 23:39:02,873 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
>> > 2014-11-05 23:39:02,875 pool-13-thread-1 DEBUG
>> > kafka.producer.async.DefaultEventHandler-52: Handling 1 events
>> > 2014-11-05 23:39:02,877
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 9,8 replyHeader:: 9,33,0 request::
>> > '/brokers/ids,F response:: v{'0}
>> > 2014-11-05 23:39:02,879
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 10,4 replyHeader:: 10,33,0 request::
>> > '/brokers/ids/0,F response::
>> >
>> >
>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>> >
>> > 2014-11-05 23:39:02,887
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 11,4 replyHeader:: 11,33,0 request::
>> >
>> >
>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
>> > response::
>> >
>> >
>> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>> >
>> > 2014-11-05 23:39:02,896
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 12,8 replyHeader:: 12,33,0 request::
>> > '/consumers/TestCheck/ids,T response::
>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
>> > 2014-11-05 23:39:02,898
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 13,4 replyHeader:: 13,33,0 request::
>> >
>> >
>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
>> > response::
>> >
>> >
>> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>> >
>> > 2014-11-05 23:39:02,908
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 14,8 replyHeader:: 14,33,0 request::
>> > '/brokers/ids,F response:: v{'0}
>> > 2014-11-05 23:39:02,910
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 15,4 replyHeader:: 15,33,0 request::
>> > '/brokers/ids/0,F response::
>> >
>> >
>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>> >
>> > 2014-11-05 23:39:02,917 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 0 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:02,917
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 16,4 replyHeader:: 16,33,-101 request::
>> > '/brokers/topics/Test,T response::
>> > 2014-11-05 23:39:02,920 pool-13-thread-2 DEBUG kafka.utils.ZkUtils$-52:
>> > Partition map for /brokers/topics/Test is Map()
>> > 2014-11-05 23:39:02,920 pool-13-thread-2 INFO
>> > kafka.consumer.ConsumerFetcherManager-68:
>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
>> > kafka.consumer.ConsumerFetcherManager-68:
>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
>> > kafka.consumer.ConsumerFetcherManager-68:
>> > [ConsumerFetcherManager-1415230742842] All connections stopped
>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
>> relevant
>> > queues for this fetcher
>> > 2014-11-05 23:39:02,921 pool-13-thread-2 DEBUG
>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
>> > this consumer iterator
>> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
>> > chunks in all the consumer message iterators
>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
>> offsets
>> > after clearing the fetcher queues
>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing partition
>> > ownership
>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
>> following
>> > partitions: ArrayBuffer() for topic Test with consumers:
>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
>> > 2014-11-05 23:39:02,922 pool-13-thread-2 WARN
>> > kafka.consumer.ZookeeperConsumerConnector-83:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], No broker
>> partitions
>> > consumed by consumer thread
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 for topic Test
>> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
>> > kafka.consumer.ZookeeperConsumerConnector-52:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
>> topic
>> > cache Map(Test -> ArrayBuffer())
>> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
>> > kafka.consumer.ZookeeperConsumerConnector-52:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per topic
>> > cache Map(Test ->
>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
>> > 2014-11-05 23:39:02,923 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
>> > 2014-11-05 23:39:02,927 pool-13-thread-2 INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
>> > 2014-11-05 23:39:02,928
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>> >
>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>> > Starting
>> > 2014-11-05 23:39:02,930 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:02,930 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:02,985
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-783: Got notification
>> > sessionid:0x1498251e8680003
>> > 2014-11-05 23:39:02,986
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-800: Got WatchedEvent
>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test for
>> > sessionid 0x1498251e8680003
>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
>> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
>> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test
>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
>> > org.I0Itec.zkclient.ZkEventThread-88: New event: ZkEvent[Data of
>> > /brokers/topics/Test changed sent to
>> >
>> >
>> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
>> > ]
>> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
>> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
>> > 2014-11-05 23:39:02,988 ZkClient-EventThread-29-172.31.25.198:2181
>> DEBUG
>> > org.I0Itec.zkclient.ZkEventThread-69: Delivering event #1 ZkEvent[Data
>> of
>> > /brokers/topics/Test changed sent to
>> >
>> >
>> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
>> > ]
>> > 2014-11-05 23:39:02,991
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 17,3 replyHeader:: 17,37,0 request::
>> > '/brokers/topics/Test,T response::
>> > s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
>> > 2014-11-05 23:39:02,993
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 18,4 replyHeader:: 18,37,0 request::
>> > '/brokers/topics/Test,T response::
>> >
>> >
>> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
>> >
>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Topic info for path
>> > /brokers/topics/Test changed to
>> > {"version":1,"partitions":{"1":[0],"0":[0]}}, triggering rebalance
>> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181
>> DEBUG
>> > org.I0Itec.zkclient.ZkEventThread-79: Delivering event #1 done
>> > 2014-11-05 23:39:02,995
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
>> > 2014-11-05 23:39:02,996
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 19,8 replyHeader:: 19,37,0 request::
>> > '/brokers/ids,F response:: v{'0}
>> > 2014-11-05 23:39:02,998
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 20,4 replyHeader:: 20,37,0 request::
>> > '/brokers/ids/0,F response::
>> >
>> >
>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>> >
>> > 2014-11-05 23:39:03,008
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 21,4 replyHeader:: 21,37,0 request::
>> >
>> >
>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
>> > response::
>> >
>> >
>> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>> >
>> > 2014-11-05 23:39:03,014
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 22,8 replyHeader:: 22,37,0 request::
>> > '/consumers/TestCheck/ids,T response::
>> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
>> > 2014-11-05 23:39:03,018
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 23,4 replyHeader:: 23,37,0 request::
>> >
>> >
>> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
>> > response::
>> >
>> >
>> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>> >
>> > 2014-11-05 23:39:03,041
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 24,8 replyHeader:: 24,37,0 request::
>> > '/brokers/ids,F response:: v{'0}
>> > 2014-11-05 23:39:03,043
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 25,4 replyHeader:: 25,37,0 request::
>> > '/brokers/ids/0,F response::
>> >
>> >
>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>> >
>> > 2014-11-05 23:39:03,058
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 26,4 replyHeader:: 26,37,0 request::
>> > '/brokers/topics/Test,T response::
>> >
>> >
>> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
>> >
>> > 2014-11-05 23:39:03,074
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.utils.ZkUtils$-52: Partition map for /brokers/topics/Test is
>> Map(1 ->
>> > List(0), 0 -> List(0))
>> > 2014-11-05 23:39:03,075
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ConsumerFetcherManager-68:
>> > [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
>> > 2014-11-05 23:39:03,082
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>> >
>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>> > Shutting down
>> > 2014-11-05 23:39:03,082 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:03,089
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>> >
>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>> > Stopped
>> > 2014-11-05 23:39:03,089
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>> >
>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>> > Shutdown completed
>> > 2014-11-05 23:39:03,089
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ConsumerFetcherManager-68:
>> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
>> > 2014-11-05 23:39:03,090
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ConsumerFetcherManager-68:
>> > [ConsumerFetcherManager-1415230742842] All connections stopped
>> > 2014-11-05 23:39:03,090
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all
>> relevant
>> > queues for this fetcher
>> > 2014-11-05 23:39:03,090
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
>> > this consumer iterator
>> > 2014-11-05 23:39:03,091
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
>> > chunks in all the consumer message iterators
>> > 2014-11-05 23:39:03,091
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
>> offsets
>> > after clearing the fetcher queues
>> > 2014-11-05 23:39:03,091
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing partition
>> > ownership
>> > 2014-11-05 23:39:03,094 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:03,096
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
>> following
>> > partitions: ArrayBuffer(0, 1) for topic Test with consumers:
>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
>> > 2014-11-05 23:39:03,098
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
>> > partition 0
>> > 2014-11-05 23:39:03,100
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 27,4 replyHeader:: 27,37,-101 request::
>> > '/consumers/TestCheck/offsets/Test/0,F response::
>> > 2014-11-05 23:39:03,116
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of Test:0:
>> > fetched offset = -1: consumed offset = -1 is -1
>> > 2014-11-05 23:39:03,117
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:0:
>> > fetched offset = -1: consumed offset = -1 is -1
>> > 2014-11-05 23:39:03,118
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.consumer.ZookeeperConsumerConnector-52:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:0: fetched
>> offset
>> > = -1: consumed offset = -1 selected new offset -1
>> > 2014-11-05 23:39:03,169
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
>> > partition 1
>> > 2014-11-05 23:39:03,171
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 28,4 replyHeader:: 28,37,-101 request::
>> > '/consumers/TestCheck/offsets/Test/1,F response::
>> > 2014-11-05 23:39:03,172
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of Test:1:
>> > fetched offset = -1: consumed offset = -1 is -1
>> > 2014-11-05 23:39:03,172
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:1:
>> > fetched offset = -1: consumed offset = -1 is -1
>> > 2014-11-05 23:39:03,172
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.consumer.ZookeeperConsumerConnector-52:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:1: fetched
>> offset
>> > = -1: consumed offset = -1 selected new offset -1
>> > 2014-11-05 23:39:03,175 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:03,175
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 29,1 replyHeader:: 29,38,-101 request::
>> >
>> >
>> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
>> > response::
>> > 2014-11-05 23:39:03,179
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 30,1 replyHeader:: 30,39,-101 request::
>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
>> > response::
>> > 2014-11-05 23:39:03,180 pool-13-thread-1 DEBUG
>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
>> > topic Test
>> > 2014-11-05 23:39:03,181 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 1 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:03,182 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:03,182 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:03,186
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 31,1 replyHeader:: 31,40,0 request::
>> > '/consumers/TestCheck/owners,,v{s{31,s{'world,'anyone}}},0 response::
>> > '/consumers/TestCheck/owners
>> > 2014-11-05 23:39:03,189 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:03,190
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 32,1 replyHeader:: 32,41,0 request::
>> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
>> response::
>> > '/consumers/TestCheck/owners/Test
>> > 2014-11-05 23:39:03,190 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:03,191 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:03,192 pool-13-thread-1 ERROR
>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
>> by
>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>> > 2014-11-05 23:39:03,193
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 33,1 replyHeader:: 33,42,0 request::
>> >
>> >
>> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
>> > response:: '/consumers/TestCheck/owners/Test/1
>> > 2014-11-05 23:39:03,193 pool-13-thread-1 INFO
>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
>> > retrying send. Remaining retries = 10
>> > 2014-11-05 23:39:03,193
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
>> > partition 1 for topic Test
>> > 2014-11-05 23:39:03,196
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 34,1 replyHeader:: 34,43,0 request::
>> >
>> >
>> '/consumers/TestCheck/owners/Test/0,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
>> > response:: '/consumers/TestCheck/owners/Test/0
>> > 2014-11-05 23:39:03,196
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
>> > partition 0 for topic Test
>> > 2014-11-05 23:39:03,196
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
>> > 2014-11-05 23:39:03,197
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.consumer.ZookeeperConsumerConnector-52:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per
>> topic
>> > cache Map(Test -> ArrayBuffer(0, 1))
>> > 2014-11-05 23:39:03,197
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> DEBUG
>> > kafka.consumer.ZookeeperConsumerConnector-52:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per topic
>> > cache Map(Test ->
>> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
>> > 2014-11-05 23:39:03,197
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
>> > Test:0: fetched offset = -1: consumed offset = -1,Test:1: fetched
>> offset =
>> > -1: consumed offset = -1
>> > 2014-11-05 23:39:03,199
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
>> >
>> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
>> > Starting
>> > 2014-11-05 23:39:03,199
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor
>> INFO
>> > kafka.consumer.ZookeeperConsumerConnector-68:
>> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
>> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
>> > 2014-11-05 23:39:03,202
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 35,8 replyHeader:: 35,43,0 request::
>> > '/brokers/ids,F response:: v{'0}
>> > 2014-11-05 23:39:03,204
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
>> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
>> > finished:false header:: 36,4 replyHeader:: 36,43,0 request::
>> > '/brokers/ids/0,F response::
>> >
>> >
>> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>> >
>> > 2014-11-05 23:39:03,214
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>> > INFO kafka.utils.VerifiableProperties-68: Verifying properties
>> > 2014-11-05 23:39:03,215
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>> > INFO kafka.utils.VerifiableProperties-68: Property client.id is
>> overridden
>> > to TestCheck
>> > 2014-11-05 23:39:03,215
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>> > INFO kafka.utils.VerifiableProperties-68: Property metadata.broker.list
>> is
>> > overridden to x.x.x.x:9092
>> > 2014-11-05 23:39:03,215
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>> > INFO kafka.utils.VerifiableProperties-68: Property request.timeout.ms
>> is
>> > overridden to 30000
>> > 2014-11-05 23:39:03,217
>> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
>> > INFO kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:x.x.x.x,port:9092 with correlation id 0 for 1 topic(s)
>> Set(Test)
>> > 2014-11-05 23:39:04,195 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 2 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:04,196 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:04,196 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:04,199 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:04,199 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:04,200 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:04,200 pool-13-thread-1 DEBUG
>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
>> > topic Test
>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 3 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:04,201 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:04,205 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:04,205 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:04,206 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:04,206 pool-13-thread-1 ERROR
>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
>> by
>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>> > 2014-11-05 23:39:04,206 pool-13-thread-1 INFO
>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
>> > retrying send. Remaining retries = 9
>> > 2014-11-05 23:39:04,698
>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>> sessionid:
>> > 0x1498251e8680002 after 0ms
>> > 2014-11-05 23:39:05,205
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>> sessionid:
>> > 0x1498251e8680003 after 0ms
>> > 2014-11-05 23:39:05,207 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 4 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:05,208 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:05,208 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:05,211 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:05,211 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:05,212 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:05,212 pool-13-thread-1 DEBUG
>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
>> > topic Test
>> > 2014-11-05 23:39:05,212 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 5 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:05,213 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:05,213 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:05,217 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:05,217 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:05,217 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:05,217 pool-13-thread-1 ERROR
>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
>> by
>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>> > 2014-11-05 23:39:05,218 pool-13-thread-1 INFO
>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
>> > retrying send. Remaining retries = 8
>> > 2014-11-05 23:39:06,218 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 6 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:06,219 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:06,219 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:06,224 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:06,224 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:06,225 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:06,225 pool-13-thread-1 DEBUG
>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
>> > topic Test
>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 7 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:06,226 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:06,230 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:06,230 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:06,231 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:06,231 pool-13-thread-1 ERROR
>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
>> by
>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>> > 2014-11-05 23:39:06,231 pool-13-thread-1 INFO
>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
>> > retrying send. Remaining retries = 7
>> > 2014-11-05 23:39:06,699
>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>> sessionid:
>> > 0x1498251e8680002 after 0ms
>> > 2014-11-05 23:39:07,208
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>> sessionid:
>> > 0x1498251e8680003 after 0ms
>> > 2014-11-05 23:39:07,232 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 8 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:07,233 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:07,233 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:07,236 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:07,236 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:07,237 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:07,237 pool-13-thread-1 DEBUG
>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
>> > topic Test
>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 9 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:07,238 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:07,241 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:07,241 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:07,242 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:07,242 pool-13-thread-1 ERROR
>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
>> by
>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>> > 2014-11-05 23:39:07,242 pool-13-thread-1 INFO
>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
>> > retrying send. Remaining retries = 6
>> > 2014-11-05 23:39:08,243 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 10 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:08,243 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:08,245 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:08,249 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:08,250 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:08,250 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
>> > topic Test
>> > 2014-11-05 23:39:08,251 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 11 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:08,252 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:08,255 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:08,256 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:08,256 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:08,257 pool-13-thread-1 ERROR
>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
>> by
>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>> > 2014-11-05 23:39:08,257 pool-13-thread-1 INFO
>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
>> > retrying send. Remaining retries = 5
>> > 2014-11-05 23:39:08,702
>> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>> sessionid:
>> > 0x1498251e8680002 after 0ms
>> > 2014-11-05 23:39:09,211
>> >
>> >
>> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
>> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
>> sessionid:
>> > 0x1498251e8680003 after 0ms
>> > 2014-11-05 23:39:09,257 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 12 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:09,258 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:09,258 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:09,261 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:09,262 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
>> > topic Test
>> > 2014-11-05 23:39:09,263 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 13 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:09,264 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:09,264 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:09,268 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:09,268 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:09,269 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:09,269 pool-13-thread-1 ERROR
>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
>> by
>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>> > 2014-11-05 23:39:09,269 pool-13-thread-1 INFO
>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
>> > retrying send. Remaining retries = 4
>> > 2014-11-05 23:39:10,270 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 14 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:10,271 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:10,271 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:10,275 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:10,275 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:10,275 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:10,276 pool-13-thread-1 DEBUG
>> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
>> > topic Test
>> > 2014-11-05 23:39:10,276 pool-13-thread-1 INFO
>> > kafka.client.ClientUtils$-68: Fetching metadata from broker
>> > id:0,host:172.31.25.198,port:9092 with correlation id 15 for 1 topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:10,277 pool-13-thread-1 DEBUG
>> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
>> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
>> > (requested 102400).
>> > 2014-11-05 23:39:10,277 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
>> > producing
>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
>> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
>> > 2014-11-05 23:39:10,280 pool-13-thread-1 DEBUG
>> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
>> topic(s)
>> > Set(Test)
>> > 2014-11-05 23:39:10,280 pool-13-thread-1 WARN
>> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
>> > [{TopicMetadata for topic Test ->
>> > No partition metadata for topic Test due to
>> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
>> > kafka.common.LeaderNotAvailableException
>> > 2014-11-05 23:39:10,280 pool-13-thread-1 ERROR
>> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
>> by
>> > topic, partition due to: Failed to fetch topic metadata for topic: Test
>> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
>> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
>> > retrying send. Remaining retries = 3
>> >
>> >
>> > Brokers Logs -
>> >
>> > [2014-11-05 23:39:02,983] INFO Topic creation
>> > {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
>> > [2014-11-05 23:39:02,988] INFO [KafkaApi-0] Auto creation of topic Test
>> > with 2 partitions and replication factor 1 is successful!
>> > (kafka.server.KafkaApis)
>> > [2014-11-05 23:39:03,092] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> > [2014-11-05 23:39:03,191] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> > [2014-11-05 23:39:04,200] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> > [2014-11-05 23:39:04,205] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> > [2014-11-05 23:39:05,211] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> > [2014-11-05 23:39:05,218] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> > [2014-11-05 23:39:06,224] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> > [2014-11-05 23:39:06,230] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> > [2014-11-05 23:39:07,236] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> > [2014-11-05 23:39:07,241] INFO Closing socket connection to /
>> 172.31.25.198
>> > .
>> > (kafka.network.Processor)
>> >
>> >
>> >
>> >
>> > On Wed, Nov 5, 2014 at 3:05 PM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>> >
>> > > Sameer,
>> > >
>> > > Yes, this is the server log. But there seems no abnormal entries in
>> it,
>> > and
>> > > it does not cover the same time range as the producer client throwing
>> > > LeaderNotAvailableException
>> > > (it was 10/24, 14:30).
>> > >
>> > > The reason that I want to check the server log at that same reason is
>> > > that LeaderNotAvailableException
>> > > is set by the brokers while handling requests, and its server log will
>> > show
>> > > what is the cause it set this error code.
>> > >
>> > > Guozhang
>> > >
>> > > On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net>
>> wrote:
>> > >
>> > > > Hi Guozhang,
>> > > >
>> > > > This is the server.log -
>> > > >
>> > > > [2014-11-04 20:21:57,510] INFO Verifying properties
>> > > > (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,545] INFO Property advertised.host.name is
>> > > overridden
>> > > > to x.x.x.x (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,545] INFO Property broker.id is overridden to
>> 0
>> > > > (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,545] INFO Property controlled.shutdown.enable
>> is
>> > > > overridden to true (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,545] INFO Property host.name is overridden to
>> > > > 172.31.25.198 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is
>> > overridden
>> > > to
>> > > > false (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden to
>> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,546] INFO Property
>> > log.retention.check.interval.ms
>> > > is
>> > > > overridden to 60000 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,546] INFO Property log.retention.hours is
>> > overridden
>> > > > to 168 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is
>> overridden
>> > > to
>> > > > 536870912 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,546] INFO Property num.io.threads is overridden
>> > to 8
>> > > > (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,547] INFO Property num.network.threads is
>> > overridden
>> > > > to 2 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,547] INFO Property num.partitions is overridden
>> > to 2
>> > > > (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,547] INFO Property port is overridden to 9092
>> > > > (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,547] INFO Property socket.receive.buffer.bytes
>> is
>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,547] INFO Property socket.request.max.bytes is
>> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,547] INFO Property socket.send.buffer.bytes is
>> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is
>> overridden
>> > > to
>> > > > localhost:2181 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,548] INFO Property
>> > zookeeper.connection.timeout.ms
>> > > is
>> > > > overridden to 1000000 (kafka.utils.VerifiableProperties)
>> > > > [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
>> > > > (kafka.server.KafkaServer)
>> > > > [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to
>> > zookeeper
>> > > on
>> > > > localhost:2181 (kafka.server.KafkaServer)
>> > > > [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs' not
>> > found,
>> > > > creating it. (kafka.log.LogManager)
>> > > > [2014-11-04 20:21:57,748] INFO Starting log cleanup with a period of
>> > > 60000
>> > > > ms. (kafka.log.LogManager)
>> > > > [2014-11-04 20:21:57,752] INFO Starting log flusher with a default
>> > period
>> > > > of 9223372036854775807 ms. (kafka.log.LogManager)
>> > > > [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
>> > > > ip-172-31-25-198.us-west-1.compute.internal:9092.
>> > > (kafka.network.Acceptor)
>> > > > [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0], Started
>> > > > (kafka.network.SocketServer)
>> > > > [2014-11-04 20:21:57,852] INFO Will not load MX4J, mx4j-tools.jar is
>> > not
>> > > in
>> > > > the classpath (kafka.utils.Mx4jLoader$)
>> > > > [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
>> > > > (kafka.server.ZookeeperLeaderElector)
>> > > > [2014-11-04 20:21:57,967] INFO Registered broker 0 at path
>> > /brokers/ids/0
>> > > > with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
>> > > > [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
>> > > > (kafka.server.KafkaServer)
>> > > > [2014-11-04 20:21:58,050] INFO New leader is 0
>> > > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
>> > > > [2014-11-04 20:22:01,809] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:22:11,812] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:29,084] INFO Topic creation
>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
>> (kafka.admin.AdminUtils$)
>> > > > [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of topic
>> Test
>> > > > with 2 partitions and replication factor 1 is successful!
>> > > > (kafka.server.KafkaApis)
>> > > > [2014-11-04 20:30:29,280] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:29,301] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:30,310] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:30,315] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:31,320] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:31,325] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:32,331] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:32,336] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:33,343] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:33,347] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:34,356] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:34,360] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:35,366] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:35,372] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:36,377] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:36,382] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:37,387] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:37,394] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:38,399] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:38,403] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:39,410] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:39,414] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:40,419] INFO Closing socket connection to /
>> > > 172.31.25.198
>> > > > .
>> > > > (kafka.network.Processor)
>> > > > [2014-11-04 20:30:41,602] INFO Topic creation
>> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
>> (kafka.admin.AdminUtils$)
>> > > >
>> > > >
>> > > >
>> > > > thanks
>> > > >
>> > > >
>> > > > On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <wa...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > The server.log should be the broker logs.
>> > > > >
>> > > > > Guozhang
>> > > > >
>> > > > > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <sy...@ailive.net>
>> > > wrote:
>> > > > >
>> > > > > > I do not see any other logs besides these.
>> > > > > > I checked the log4j.properties and where the logs are getting
>> > stored.
>> > > > > > The only other logs (apart from the controller logs) in the
>> > directory
>> > > > are
>> > > > > > zookeeper-gc.log,kafkaServer-gc.log, server.log and
>> > state-change.log
>> > > > > >
>> > > > > > Thanks
>> > > > > >
>> > > > > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <
>> wangguoz@gmail.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > Sorry for getting late on this Sameer.
>> > > > > > >
>> > > > > > > The logs you showed above are from controller logs, do you see
>> > > > brokers
>> > > > > > logs
>> > > > > > > besides that?
>> > > > > > >
>> > > > > > > Guozhang
>> > > > > > >
>> > > > > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <
>> syami@ailive.net>
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Guozhang,
>> > > > > > > >
>> > > > > > > > Is there any clue in the above data as to what is wrong?
>> > > > > > > >
>> > > > > > > > thanks
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <
>> syami@ailive.net
>> > >
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > This is the one from the file specified in
>> log4j.properties -
>> > > > > > > > >
>> > > > > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on
>> > > > Controller
>> > > > > > 0]:
>> > > > > > > > > Topic change listener fired for path /brokers/topics with
>> > > > children
>> > > > > > Test
>> > > > > > > > >
>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener on
>> > > Controller
>> > > > > 0]:
>> > > > > > > New
>> > > > > > > > > topics: [Set(Test)], deleted topics: [Set()], new
>> partition
>> > > > replica
>> > > > > > > > > assignment [Map([Test,0] -> List(0), [Test,1] -> List(0))]
>> > > > > > > > >
>> (kafka.controller.PartitionStateMachine$TopicChangeListener)
>> > > > > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New topic
>> > > creation
>> > > > > > > > callback
>> > > > > > > > > for [Test,0],[Test,1] (kafka.controller.KafkaController)
>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New
>> partition
>> > > > > creation
>> > > > > > > > > callback for [Test,0],[Test,1]
>> > > (kafka.controller.KafkaController)
>> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Partition state machine on
>> > > > > Controller
>> > > > > > > 0]:
>> > > > > > > > > Invoking state change to NewPartition for partitions
>> > > > > > [Test,0],[Test,1]
>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>> > > > > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine on
>> > > > controller
>> > > > > > 0]:
>> > > > > > > > > Invoking state change to NewReplica for replicas
>> > > > > > > > >
>> > > > >
>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
>> > > > > > > > > [2014-10-28 22:38:36,111] INFO [Partition state machine on
>> > > > > Controller
>> > > > > > > 0]:
>> > > > > > > > > Invoking state change to OnlinePartition for partitions
>> > > > > > > [Test,0],[Test,1]
>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>> > > > > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state machine
>> on
>> > > > > > Controller
>> > > > > > > > 0]:
>> > > > > > > > > Live assigned replicas for partition [Test,0] are:
>> [List(0)]
>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>> > > > > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state machine
>> on
>> > > > > > Controller
>> > > > > > > > 0]:
>> > > > > > > > > Initializing leader and isr for partition [Test,0] to
>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>> > > > > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state machine
>> on
>> > > > > > Controller
>> > > > > > > > 0]:
>> > > > > > > > > Live assigned replicas for partition [Test,1] are:
>> [List(0)]
>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>> > > > > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state machine
>> on
>> > > > > > Controller
>> > > > > > > > 0]:
>> > > > > > > > > Initializing leader and isr for partition [Test,1] to
>> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
>> > > > > > > > > (kafka.controller.PartitionStateMachine)
>> > > > > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine on
>> > > > controller
>> > > > > > 0]:
>> > > > > > > > > Invoking state change to OnlineReplica for replicas
>> > > > > > > > >
>> > > > >
>> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
>> > > > > > > > > (kafka.controller.ReplicaStateMachine)
>> > > > > > > > > [2014-10-28 22:39:28,160] ERROR
>> > > > > > [Controller-0-to-broker-0-send-thread],
>> > > > > > > > > Controller 0's connection to broker
>> > id:0,host:x.x.x.x,port:9092
>> > > > was
>> > > > > > > > > unsuccessful (kafka.controller.RequestSendThread)
>> > > > > > > > > java.net.ConnectException: Connection timed out
>> > > > > > > > > at sun.nio.ch.Net.connect0(Native Method)
>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
>> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:449)
>> > > > > > > > > at
>> > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
>> > > > > > > > > at
>> > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>> > > > > > > > > at
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
>> > > > > > > > > at
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
>> > > > > > > > > at
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
>> > > > > > > > > at
>> > > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>> > > > > > > > > [2014-10-28 22:39:28,460] ERROR
>> > > > > > [Controller-0-to-broker-0-send-thread],
>> > > > > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr request
>> with
>> > > > > > > correlation
>> > > > > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092. Reconnecting
>> to
>> > > > broker.
>> > > > > > > > > (kafka.controller.RequestSendThread)
>> > > > > > > > > java.nio.channels.ClosedChannelException
>> > > > > > > > > at
>> > > > kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
>> > > > > > > > > at
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
>> > > > > > > > > at
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
>> > > > > > > > > at
>> > > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
>> > > > > > > > >
>> > > > > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
>> > > > wangguoz@gmail.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Sameer, the one you sent me is from the producer logs,
>> not
>> > the
>> > > > > > broker
>> > > > > > > > >> logs.
>> > > > > > > > >> And /tmp/kafka-logs actually store the message log (i.e.
>> the
>> > > > > "data")
>> > > > > > > > >> instead of the server logs. Their locations should be
>> > > specified
>> > > > in
>> > > > > > > your
>> > > > > > > > >> log4j.properties file.
>> > > > > > > > >>
>> > > > > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <
>> > > syami@ailive.net>
>> > > > > > > wrote:
>> > > > > > > > >>
>> > > > > > > > >> > Is this what you are looking for -
>> > > > > > > > >> >
>> > > > > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
>> > > > 172.31.16.31:9092
>> > > > > > for
>> > > > > > > > >> > producing
>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
>> > > > > > > 172.31.16.31:9092
>> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
>> > metadata
>> > > > for
>> > > > > 1
>> > > > > > > > >> topic(s)
>> > > > > > > > >> > Set(Test)
>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
>> > fetching
>> > > > > > metadata
>> > > > > > > > >> > [{TopicMetadata for topic Test ->
>> > > > > > > > >> > No partition metadata for topic Test due to
>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
>> > [Test]:
>> > > > > class
>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
>> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
>> > > > partition
>> > > > > > info
>> > > > > > > > for
>> > > > > > > > >> > topic Test
>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
>> > broker
>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 9
>> > for 1
>> > > > > > > topic(s)
>> > > > > > > > >> > Set(Test)
>> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
>> > > > > SO_TIMEOUT =
>> > > > > > > > 10000
>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
>> > > > SO_SNDBUF
>> > > > > =
>> > > > > > > > 102400
>> > > > > > > > >> > (requested 102400).
>> > > > > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
>> > > > 172.31.16.31:9092
>> > > > > > for
>> > > > > > > > >> > producing
>> > > > > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
>> > > > > > > 172.31.16.31:9092
>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
>> > metadata
>> > > > for
>> > > > > 1
>> > > > > > > > >> topic(s)
>> > > > > > > > >> > Set(Test)
>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
>> > fetching
>> > > > > > metadata
>> > > > > > > > >> > [{TopicMetadata for topic Test ->
>> > > > > > > > >> > No partition metadata for topic Test due to
>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
>> > [Test]:
>> > > > > class
>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
>> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
>> > > collate
>> > > > > > > > messages
>> > > > > > > > >> by
>> > > > > > > > >> > topic, partition due to: Failed to fetch topic metadata
>> > for
>> > > > > topic:
>> > > > > > > > Test
>> > > > > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off
>> for
>> > > 1000
>> > > > > ms
>> > > > > > > > before
>> > > > > > > > >> > retrying send. Remaining retries = 6
>> > > > > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
>> > broker
>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 10
>> > for
>> > > 1
>> > > > > > > topic(s)
>> > > > > > > > >> > Set(Test)
>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
>> > > > > SO_TIMEOUT =
>> > > > > > > > 10000
>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
>> > > > SO_SNDBUF
>> > > > > =
>> > > > > > > > 102400
>> > > > > > > > >> > (requested 102400).
>> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
>> > > > 172.31.16.31:9092
>> > > > > > for
>> > > > > > > > >> > producing
>> > > > > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
>> > > > > > > 172.31.16.31:9092
>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
>> > metadata
>> > > > for
>> > > > > 1
>> > > > > > > > >> topic(s)
>> > > > > > > > >> > Set(Test)
>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
>> > fetching
>> > > > > > metadata
>> > > > > > > > >> > [{TopicMetadata for topic Test ->
>> > > > > > > > >> > No partition metadata for topic Test due to
>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
>> > [Test]:
>> > > > > class
>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
>> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
>> > > > partition
>> > > > > > info
>> > > > > > > > for
>> > > > > > > > >> > topic Test
>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
>> > broker
>> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 11
>> > for
>> > > 1
>> > > > > > > topic(s)
>> > > > > > > > >> > Set(Test)
>> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
>> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
>> > > > > SO_TIMEOUT =
>> > > > > > > > 10000
>> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
>> > > > SO_SNDBUF
>> > > > > =
>> > > > > > > > 102400
>> > > > > > > > >> > (requested 102400).
>> > > > > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
>> > > > 172.31.16.31:9092
>> > > > > > for
>> > > > > > > > >> > producing
>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
>> > > > > > > 172.31.16.31:9092
>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
>> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
>> > metadata
>> > > > for
>> > > > > 1
>> > > > > > > > >> topic(s)
>> > > > > > > > >> > Set(Test)
>> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
>> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
>> > fetching
>> > > > > > metadata
>> > > > > > > > >> > [{TopicMetadata for topic Test ->
>> > > > > > > > >> > No partition metadata for topic Test due to
>> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
>> > [Test]:
>> > > > > class
>> > > > > > > > >> > kafka.common.LeaderNotAvailableException
>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
>> > > collate
>> > > > > > > > messages
>> > > > > > > > >> by
>> > > > > > > > >> > topic, partition due to: Failed to fetch topic metadata
>> > for
>> > > > > topic:
>> > > > > > > > Test
>> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
>> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off
>> for
>> > > 1000
>> > > > > ms
>> > > > > > > > before
>> > > > > > > > >> > retrying send. Remaining retries = 5
>> > > > > > > > >> >
>> > > > > > > > >> > As for /tmp/kafka-logs, I can see only these files -
>> > > > > > > > >> > recovery-point-offset-checkpoint
>> > > > replication-offset-checkpoint
>> > > > > > > > >> >
>> > > > > > > > >> > I do not see the usual Kafka log files here.
>> > > > > > > > >> >
>> > > > > > > > >> > thanks
>> > > > > > > > >> > Sameer
>> > > > > > > > >> >
>> > > > > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
>> > > > > > wangguoz@gmail.com>
>> > > > > > > > >> wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > Sameer, could you check broker logs and see if it has
>> > > > similar
>> > > > > > > > >> > > LeaderNotAvailable exceptions?
>> > > > > > > > >> > >
>> > > > > > > > >> > > Guozhang
>> > > > > > > > >> > >
>> > > > > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
>> > > > > syami@ailive.net
>> > > > > > >
>> > > > > > > > >> wrote:
>> > > > > > > > >> > >
>> > > > > > > > >> > > > Yes. I can telnet from outside using the public ip
>> of
>> > > the
>> > > > > > > instance
>> > > > > > > > >> and
>> > > > > > > > >> > > port
>> > > > > > > > >> > > > of 9092
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <
>> > > > > > > > >> roger.hoover@gmail.com
>> > > > > > > > >> > >
>> > > > > > > > >> > > > wrote:
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > > Just a guess but could it be a firewall issue?
>> Did
>> > > you
>> > > > > > enable
>> > > > > > > > >> > > > connections
>> > > > > > > > >> > > > > to port 9092 from outside EC2 in a security
>> group?
>> > > Can
>> > > > > you
>> > > > > > > > >> telnet to
>> > > > > > > > >> > > > each
>> > > > > > > > >> > > > > broker IP and port?
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami <
>> > > > > > > syami@ailive.net
>> > > > > > > > >
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > > There was a typo earlier.
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > This is the output -
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Topic:Test PartitionCount:2
>> ReplicationFactor:1
>> > > > > > > Configs:
>> > > > > > > > >> > > > > > Topic: Test Partition: 0 Leader:
>> 0
>> > > > > > > Replicas:
>> > > > > > > > 0
>> > > > > > > > >> > > > Isr: 0
>> > > > > > > > >> > > > > > Topic: Test Partition: 1 Leader:
>> 0
>> > > > > > > Replicas:
>> > > > > > > > 0
>> > > > > > > > >> > > > Isr: 0
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > thanks
>> > > > > > > > >> > > > > > Sameer
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang Wang
>> <
>> > > > > > > > >> wangguoz@gmail.com
>> > > > > > > > >> > >
>> > > > > > > > >> > > > > wrote:
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > > Sameer,
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > The topic name is not consistent from your
>> > > describe
>> > > > > > > outputs.
>> > > > > > > > >> > > Normally
>> > > > > > > > >> > > > > > they
>> > > > > > > > >> > > > > > > should be either
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Topic:*Test* PartitionCount:2
>> > > > > ReplicationFactor:1
>> > > > > > > > >> > > Configs:
>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 0
>> > Leader: 0
>> > > > > > > > >> Replicas: 0
>> > > > > > > > >> > > > > > Isr: 0
>> > > > > > > > >> > > > > > > Topic: *Test* Partition: 1
>> > Leader: 0
>> > > > > > > > >> Replicas: 0
>> > > > > > > > >> > > > > > Isr: 0
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Or
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
>> > > > > > > > ReplicationFactor:1
>> > > > > > > > >> > > > > Configs:
>> > > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 0
>> > > > Leader:
>> > > > > 0
>> > > > > > > > >> > > Replicas:
>> > > > > > > > >> > > > 0
>> > > > > > > > >> > > > > > > Isr: 0
>> > > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 1
>> > > > Leader:
>> > > > > 0
>> > > > > > > > >> > > Replicas:
>> > > > > > > > >> > > > 0
>> > > > > > > > >> > > > > > > Isr: 0
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Did you manually modify the topic name in ZK?
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Guozhang
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer Yami
>> <
>> > > > > > > > >> syami@ailive.net>
>> > > > > > > > >> > > > wrote:
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > > Hi Guozhang,
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > I was wondering if there is anything wrong
>> in
>> > > the
>> > > > > > above
>> > > > > > > > >> > > --describe
>> > > > > > > > >> > > > > > data.
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > thanks
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer
>> Yami <
>> > > > > > > > >> syami@ailive.net
>> > > > > > > > >> > >
>> > > > > > > > >> > > > > wrote:
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > > This is what I get -
>> > > > > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper
>> > > localhost:2181
>> > > > > > > > --describe
>> > > > > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
>> > > > > > > > >> ReplicationFactor:1
>> > > > > > > > >> > > > > > Configs:
>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 0
>> > > Leader: 0
>> > > > > > > > >> > Replicas: 0
>> > > > > > > > >> > > > > > > Isr: 0
>> > > > > > > > >> > > > > > > > > Topic: Test Partition: 1
>> > > Leader: 0
>> > > > > > > > >> > Replicas: 0
>> > > > > > > > >> > > > > > > Isr: 0
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > The topics are being created at run-time.
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > thanks
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM, Guozhang
>> > > Wang <
>> > > > > > > > >> > > > wangguoz@gmail.com
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > > > wrote:
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >> Could you use the topic tools
>> "--describe"
>> > to
>> > > > > check
>> > > > > > > if
>> > > > > > > > >> the
>> > > > > > > > >> > > topic
>> > > > > > > > >> > > > > > > Test's
>> > > > > > > > >> > > > > > > > >> partitions leaders are available?
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > >
>> > > > > http://kafka.apache.org/documentation.html#basic_ops_add_topic
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > > > > > > > >> Guozhang
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM, Sameer
>> > Yami
>> > > <
>> > > > > > > > >> > > syami@ailive.net>
>> > > > > > > > >> > > > > > > wrote:
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > > > > > > > >> > That is not the case.
>> > > > > > > > >> > > > > > > > >> > This detailed error is below. Due to
>> this
>> > > > > error ,
>> > > > > > > the
>> > > > > > > > >> > > producer
>> > > > > > > > >> > > > > is
>> > > > > > > > >> > > > > > > not
>> > > > > > > > >> > > > > > > > >> able
>> > > > > > > > >> > > > > > > > >> > to send messages
>> > > > > > > > >> > > > > > > > >> >
>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG
>> > > > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-52:
>> > > > > > > > >> > > > > > > > >> > Getting broker partition info for
>> topic
>> > > Test
>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main INFO
>> > > > > > > > >> > > > > kafka.client.ClientUtils$-68:
>> > > > > > > > >> > > > > > > > >> Fetching
>> > > > > > > > >> > > > > > > > >> > metadata from broker
>> > > > > > > > id:0,host:54.215.200.61,port:9092
>> > > > > > > > >> > with
>> > > > > > > > >> > > > > > > > correlation
>> > > > > > > > >> > > > > > > > >> id
>> > > > > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG
>> > > > > > > > >> > > > > > > kafka.network.BlockingChannel-52:
>> > > > > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT = 10000
>> > > > > (requested
>> > > > > > > > >> 10000),
>> > > > > > > > >> > > > > > SO_RCVBUF
>> > > > > > > > >> > > > > > > =
>> > > > > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF =
>> 102400
>> > > > > > > (requested
>> > > > > > > > >> > > 102400).
>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main INFO
>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
>> > > > > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092 for
>> > > > producing
>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main INFO
>> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
>> > > > > > > > >> > > > > > > > >> > Disconnecting from 54.215.200.61:9092
>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG
>> > > > > > > > >> > > > > kafka.client.ClientUtils$-52:
>> > > > > > > > >> > > > > > > > >> > Successfully fetched metadata for 1
>> > > topic(s)
>> > > > > > > > Set(Test)
>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main WARN
>> > > > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-83:
>> > > > > > > > >> > > > > > > > >> > Error while fetching metadata
>> > > [{TopicMetadata
>> > > > > for
>> > > > > > > > topic
>> > > > > > > > >> > Test
>> > > > > > > > >> > > > ->
>> > > > > > > > >> > > > > > > > >> > No partition metadata for topic Test
>> due
>> > to
>> > > > > > > > >> > > > > > > > >> >
>> > kafka.common.LeaderNotAvailableException}]
>> > > > for
>> > > > > > > topic
>> > > > > > > > >> > [Test]:
>> > > > > > > > >> > > > > class
>> > > > > > > > >> > > > > > > > >> >
>> kafka.common.LeaderNotAvailableException
>> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR
>> > > > > > > > >> > > > > > > > >> >
>> > > kafka.producer.async.DefaultEventHandler-97:
>> > > > > > Failed
>> > > > > > > > to
>> > > > > > > > >> > > collate
>> > > > > > > > >> > > > > > > > messages
>> > > > > > > > >> > > > > > > > >> by
>> > > > > > > > >> > > > > > > > >> > topic, partition due to: Failed to
>> fetch
>> > > > topic
>> > > > > > > > metadata
>> > > > > > > > >> > for
>> > > > > > > > >> > > > > topic:
>> > > > > > > > >> > > > > > > > Test
>> > > > > > > > >> > > > > > > > >> >
>> > > > > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM,
>> Guozhang
>> > > > Wang
>> > > > > <
>> > > > > > > > >> > > > > > wangguoz@gmail.com>
>> > > > > > > > >> > > > > > > > >> wrote:
>> > > > > > > > >> > > > > > > > >> >
>> > > > > > > > >> > > > > > > > >> > > Could you see if this is the case:
>> > > > > > > > >> > > > > > > > >> > >
>> > > > > > > > >> > > > > > > > >> > >
>> > > > > > > > >> > > > > > > > >> > >
>> > > > > > > > >> > > > > > > > >> >
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
>> > > > > > > > >> > > > > > > > >> > > ?
>> > > > > > > > >> > > > > > > > >> > >
>> > > > > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM,
>> > Sameer
>> > > > > Yami <
>> > > > > > > > >> > > > > syami@ailive.net
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > > >> wrote:
>> > > > > > > > >> > > > > > > > >> > >
>> > > > > > > > >> > > > > > > > >> > > > Hi,
>> > > > > > > > >> > > > > > > > >> > > >
>> > > > > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper test
>> > setup
>> > > > on a
>> > > > > > > > single
>> > > > > > > > >> EC2
>> > > > > > > > >> > > > > > machine.
>> > > > > > > > >> > > > > > > > The
>> > > > > > > > >> > > > > > > > >> > > > host.name is the private ip
>> address
>> > > and
>> > > > > the
>> > > > > > > > >> > > > > > > advertised.host.name
>> > > > > > > > >> > > > > > > > is
>> > > > > > > > >> > > > > > > > >> > the
>> > > > > > > > >> > > > > > > > >> > > > public address in
>> server.properties.
>> > > > > > > > >> > > > > > > > >> > > >
>> > > > > > > > >> > > > > > > > >> > > > We are trying to publish from
>> outside
>> > > EC2
>> > > > > > using
>> > > > > > > > the
>> > > > > > > > >> > > public
>> > > > > > > > >> > > > > ip
>> > > > > > > > >> > > > > > of
>> > > > > > > > >> > > > > > > > the
>> > > > > > > > >> > > > > > > > >> > EC2
>> > > > > > > > >> > > > > > > > >> > > > machine but get this error on the
>> > > client
>> > > > > > side -
>> > > > > > > > >> > > > > > > > >> > > > No partition metadata for topic
>> test
>> > > due
>> > > > to
>> > > > > > > > >> > > > > > > > >> > > >
>> > > > kafka.common.LeaderNotAvailableException}]
>> > > > > > for
>> > > > > > > > >> topic
>> > > > > > > > >> > > > [test]:
>> > > > > > > > >> > > > > > > class
>> > > > > > > > >> > > > > > > > >> > > >
>> > > kafka.common.LeaderNotAvailableException
>> > > > > > > > >> > > > > > > > >> > > >
>> > > > > > > > >> > > > > > > > >> > > > The zookeeper has the public ip
>> > stored
>> > > in
>> > > > > > > > >> > > /brokers/ids/0.
>> > > > > > > > >> > > > > > > > >> > > > We have tried multiple
>> combinations
>> > but
>> > > > it
>> > > > > > > works
>> > > > > > > > >> ONLY
>> > > > > > > > >> > > when
>> > > > > > > > >> > > > > the
>> > > > > > > > >> > > > > > > > >> > advertised
>> > > > > > > > >> > > > > > > > >> > > > host name is not used and the
>> client
>> > is
>> > > > > > > internal
>> > > > > > > > to
>> > > > > > > > >> > EC2.
>> > > > > > > > >> > > > > > > > >> > > >
>> > > > > > > > >> > > > > > > > >> > > > Please let me know what the
>> problem
>> > > could
>> > > > > be.
>> > > > > > > > >> > > > > > > > >> > > >
>> > > > > > > > >> > > > > > > > >> > > > thanks
>> > > > > > > > >> > > > > > > > >> > > > Sameer
>> > > > > > > > >> > > > > > > > >> > > >
>> > > > > > > > >> > > > > > > > >> > >
>> > > > > > > > >> > > > > > > > >> > >
>> > > > > > > > >> > > > > > > > >> > >
>> > > > > > > > >> > > > > > > > >> > > --
>> > > > > > > > >> > > > > > > > >> > > -- Guozhang
>> > > > > > > > >> > > > > > > > >> > >
>> > > > > > > > >> > > > > > > > >> >
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > > > > > > > >> --
>> > > > > > > > >> > > > > > > > >> -- Guozhang
>> > > > > > > > >> > > > > > > > >>
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > --
>> > > > > > > > >> > > > > > > -- Guozhang
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > >
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >> > > --
>> > > > > > > > >> > > -- Guozhang
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >> --
>> > > > > > > > >> -- Guozhang
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > -- Guozhang
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > -- Guozhang
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -- Guozhang
>> > >
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
Re: Cannot connect to Kafka from outside of EC2
Posted by Sameer Yami <sy...@ailive.net>.
The version is kafka_2.10-0.8.1.1. It is not the latest trunk.
Will try enabling debug version.
thanks
On Thu, Nov 6, 2014 at 9:37 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Sameer,
>
> The server logs do not contain any non-INFO logs, which is a bit wired. Did
> you deploy the current trunk of Kafka? Also could you enable DEBUG level
> logging on Kafka brokers?
>
> Guozhang
>
> On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami <sy...@ailive.net> wrote:
>
> > The server.log was taken separately.
> > We ran the test again and the server and producer logs are below (to get
> > same timings).
> >
> >
> > Thanks!
> >
> >
> >
> ------------------------------------------------------------------------------------
> >
> >
> >
> > Producer Logs -
> >
> >
> > 2014-11-05 23:38:58,693
> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> sessionid:
> > 0x1498251e8680002 after 0ms
> > 2014-11-05 23:39:00,695
> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> sessionid:
> > 0x1498251e8680002 after 0ms
> > 2014-11-05 23:39:02,696
> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> sessionid:
> > 0x1498251e8680002 after 0ms
> > 2014-11-05 23:39:02,828 pool-13-thread-2 INFO
> > kafka.utils.VerifiableProperties-68: Verifying properties
> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > kafka.utils.VerifiableProperties-68: Property auto.commit.interval.ms is
> > overridden to 1000
> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > kafka.utils.VerifiableProperties-68: Property auto.offset.reset is
> > overridden to smallest
> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > kafka.utils.VerifiableProperties-68: Property consumer.timeout.ms is
> > overridden to 10
> > 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> > kafka.utils.VerifiableProperties-68: Property group.id is overridden to
> > TestCheck
> > 2014-11-05 23:39:02,830 pool-13-thread-2 WARN
> > kafka.utils.VerifiableProperties-83: Property serializer.class is not
> valid
> > 2014-11-05 23:39:02,830 pool-13-thread-2 INFO
> > kafka.utils.VerifiableProperties-68: Property zookeeper.connect is
> > overridden to 172.31.25.198:2181
> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Connecting to
> > zookeeper instance at 172.31.25.198:2181
> > 2014-11-05 23:39:02,831 pool-13-thread-2 DEBUG
> > org.I0Itec.zkclient.ZkConnection-63: Creating new ZookKeeper instance to
> > connect to 172.31.25.198:2181.
> > 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> > org.apache.zookeeper.ZooKeeper-379: Initiating client connection,
> > connectString=172.31.25.198:2181 sessionTimeout=6000
> > watcher=org.I0Itec.zkclient.ZkClient@3903b165
> > 2014-11-05 23:39:02,831 ZkClient-EventThread-29-172.31.25.198:2181 INFO
> > org.I0Itec.zkclient.ZkEventThread-64: Starting ZkClient event thread.
> > 2014-11-05 23:39:02,831 pool-13-thread-1 INFO
> > kafka.utils.VerifiableProperties-68: Verifying properties
> > 2014-11-05 23:39:02,836 pool-13-thread-2-SendThread() INFO
> > org.apache.zookeeper.ClientCnxn-1061: Opening socket connection to
> server /
> > 172.31.25.198:2181
> > 2014-11-05 23:39:02,836 pool-13-thread-1 WARN
> > kafka.utils.VerifiableProperties-83: Property batch.size is not valid
> > 2014-11-05 23:39:02,832 pool-13-thread-2 DEBUG
> > org.I0Itec.zkclient.ZkClient-878: Awaiting connection to Zookeeper server
> > 2014-11-05 23:39:02,836 pool-13-thread-1 INFO
> > kafka.utils.VerifiableProperties-68: Property message.send.max.retries is
> > overridden to 10
> > 2014-11-05 23:39:02,836 pool-13-thread-2 DEBUG
> > org.I0Itec.zkclient.ZkClient-628: Waiting for keeper state SyncConnected
> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> > kafka.utils.VerifiableProperties-68: Property metadata.broker.list is
> > overridden to 172.31.25.198:9092
> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> > kafka.utils.VerifiableProperties-68: Property retry.backoff.ms is
> > overridden to 1000
> > 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> > kafka.utils.VerifiableProperties-68: Property serializer.class is
> > overridden to kafka.serializer.StringEncoder
> > 2014-11-05 23:39:02,837
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > INFO org.apache.zookeeper.ClientCnxn-950: Socket connection established
> to
> > ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
> initiating
> > session
> > 2014-11-05 23:39:02,838
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-999: Session establishment request
> > sent on ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181
> > 2014-11-05 23:39:02,837 pool-13-thread-1 WARN
> > kafka.utils.VerifiableProperties-83: Property zk.connectiontimeout.ms is
> > not valid
> > 2014-11-05 23:39:02,841
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > INFO org.apache.zookeeper.ClientCnxn-739: Session establishment complete
> on
> > server ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
> > sessionid = 0x1498251e8680003, negotiated timeout = 6000
> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> > state:SyncConnected type:None path:null
> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread INFO
> > org.I0Itec.zkclient.ZkClient-449: zookeeper state changed (SyncConnected)
> > 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
> > 2014-11-05 23:39:02,841 pool-13-thread-2 DEBUG
> > org.I0Itec.zkclient.ZkClient-638: State is SyncConnected
> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> > kafka.utils.KafkaScheduler-52: Initializing task scheduler.
> > 2014-11-05 23:39:02,842 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting auto
> > committer every 1000 ms
> > 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> > kafka.utils.KafkaScheduler-52: Scheduling task kafka-consumer-autocommit
> > with initial delay 1000 ms and period 1000 ms.
> > 2014-11-05 23:39:02,843 pool-13-thread-2 DEBUG
> > kafka.consumer.ZookeeperConsumerConnector-52:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], entering consume
> > 2014-11-05 23:39:02,843 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin registering
> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> > 2014-11-05 23:39:02,846
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 1,1 replyHeader:: 1,29,-101 request::
> >
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> > response::
> > 2014-11-05 23:39:02,854
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 2,1 replyHeader:: 2,30,-101 request::
> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> > 2014-11-05 23:39:02,858
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 3,1 replyHeader:: 3,31,0 request::
> > '/consumers/TestCheck,,v{s{31,s{'world,'anyone}}},0 response::
> > '/consumers/TestCheck
> > 2014-11-05 23:39:02,861
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 4,1 replyHeader:: 4,32,0 request::
> > '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> > '/consumers/TestCheck/ids
> > 2014-11-05 23:39:02,866
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 5,1 replyHeader:: 5,33,0 request::
> >
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> > response::
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> > 2014-11-05 23:39:02,867 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end registering
> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> > kafka.consumer.ZookeeperConsumerConnector-52:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Adding topicThreadId
> > (Test,TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0) and queue []
> to
> > topicThreadIdAndQueues data structure
> > 2014-11-05 23:39:02,868
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting watcher
> > executor thread for consumer
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> > 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> > kafka.consumer.ZookeeperConsumerConnector-52:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], adding topic Test
> and
> > 1 streams to map.
> > 2014-11-05 23:39:02,870
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 6,3 replyHeader:: 6,33,0 request::
> > '/consumers/TestCheck/ids,T response::
> > s{32,32,1415230742859,1415230742859,0,1,0,0,0,1,33}
> > 2014-11-05 23:39:02,872
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 7,8 replyHeader:: 7,33,0 request::
> > '/consumers/TestCheck/ids,T response::
> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> > 2014-11-05 23:39:02,873
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 8,3 replyHeader:: 8,33,-101 request::
> > '/brokers/topics/Test,T response::
> > 2014-11-05 23:39:02,873 pool-13-thread-2 DEBUG
> > org.I0Itec.zkclient.ZkClient-137: Subscribed data changes for
> > /brokers/topics/Test
> > 2014-11-05 23:39:02,873 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > 2014-11-05 23:39:02,875 pool-13-thread-1 DEBUG
> > kafka.producer.async.DefaultEventHandler-52: Handling 1 events
> > 2014-11-05 23:39:02,877
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 9,8 replyHeader:: 9,33,0 request::
> > '/brokers/ids,F response:: v{'0}
> > 2014-11-05 23:39:02,879
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 10,4 replyHeader:: 10,33,0 request::
> > '/brokers/ids/0,F response::
> >
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >
> > 2014-11-05 23:39:02,887
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 11,4 replyHeader:: 11,33,0 request::
> >
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > response::
> >
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> >
> > 2014-11-05 23:39:02,896
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 12,8 replyHeader:: 12,33,0 request::
> > '/consumers/TestCheck/ids,T response::
> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> > 2014-11-05 23:39:02,898
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 13,4 replyHeader:: 13,33,0 request::
> >
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > response::
> >
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> >
> > 2014-11-05 23:39:02,908
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 14,8 replyHeader:: 14,33,0 request::
> > '/brokers/ids,F response:: v{'0}
> > 2014-11-05 23:39:02,910
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 15,4 replyHeader:: 15,33,0 request::
> > '/brokers/ids/0,F response::
> >
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >
> > 2014-11-05 23:39:02,917 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 0 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:02,917
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 16,4 replyHeader:: 16,33,-101 request::
> > '/brokers/topics/Test,T response::
> > 2014-11-05 23:39:02,920 pool-13-thread-2 DEBUG kafka.utils.ZkUtils$-52:
> > Partition map for /brokers/topics/Test is Map()
> > 2014-11-05 23:39:02,920 pool-13-thread-2 INFO
> > kafka.consumer.ConsumerFetcherManager-68:
> > [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > kafka.consumer.ConsumerFetcherManager-68:
> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > kafka.consumer.ConsumerFetcherManager-68:
> > [ConsumerFetcherManager-1415230742842] All connections stopped
> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all relevant
> > queues for this fetcher
> > 2014-11-05 23:39:02,921 pool-13-thread-2 DEBUG
> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
> > this consumer iterator
> > 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
> > chunks in all the consumer message iterators
> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
> offsets
> > after clearing the fetcher queues
> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing partition
> > ownership
> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
> following
> > partitions: ArrayBuffer() for topic Test with consumers:
> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> > 2014-11-05 23:39:02,922 pool-13-thread-2 WARN
> > kafka.consumer.ZookeeperConsumerConnector-83:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], No broker partitions
> > consumed by consumer thread
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 for topic Test
> > 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> > kafka.consumer.ZookeeperConsumerConnector-52:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per topic
> > cache Map(Test -> ArrayBuffer())
> > 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> > kafka.consumer.ZookeeperConsumerConnector-52:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per topic
> > cache Map(Test ->
> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> > 2014-11-05 23:39:02,923 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
> > 2014-11-05 23:39:02,927 pool-13-thread-2 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > 2014-11-05 23:39:02,928
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > Starting
> > 2014-11-05 23:39:02,930 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:02,930 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:02,985
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-783: Got notification
> > sessionid:0x1498251e8680003
> > 2014-11-05 23:39:02,986
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-800: Got WatchedEvent
> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test for
> > sessionid 0x1498251e8680003
> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> > org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> > state:SyncConnected type:NodeCreated path:/brokers/topics/Test
> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> > org.I0Itec.zkclient.ZkEventThread-88: New event: ZkEvent[Data of
> > /brokers/topics/Test changed sent to
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> > ]
> > 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> > org.I0Itec.zkclient.ZkClient-395: Leaving process event
> > 2014-11-05 23:39:02,988 ZkClient-EventThread-29-172.31.25.198:2181 DEBUG
> > org.I0Itec.zkclient.ZkEventThread-69: Delivering event #1 ZkEvent[Data of
> > /brokers/topics/Test changed sent to
> >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> > ]
> > 2014-11-05 23:39:02,991
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 17,3 replyHeader:: 17,37,0 request::
> > '/brokers/topics/Test,T response::
> > s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> > 2014-11-05 23:39:02,993
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 18,4 replyHeader:: 18,37,0 request::
> > '/brokers/topics/Test,T response::
> >
> >
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> >
> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181 INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Topic info for path
> > /brokers/topics/Test changed to
> > {"version":1,"partitions":{"1":[0],"0":[0]}}, triggering rebalance
> > 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181 DEBUG
> > org.I0Itec.zkclient.ZkEventThread-79: Delivering event #1 done
> > 2014-11-05 23:39:02,995
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > 2014-11-05 23:39:02,996
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 19,8 replyHeader:: 19,37,0 request::
> > '/brokers/ids,F response:: v{'0}
> > 2014-11-05 23:39:02,998
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 20,4 replyHeader:: 20,37,0 request::
> > '/brokers/ids/0,F response::
> >
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >
> > 2014-11-05 23:39:03,008
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 21,4 replyHeader:: 21,37,0 request::
> >
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > response::
> >
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> >
> > 2014-11-05 23:39:03,014
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 22,8 replyHeader:: 22,37,0 request::
> > '/consumers/TestCheck/ids,T response::
> > v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> > 2014-11-05 23:39:03,018
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 23,4 replyHeader:: 23,37,0 request::
> >
> >
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> > response::
> >
> >
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
> >
> > 2014-11-05 23:39:03,041
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 24,8 replyHeader:: 24,37,0 request::
> > '/brokers/ids,F response:: v{'0}
> > 2014-11-05 23:39:03,043
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 25,4 replyHeader:: 25,37,0 request::
> > '/brokers/ids/0,F response::
> >
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >
> > 2014-11-05 23:39:03,058
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 26,4 replyHeader:: 26,37,0 request::
> > '/brokers/topics/Test,T response::
> >
> >
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> >
> > 2014-11-05 23:39:03,074
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.utils.ZkUtils$-52: Partition map for /brokers/topics/Test is Map(1
> ->
> > List(0), 0 -> List(0))
> > 2014-11-05 23:39:03,075
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ConsumerFetcherManager-68:
> > [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
> > 2014-11-05 23:39:03,082
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > Shutting down
> > 2014-11-05 23:39:03,082 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:03,089
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > Stopped
> > 2014-11-05 23:39:03,089
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > Shutdown completed
> > 2014-11-05 23:39:03,089
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ConsumerFetcherManager-68:
> > [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> > 2014-11-05 23:39:03,090
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ConsumerFetcherManager-68:
> > [ConsumerFetcherManager-1415230742842] All connections stopped
> > 2014-11-05 23:39:03,090
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all relevant
> > queues for this fetcher
> > 2014-11-05 23:39:03,090
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
> > this consumer iterator
> > 2014-11-05 23:39:03,091
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
> > chunks in all the consumer message iterators
> > 2014-11-05 23:39:03,091
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all
> offsets
> > after clearing the fetcher queues
> > 2014-11-05 23:39:03,091
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing partition
> > ownership
> > 2014-11-05 23:39:03,094 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:03,096
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the
> following
> > partitions: ArrayBuffer(0, 1) for topic Test with consumers:
> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> > 2014-11-05 23:39:03,098
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
> > partition 0
> > 2014-11-05 23:39:03,100
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 27,4 replyHeader:: 27,37,-101 request::
> > '/consumers/TestCheck/offsets/Test/0,F response::
> > 2014-11-05 23:39:03,116
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of Test:0:
> > fetched offset = -1: consumed offset = -1 is -1
> > 2014-11-05 23:39:03,117
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:0:
> > fetched offset = -1: consumed offset = -1 is -1
> > 2014-11-05 23:39:03,118
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.consumer.ZookeeperConsumerConnector-52:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:0: fetched
> offset
> > = -1: consumed offset = -1 selected new offset -1
> > 2014-11-05 23:39:03,169
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
> > partition 1
> > 2014-11-05 23:39:03,171
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 28,4 replyHeader:: 28,37,-101 request::
> > '/consumers/TestCheck/offsets/Test/1,F response::
> > 2014-11-05 23:39:03,172
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.consumer.PartitionTopicInfo-52: initial consumer offset of Test:1:
> > fetched offset = -1: consumed offset = -1 is -1
> > 2014-11-05 23:39:03,172
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:1:
> > fetched offset = -1: consumed offset = -1 is -1
> > 2014-11-05 23:39:03,172
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.consumer.ZookeeperConsumerConnector-52:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:1: fetched
> offset
> > = -1: consumed offset = -1 selected new offset -1
> > 2014-11-05 23:39:03,175 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:03,175
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 29,1 replyHeader:: 29,38,-101 request::
> >
> >
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> > response::
> > 2014-11-05 23:39:03,179
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 30,1 replyHeader:: 30,39,-101 request::
> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
> > response::
> > 2014-11-05 23:39:03,180 pool-13-thread-1 DEBUG
> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> > topic Test
> > 2014-11-05 23:39:03,181 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 1 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:03,182 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:03,182 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:03,186
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 31,1 replyHeader:: 31,40,0 request::
> > '/consumers/TestCheck/owners,,v{s{31,s{'world,'anyone}}},0 response::
> > '/consumers/TestCheck/owners
> > 2014-11-05 23:39:03,189 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:03,190
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 32,1 replyHeader:: 32,41,0 request::
> > '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
> response::
> > '/consumers/TestCheck/owners/Test
> > 2014-11-05 23:39:03,190 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:03,191 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:03,192 pool-13-thread-1 ERROR
> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
> by
> > topic, partition due to: Failed to fetch topic metadata for topic: Test
> > 2014-11-05 23:39:03,193
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 33,1 replyHeader:: 33,42,0 request::
> >
> >
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> > response:: '/consumers/TestCheck/owners/Test/1
> > 2014-11-05 23:39:03,193 pool-13-thread-1 INFO
> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> > retrying send. Remaining retries = 10
> > 2014-11-05 23:39:03,193
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
> > partition 1 for topic Test
> > 2014-11-05 23:39:03,196
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 34,1 replyHeader:: 34,43,0 request::
> >
> >
> '/consumers/TestCheck/owners/Test/0,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> > response:: '/consumers/TestCheck/owners/Test/0
> > 2014-11-05 23:39:03,196
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
> > partition 0 for topic Test
> > 2014-11-05 23:39:03,196
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
> > 2014-11-05 23:39:03,197
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.consumer.ZookeeperConsumerConnector-52:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per topic
> > cache Map(Test -> ArrayBuffer(0, 1))
> > 2014-11-05 23:39:03,197
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> > kafka.consumer.ZookeeperConsumerConnector-52:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per topic
> > cache Map(Test ->
> > List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> > 2014-11-05 23:39:03,197
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
> > Test:0: fetched offset = -1: consumed offset = -1,Test:1: fetched offset
> =
> > -1: consumed offset = -1
> > 2014-11-05 23:39:03,199
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> > Starting
> > 2014-11-05 23:39:03,199
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> > kafka.consumer.ZookeeperConsumerConnector-68:
> > [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
> > consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> > 2014-11-05 23:39:03,202
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 35,8 replyHeader:: 35,43,0 request::
> > '/brokers/ids,F response:: v{'0}
> > 2014-11-05 23:39:03,204
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> > sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> > finished:false header:: 36,4 replyHeader:: 36,43,0 request::
> > '/brokers/ids/0,F response::
> >
> >
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
> >
> > 2014-11-05 23:39:03,214
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > INFO kafka.utils.VerifiableProperties-68: Verifying properties
> > 2014-11-05 23:39:03,215
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > INFO kafka.utils.VerifiableProperties-68: Property client.id is
> overridden
> > to TestCheck
> > 2014-11-05 23:39:03,215
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > INFO kafka.utils.VerifiableProperties-68: Property metadata.broker.list
> is
> > overridden to x.x.x.x:9092
> > 2014-11-05 23:39:03,215
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > INFO kafka.utils.VerifiableProperties-68: Property request.timeout.ms is
> > overridden to 30000
> > 2014-11-05 23:39:03,217
> > TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> > INFO kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:x.x.x.x,port:9092 with correlation id 0 for 1 topic(s)
> Set(Test)
> > 2014-11-05 23:39:04,195 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 2 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:04,196 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:04,196 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:04,199 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:04,199 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:04,200 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:04,200 pool-13-thread-1 DEBUG
> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> > topic Test
> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 3 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:04,201 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:04,205 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:04,205 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:04,206 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:04,206 pool-13-thread-1 ERROR
> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
> by
> > topic, partition due to: Failed to fetch topic metadata for topic: Test
> > 2014-11-05 23:39:04,206 pool-13-thread-1 INFO
> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> > retrying send. Remaining retries = 9
> > 2014-11-05 23:39:04,698
> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> sessionid:
> > 0x1498251e8680002 after 0ms
> > 2014-11-05 23:39:05,205
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> sessionid:
> > 0x1498251e8680003 after 0ms
> > 2014-11-05 23:39:05,207 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 4 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:05,208 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:05,208 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:05,211 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:05,211 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:05,212 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:05,212 pool-13-thread-1 DEBUG
> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> > topic Test
> > 2014-11-05 23:39:05,212 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 5 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:05,213 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:05,213 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:05,217 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:05,217 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:05,217 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:05,217 pool-13-thread-1 ERROR
> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
> by
> > topic, partition due to: Failed to fetch topic metadata for topic: Test
> > 2014-11-05 23:39:05,218 pool-13-thread-1 INFO
> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> > retrying send. Remaining retries = 8
> > 2014-11-05 23:39:06,218 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 6 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:06,219 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:06,219 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:06,224 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:06,224 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:06,225 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:06,225 pool-13-thread-1 DEBUG
> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> > topic Test
> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 7 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:06,226 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:06,230 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:06,230 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:06,231 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:06,231 pool-13-thread-1 ERROR
> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
> by
> > topic, partition due to: Failed to fetch topic metadata for topic: Test
> > 2014-11-05 23:39:06,231 pool-13-thread-1 INFO
> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> > retrying send. Remaining retries = 7
> > 2014-11-05 23:39:06,699
> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> sessionid:
> > 0x1498251e8680002 after 0ms
> > 2014-11-05 23:39:07,208
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> sessionid:
> > 0x1498251e8680003 after 0ms
> > 2014-11-05 23:39:07,232 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 8 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:07,233 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:07,233 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:07,236 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:07,236 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:07,237 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:07,237 pool-13-thread-1 DEBUG
> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> > topic Test
> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 9 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:07,238 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:07,241 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:07,241 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:07,242 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:07,242 pool-13-thread-1 ERROR
> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
> by
> > topic, partition due to: Failed to fetch topic metadata for topic: Test
> > 2014-11-05 23:39:07,242 pool-13-thread-1 INFO
> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> > retrying send. Remaining retries = 6
> > 2014-11-05 23:39:08,243 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 10 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:08,243 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:08,245 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:08,249 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:08,250 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:08,250 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> > topic Test
> > 2014-11-05 23:39:08,251 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 11 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:08,252 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:08,255 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:08,256 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:08,256 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:08,257 pool-13-thread-1 ERROR
> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
> by
> > topic, partition due to: Failed to fetch topic metadata for topic: Test
> > 2014-11-05 23:39:08,257 pool-13-thread-1 INFO
> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> > retrying send. Remaining retries = 5
> > 2014-11-05 23:39:08,702
> > Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> sessionid:
> > 0x1498251e8680002 after 0ms
> > 2014-11-05 23:39:09,211
> >
> >
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> > DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for
> sessionid:
> > 0x1498251e8680003 after 0ms
> > 2014-11-05 23:39:09,257 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 12 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:09,258 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:09,258 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:09,261 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:09,262 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> > topic Test
> > 2014-11-05 23:39:09,263 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 13 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:09,264 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:09,264 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:09,268 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:09,268 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:09,269 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:09,269 pool-13-thread-1 ERROR
> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
> by
> > topic, partition due to: Failed to fetch topic metadata for topic: Test
> > 2014-11-05 23:39:09,269 pool-13-thread-1 INFO
> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> > retrying send. Remaining retries = 4
> > 2014-11-05 23:39:10,270 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 14 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:10,271 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:10,271 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:10,275 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:10,275 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:10,275 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:10,276 pool-13-thread-1 DEBUG
> > kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> > topic Test
> > 2014-11-05 23:39:10,276 pool-13-thread-1 INFO
> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > id:0,host:172.31.25.198,port:9092 with correlation id 15 for 1 topic(s)
> > Set(Test)
> > 2014-11-05 23:39:10,277 pool-13-thread-1 DEBUG
> > kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> > (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> > (requested 102400).
> > 2014-11-05 23:39:10,277 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> > producing
> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> > kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> > 2014-11-05 23:39:10,280 pool-13-thread-1 DEBUG
> > kafka.client.ClientUtils$-52: Successfully fetched metadata for 1
> topic(s)
> > Set(Test)
> > 2014-11-05 23:39:10,280 pool-13-thread-1 WARN
> > kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> > [{TopicMetadata for topic Test ->
> > No partition metadata for topic Test due to
> > kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> > kafka.common.LeaderNotAvailableException
> > 2014-11-05 23:39:10,280 pool-13-thread-1 ERROR
> > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages
> by
> > topic, partition due to: Failed to fetch topic metadata for topic: Test
> > 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> > retrying send. Remaining retries = 3
> >
> >
> > Brokers Logs -
> >
> > [2014-11-05 23:39:02,983] INFO Topic creation
> > {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
> > [2014-11-05 23:39:02,988] INFO [KafkaApi-0] Auto creation of topic Test
> > with 2 partitions and replication factor 1 is successful!
> > (kafka.server.KafkaApis)
> > [2014-11-05 23:39:03,092] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-05 23:39:03,191] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-05 23:39:04,200] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-05 23:39:04,205] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-05 23:39:05,211] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-05 23:39:05,218] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-05 23:39:06,224] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-05 23:39:06,230] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-05 23:39:07,236] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-05 23:39:07,241] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> >
> >
> >
> >
> > On Wed, Nov 5, 2014 at 3:05 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Sameer,
> > >
> > > Yes, this is the server log. But there seems no abnormal entries in it,
> > and
> > > it does not cover the same time range as the producer client throwing
> > > LeaderNotAvailableException
> > > (it was 10/24, 14:30).
> > >
> > > The reason that I want to check the server log at that same reason is
> > > that LeaderNotAvailableException
> > > is set by the brokers while handling requests, and its server log will
> > show
> > > what is the cause it set this error code.
> > >
> > > Guozhang
> > >
> > > On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net> wrote:
> > >
> > > > Hi Guozhang,
> > > >
> > > > This is the server.log -
> > > >
> > > > [2014-11-04 20:21:57,510] INFO Verifying properties
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,545] INFO Property advertised.host.name is
> > > overridden
> > > > to x.x.x.x (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,545] INFO Property broker.id is overridden to 0
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,545] INFO Property controlled.shutdown.enable is
> > > > overridden to true (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,545] INFO Property host.name is overridden to
> > > > 172.31.25.198 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is
> > overridden
> > > to
> > > > false (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden to
> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,546] INFO Property
> > log.retention.check.interval.ms
> > > is
> > > > overridden to 60000 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,546] INFO Property log.retention.hours is
> > overridden
> > > > to 168 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is
> overridden
> > > to
> > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,546] INFO Property num.io.threads is overridden
> > to 8
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,547] INFO Property num.network.threads is
> > overridden
> > > > to 2 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,547] INFO Property num.partitions is overridden
> > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,547] INFO Property port is overridden to 9092
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,547] INFO Property socket.receive.buffer.bytes
> is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,547] INFO Property socket.request.max.bytes is
> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,547] INFO Property socket.send.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is
> overridden
> > > to
> > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,548] INFO Property
> > zookeeper.connection.timeout.ms
> > > is
> > > > overridden to 1000000 (kafka.utils.VerifiableProperties)
> > > > [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
> > > > (kafka.server.KafkaServer)
> > > > [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to
> > zookeeper
> > > on
> > > > localhost:2181 (kafka.server.KafkaServer)
> > > > [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs' not
> > found,
> > > > creating it. (kafka.log.LogManager)
> > > > [2014-11-04 20:21:57,748] INFO Starting log cleanup with a period of
> > > 60000
> > > > ms. (kafka.log.LogManager)
> > > > [2014-11-04 20:21:57,752] INFO Starting log flusher with a default
> > period
> > > > of 9223372036854775807 ms. (kafka.log.LogManager)
> > > > [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
> > > > ip-172-31-25-198.us-west-1.compute.internal:9092.
> > > (kafka.network.Acceptor)
> > > > [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0], Started
> > > > (kafka.network.SocketServer)
> > > > [2014-11-04 20:21:57,852] INFO Will not load MX4J, mx4j-tools.jar is
> > not
> > > in
> > > > the classpath (kafka.utils.Mx4jLoader$)
> > > > [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
> > > > (kafka.server.ZookeeperLeaderElector)
> > > > [2014-11-04 20:21:57,967] INFO Registered broker 0 at path
> > /brokers/ids/0
> > > > with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
> > > > [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
> > > > (kafka.server.KafkaServer)
> > > > [2014-11-04 20:21:58,050] INFO New leader is 0
> > > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > > > [2014-11-04 20:22:01,809] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:22:11,812] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:29,084] INFO Topic creation
> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
> (kafka.admin.AdminUtils$)
> > > > [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of topic
> Test
> > > > with 2 partitions and replication factor 1 is successful!
> > > > (kafka.server.KafkaApis)
> > > > [2014-11-04 20:30:29,280] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:29,301] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:30,310] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:30,315] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:31,320] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:31,325] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:32,331] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:32,336] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:33,343] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:33,347] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:34,356] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:34,360] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:35,366] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:35,372] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:36,377] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:36,382] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:37,387] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:37,394] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:38,399] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:38,403] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:39,410] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:39,414] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:40,419] INFO Closing socket connection to /
> > > 172.31.25.198
> > > > .
> > > > (kafka.network.Processor)
> > > > [2014-11-04 20:30:41,602] INFO Topic creation
> > > > {"version":1,"partitions":{"1":[0],"0":[0]}}
> (kafka.admin.AdminUtils$)
> > > >
> > > >
> > > >
> > > > thanks
> > > >
> > > >
> > > > On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > > > The server.log should be the broker logs.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <sy...@ailive.net>
> > > wrote:
> > > > >
> > > > > > I do not see any other logs besides these.
> > > > > > I checked the log4j.properties and where the logs are getting
> > stored.
> > > > > > The only other logs (apart from the controller logs) in the
> > directory
> > > > are
> > > > > > zookeeper-gc.log,kafkaServer-gc.log, server.log and
> > state-change.log
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <
> wangguoz@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Sorry for getting late on this Sameer.
> > > > > > >
> > > > > > > The logs you showed above are from controller logs, do you see
> > > > brokers
> > > > > > logs
> > > > > > > besides that?
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <syami@ailive.net
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Guozhang,
> > > > > > > >
> > > > > > > > Is there any clue in the above data as to what is wrong?
> > > > > > > >
> > > > > > > > thanks
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <
> syami@ailive.net
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > This is the one from the file specified in
> log4j.properties -
> > > > > > > > >
> > > > > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on
> > > > Controller
> > > > > > 0]:
> > > > > > > > > Topic change listener fired for path /brokers/topics with
> > > > children
> > > > > > Test
> > > > > > > > >
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener on
> > > Controller
> > > > > 0]:
> > > > > > > New
> > > > > > > > > topics: [Set(Test)], deleted topics: [Set()], new partition
> > > > replica
> > > > > > > > > assignment [Map([Test,0] -> List(0), [Test,1] -> List(0))]
> > > > > > > > >
> (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New topic
> > > creation
> > > > > > > > callback
> > > > > > > > > for [Test,0],[Test,1] (kafka.controller.KafkaController)
> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New
> partition
> > > > > creation
> > > > > > > > > callback for [Test,0],[Test,1]
> > > (kafka.controller.KafkaController)
> > > > > > > > > [2014-10-28 22:38:36,095] INFO [Partition state machine on
> > > > > Controller
> > > > > > > 0]:
> > > > > > > > > Invoking state change to NewPartition for partitions
> > > > > > [Test,0],[Test,1]
> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine on
> > > > controller
> > > > > > 0]:
> > > > > > > > > Invoking state change to NewReplica for replicas
> > > > > > > > >
> > > > >
> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > > > > [2014-10-28 22:38:36,111] INFO [Partition state machine on
> > > > > Controller
> > > > > > > 0]:
> > > > > > > > > Invoking state change to OnlinePartition for partitions
> > > > > > > [Test,0],[Test,1]
> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state machine on
> > > > > > Controller
> > > > > > > > 0]:
> > > > > > > > > Live assigned replicas for partition [Test,0] are:
> [List(0)]
> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state machine on
> > > > > > Controller
> > > > > > > > 0]:
> > > > > > > > > Initializing leader and isr for partition [Test,0] to
> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state machine on
> > > > > > Controller
> > > > > > > > 0]:
> > > > > > > > > Live assigned replicas for partition [Test,1] are:
> [List(0)]
> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state machine on
> > > > > > Controller
> > > > > > > > 0]:
> > > > > > > > > Initializing leader and isr for partition [Test,1] to
> > > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine on
> > > > controller
> > > > > > 0]:
> > > > > > > > > Invoking state change to OnlineReplica for replicas
> > > > > > > > >
> > > > >
> [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > > > > [2014-10-28 22:39:28,160] ERROR
> > > > > > [Controller-0-to-broker-0-send-thread],
> > > > > > > > > Controller 0's connection to broker
> > id:0,host:x.x.x.x,port:9092
> > > > was
> > > > > > > > > unsuccessful (kafka.controller.RequestSendThread)
> > > > > > > > > java.net.ConnectException: Connection timed out
> > > > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:449)
> > > > > > > > > at
> > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> > > > > > > > > at
> > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > > > > at
> > > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > > > > [2014-10-28 22:39:28,460] ERROR
> > > > > > [Controller-0-to-broker-0-send-thread],
> > > > > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr request
> with
> > > > > > > correlation
> > > > > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092. Reconnecting to
> > > > broker.
> > > > > > > > > (kafka.controller.RequestSendThread)
> > > > > > > > > java.nio.channels.ClosedChannelException
> > > > > > > > > at
> > > > kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > > > > at
> > > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > > > >
> > > > > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
> > > > wangguoz@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Sameer, the one you sent me is from the producer logs, not
> > the
> > > > > > broker
> > > > > > > > >> logs.
> > > > > > > > >> And /tmp/kafka-logs actually store the message log (i.e.
> the
> > > > > "data")
> > > > > > > > >> instead of the server logs. Their locations should be
> > > specified
> > > > in
> > > > > > > your
> > > > > > > > >> log4j.properties file.
> > > > > > > > >>
> > > > > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <
> > > syami@ailive.net>
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> > Is this what you are looking for -
> > > > > > > > >> >
> > > > > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > > > 172.31.16.31:9092
> > > > > > for
> > > > > > > > >> > producing
> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > > > 172.31.16.31:9092
> > > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > metadata
> > > > for
> > > > > 1
> > > > > > > > >> topic(s)
> > > > > > > > >> > Set(Test)
> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > fetching
> > > > > > metadata
> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > > > >> > No partition metadata for topic Test due to
> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> > [Test]:
> > > > > class
> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> > > > partition
> > > > > > info
> > > > > > > > for
> > > > > > > > >> > topic Test
> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
> > broker
> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 9
> > for 1
> > > > > > > topic(s)
> > > > > > > > >> > Set(Test)
> > > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > > > > SO_TIMEOUT =
> > > > > > > > 10000
> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> > > > SO_SNDBUF
> > > > > =
> > > > > > > > 102400
> > > > > > > > >> > (requested 102400).
> > > > > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > > > 172.31.16.31:9092
> > > > > > for
> > > > > > > > >> > producing
> > > > > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > > > 172.31.16.31:9092
> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > metadata
> > > > for
> > > > > 1
> > > > > > > > >> topic(s)
> > > > > > > > >> > Set(Test)
> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > fetching
> > > > > > metadata
> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > > > >> > No partition metadata for topic Test due to
> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> > [Test]:
> > > > > class
> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
> > > collate
> > > > > > > > messages
> > > > > > > > >> by
> > > > > > > > >> > topic, partition due to: Failed to fetch topic metadata
> > for
> > > > > topic:
> > > > > > > > Test
> > > > > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off
> for
> > > 1000
> > > > > ms
> > > > > > > > before
> > > > > > > > >> > retrying send. Remaining retries = 6
> > > > > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
> > broker
> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 10
> > for
> > > 1
> > > > > > > topic(s)
> > > > > > > > >> > Set(Test)
> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > > > > SO_TIMEOUT =
> > > > > > > > 10000
> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> > > > SO_SNDBUF
> > > > > =
> > > > > > > > 102400
> > > > > > > > >> > (requested 102400).
> > > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > > > 172.31.16.31:9092
> > > > > > for
> > > > > > > > >> > producing
> > > > > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > > > 172.31.16.31:9092
> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > metadata
> > > > for
> > > > > 1
> > > > > > > > >> topic(s)
> > > > > > > > >> > Set(Test)
> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > fetching
> > > > > > metadata
> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > > > >> > No partition metadata for topic Test due to
> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> > [Test]:
> > > > > class
> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> > > > partition
> > > > > > info
> > > > > > > > for
> > > > > > > > >> > topic Test
> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
> > broker
> > > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 11
> > for
> > > 1
> > > > > > > topic(s)
> > > > > > > > >> > Set(Test)
> > > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
> > > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > > > > SO_TIMEOUT =
> > > > > > > > 10000
> > > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> > > > SO_SNDBUF
> > > > > =
> > > > > > > > 102400
> > > > > > > > >> > (requested 102400).
> > > > > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > > > 172.31.16.31:9092
> > > > > > for
> > > > > > > > >> > producing
> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > > > 172.31.16.31:9092
> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
> > > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> > metadata
> > > > for
> > > > > 1
> > > > > > > > >> topic(s)
> > > > > > > > >> > Set(Test)
> > > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
> > > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> > fetching
> > > > > > metadata
> > > > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > > > >> > No partition metadata for topic Test due to
> > > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> > [Test]:
> > > > > class
> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
> > > collate
> > > > > > > > messages
> > > > > > > > >> by
> > > > > > > > >> > topic, partition due to: Failed to fetch topic metadata
> > for
> > > > > topic:
> > > > > > > > Test
> > > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off
> for
> > > 1000
> > > > > ms
> > > > > > > > before
> > > > > > > > >> > retrying send. Remaining retries = 5
> > > > > > > > >> >
> > > > > > > > >> > As for /tmp/kafka-logs, I can see only these files -
> > > > > > > > >> > recovery-point-offset-checkpoint
> > > > replication-offset-checkpoint
> > > > > > > > >> >
> > > > > > > > >> > I do not see the usual Kafka log files here.
> > > > > > > > >> >
> > > > > > > > >> > thanks
> > > > > > > > >> > Sameer
> > > > > > > > >> >
> > > > > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
> > > > > > wangguoz@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Sameer, could you check broker logs and see if it has
> > > > similar
> > > > > > > > >> > > LeaderNotAvailable exceptions?
> > > > > > > > >> > >
> > > > > > > > >> > > Guozhang
> > > > > > > > >> > >
> > > > > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
> > > > > syami@ailive.net
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Yes. I can telnet from outside using the public ip
> of
> > > the
> > > > > > > instance
> > > > > > > > >> and
> > > > > > > > >> > > port
> > > > > > > > >> > > > of 9092
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <
> > > > > > > > >> roger.hoover@gmail.com
> > > > > > > > >> > >
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Just a guess but could it be a firewall issue?
> Did
> > > you
> > > > > > enable
> > > > > > > > >> > > > connections
> > > > > > > > >> > > > > to port 9092 from outside EC2 in a security group?
> > > Can
> > > > > you
> > > > > > > > >> telnet to
> > > > > > > > >> > > > each
> > > > > > > > >> > > > > broker IP and port?
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami <
> > > > > > > syami@ailive.net
> > > > > > > > >
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > There was a typo earlier.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > This is the output -
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Topic:Test PartitionCount:2
> ReplicationFactor:1
> > > > > > > Configs:
> > > > > > > > >> > > > > > Topic: Test Partition: 0 Leader: 0
> > > > > > > Replicas:
> > > > > > > > 0
> > > > > > > > >> > > > Isr: 0
> > > > > > > > >> > > > > > Topic: Test Partition: 1 Leader: 0
> > > > > > > Replicas:
> > > > > > > > 0
> > > > > > > > >> > > > Isr: 0
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > thanks
> > > > > > > > >> > > > > > Sameer
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang Wang <
> > > > > > > > >> wangguoz@gmail.com
> > > > > > > > >> > >
> > > > > > > > >> > > > > wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Sameer,
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > The topic name is not consistent from your
> > > describe
> > > > > > > outputs.
> > > > > > > > >> > > Normally
> > > > > > > > >> > > > > > they
> > > > > > > > >> > > > > > > should be either
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Topic:*Test* PartitionCount:2
> > > > > ReplicationFactor:1
> > > > > > > > >> > > Configs:
> > > > > > > > >> > > > > > > Topic: *Test* Partition: 0
> > Leader: 0
> > > > > > > > >> Replicas: 0
> > > > > > > > >> > > > > > Isr: 0
> > > > > > > > >> > > > > > > Topic: *Test* Partition: 1
> > Leader: 0
> > > > > > > > >> Replicas: 0
> > > > > > > > >> > > > > > Isr: 0
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Or
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
> > > > > > > > ReplicationFactor:1
> > > > > > > > >> > > > > Configs:
> > > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 0
> > > > Leader:
> > > > > 0
> > > > > > > > >> > > Replicas:
> > > > > > > > >> > > > 0
> > > > > > > > >> > > > > > > Isr: 0
> > > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 1
> > > > Leader:
> > > > > 0
> > > > > > > > >> > > Replicas:
> > > > > > > > >> > > > 0
> > > > > > > > >> > > > > > > Isr: 0
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Did you manually modify the topic name in ZK?
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Guozhang
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer Yami <
> > > > > > > > >> syami@ailive.net>
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > Hi Guozhang,
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > I was wondering if there is anything wrong
> in
> > > the
> > > > > > above
> > > > > > > > >> > > --describe
> > > > > > > > >> > > > > > data.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > thanks
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer
> Yami <
> > > > > > > > >> syami@ailive.net
> > > > > > > > >> > >
> > > > > > > > >> > > > > wrote:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > This is what I get -
> > > > > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper
> > > localhost:2181
> > > > > > > > --describe
> > > > > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
> > > > > > > > >> ReplicationFactor:1
> > > > > > > > >> > > > > > Configs:
> > > > > > > > >> > > > > > > > > Topic: Test Partition: 0
> > > Leader: 0
> > > > > > > > >> > Replicas: 0
> > > > > > > > >> > > > > > > Isr: 0
> > > > > > > > >> > > > > > > > > Topic: Test Partition: 1
> > > Leader: 0
> > > > > > > > >> > Replicas: 0
> > > > > > > > >> > > > > > > Isr: 0
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > The topics are being created at run-time.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > thanks
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM, Guozhang
> > > Wang <
> > > > > > > > >> > > > wangguoz@gmail.com
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > > wrote:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >> Could you use the topic tools
> "--describe"
> > to
> > > > > check
> > > > > > > if
> > > > > > > > >> the
> > > > > > > > >> > > topic
> > > > > > > > >> > > > > > > Test's
> > > > > > > > >> > > > > > > > >> partitions leaders are available?
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > >
> > > > > http://kafka.apache.org/documentation.html#basic_ops_add_topic
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > > > > > > > >> Guozhang
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM, Sameer
> > Yami
> > > <
> > > > > > > > >> > > syami@ailive.net>
> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > > > > > > > >> > That is not the case.
> > > > > > > > >> > > > > > > > >> > This detailed error is below. Due to
> this
> > > > > error ,
> > > > > > > the
> > > > > > > > >> > > producer
> > > > > > > > >> > > > > is
> > > > > > > > >> > > > > > > not
> > > > > > > > >> > > > > > > > >> able
> > > > > > > > >> > > > > > > > >> > to send messages
> > > > > > > > >> > > > > > > > >> >
> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG
> > > > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-52:
> > > > > > > > >> > > > > > > > >> > Getting broker partition info for topic
> > > Test
> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main INFO
> > > > > > > > >> > > > > kafka.client.ClientUtils$-68:
> > > > > > > > >> > > > > > > > >> Fetching
> > > > > > > > >> > > > > > > > >> > metadata from broker
> > > > > > > > id:0,host:54.215.200.61,port:9092
> > > > > > > > >> > with
> > > > > > > > >> > > > > > > > correlation
> > > > > > > > >> > > > > > > > >> id
> > > > > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG
> > > > > > > > >> > > > > > > kafka.network.BlockingChannel-52:
> > > > > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT = 10000
> > > > > (requested
> > > > > > > > >> 10000),
> > > > > > > > >> > > > > > SO_RCVBUF
> > > > > > > > >> > > > > > > =
> > > > > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF =
> 102400
> > > > > > > (requested
> > > > > > > > >> > > 102400).
> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main INFO
> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092 for
> > > > producing
> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main INFO
> > > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > > > > >> > > > > > > > >> > Disconnecting from 54.215.200.61:9092
> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG
> > > > > > > > >> > > > > kafka.client.ClientUtils$-52:
> > > > > > > > >> > > > > > > > >> > Successfully fetched metadata for 1
> > > topic(s)
> > > > > > > > Set(Test)
> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main WARN
> > > > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-83:
> > > > > > > > >> > > > > > > > >> > Error while fetching metadata
> > > [{TopicMetadata
> > > > > for
> > > > > > > > topic
> > > > > > > > >> > Test
> > > > > > > > >> > > > ->
> > > > > > > > >> > > > > > > > >> > No partition metadata for topic Test
> due
> > to
> > > > > > > > >> > > > > > > > >> >
> > kafka.common.LeaderNotAvailableException}]
> > > > for
> > > > > > > topic
> > > > > > > > >> > [Test]:
> > > > > > > > >> > > > > class
> > > > > > > > >> > > > > > > > >> >
> kafka.common.LeaderNotAvailableException
> > > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR
> > > > > > > > >> > > > > > > > >> >
> > > kafka.producer.async.DefaultEventHandler-97:
> > > > > > Failed
> > > > > > > > to
> > > > > > > > >> > > collate
> > > > > > > > >> > > > > > > > messages
> > > > > > > > >> > > > > > > > >> by
> > > > > > > > >> > > > > > > > >> > topic, partition due to: Failed to
> fetch
> > > > topic
> > > > > > > > metadata
> > > > > > > > >> > for
> > > > > > > > >> > > > > topic:
> > > > > > > > >> > > > > > > > Test
> > > > > > > > >> > > > > > > > >> >
> > > > > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM,
> Guozhang
> > > > Wang
> > > > > <
> > > > > > > > >> > > > > > wangguoz@gmail.com>
> > > > > > > > >> > > > > > > > >> wrote:
> > > > > > > > >> > > > > > > > >> >
> > > > > > > > >> > > > > > > > >> > > Could you see if this is the case:
> > > > > > > > >> > > > > > > > >> > >
> > > > > > > > >> > > > > > > > >> > >
> > > > > > > > >> > > > > > > > >> > >
> > > > > > > > >> > > > > > > > >> >
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> > > > > > > > >> > > > > > > > >> > > ?
> > > > > > > > >> > > > > > > > >> > >
> > > > > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM,
> > Sameer
> > > > > Yami <
> > > > > > > > >> > > > > syami@ailive.net
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > >> wrote:
> > > > > > > > >> > > > > > > > >> > >
> > > > > > > > >> > > > > > > > >> > > > Hi,
> > > > > > > > >> > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper test
> > setup
> > > > on a
> > > > > > > > single
> > > > > > > > >> EC2
> > > > > > > > >> > > > > > machine.
> > > > > > > > >> > > > > > > > The
> > > > > > > > >> > > > > > > > >> > > > host.name is the private ip
> address
> > > and
> > > > > the
> > > > > > > > >> > > > > > > advertised.host.name
> > > > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > > > >> > the
> > > > > > > > >> > > > > > > > >> > > > public address in
> server.properties.
> > > > > > > > >> > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > >> > > > We are trying to publish from
> outside
> > > EC2
> > > > > > using
> > > > > > > > the
> > > > > > > > >> > > public
> > > > > > > > >> > > > > ip
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > >> > EC2
> > > > > > > > >> > > > > > > > >> > > > machine but get this error on the
> > > client
> > > > > > side -
> > > > > > > > >> > > > > > > > >> > > > No partition metadata for topic
> test
> > > due
> > > > to
> > > > > > > > >> > > > > > > > >> > > >
> > > > kafka.common.LeaderNotAvailableException}]
> > > > > > for
> > > > > > > > >> topic
> > > > > > > > >> > > > [test]:
> > > > > > > > >> > > > > > > class
> > > > > > > > >> > > > > > > > >> > > >
> > > kafka.common.LeaderNotAvailableException
> > > > > > > > >> > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > >> > > > The zookeeper has the public ip
> > stored
> > > in
> > > > > > > > >> > > /brokers/ids/0.
> > > > > > > > >> > > > > > > > >> > > > We have tried multiple combinations
> > but
> > > > it
> > > > > > > works
> > > > > > > > >> ONLY
> > > > > > > > >> > > when
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > > >> > advertised
> > > > > > > > >> > > > > > > > >> > > > host name is not used and the
> client
> > is
> > > > > > > internal
> > > > > > > > to
> > > > > > > > >> > EC2.
> > > > > > > > >> > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > >> > > > Please let me know what the problem
> > > could
> > > > > be.
> > > > > > > > >> > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > >> > > > thanks
> > > > > > > > >> > > > > > > > >> > > > Sameer
> > > > > > > > >> > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > >> > >
> > > > > > > > >> > > > > > > > >> > >
> > > > > > > > >> > > > > > > > >> > >
> > > > > > > > >> > > > > > > > >> > > --
> > > > > > > > >> > > > > > > > >> > > -- Guozhang
> > > > > > > > >> > > > > > > > >> > >
> > > > > > > > >> > > > > > > > >> >
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > > > > > > > >> --
> > > > > > > > >> > > > > > > > >> -- Guozhang
> > > > > > > > >> > > > > > > > >>
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > --
> > > > > > > > >> > > > > > > -- Guozhang
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > --
> > > > > > > > >> > > -- Guozhang
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> -- Guozhang
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Cannot connect to Kafka from outside of EC2
Posted by Guozhang Wang <wa...@gmail.com>.
Sameer,
The server logs do not contain any non-INFO logs, which is a bit wired. Did
you deploy the current trunk of Kafka? Also could you enable DEBUG level
logging on Kafka brokers?
Guozhang
On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami <sy...@ailive.net> wrote:
> The server.log was taken separately.
> We ran the test again and the server and producer logs are below (to get
> same timings).
>
>
> Thanks!
>
>
> ------------------------------------------------------------------------------------
>
>
>
> Producer Logs -
>
>
> 2014-11-05 23:38:58,693
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
> 0x1498251e8680002 after 0ms
> 2014-11-05 23:39:00,695
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
> 0x1498251e8680002 after 0ms
> 2014-11-05 23:39:02,696
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
> 0x1498251e8680002 after 0ms
> 2014-11-05 23:39:02,828 pool-13-thread-2 INFO
> kafka.utils.VerifiableProperties-68: Verifying properties
> 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> kafka.utils.VerifiableProperties-68: Property auto.commit.interval.ms is
> overridden to 1000
> 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> kafka.utils.VerifiableProperties-68: Property auto.offset.reset is
> overridden to smallest
> 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> kafka.utils.VerifiableProperties-68: Property consumer.timeout.ms is
> overridden to 10
> 2014-11-05 23:39:02,829 pool-13-thread-2 INFO
> kafka.utils.VerifiableProperties-68: Property group.id is overridden to
> TestCheck
> 2014-11-05 23:39:02,830 pool-13-thread-2 WARN
> kafka.utils.VerifiableProperties-83: Property serializer.class is not valid
> 2014-11-05 23:39:02,830 pool-13-thread-2 INFO
> kafka.utils.VerifiableProperties-68: Property zookeeper.connect is
> overridden to 172.31.25.198:2181
> 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Connecting to
> zookeeper instance at 172.31.25.198:2181
> 2014-11-05 23:39:02,831 pool-13-thread-2 DEBUG
> org.I0Itec.zkclient.ZkConnection-63: Creating new ZookKeeper instance to
> connect to 172.31.25.198:2181.
> 2014-11-05 23:39:02,831 pool-13-thread-2 INFO
> org.apache.zookeeper.ZooKeeper-379: Initiating client connection,
> connectString=172.31.25.198:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@3903b165
> 2014-11-05 23:39:02,831 ZkClient-EventThread-29-172.31.25.198:2181 INFO
> org.I0Itec.zkclient.ZkEventThread-64: Starting ZkClient event thread.
> 2014-11-05 23:39:02,831 pool-13-thread-1 INFO
> kafka.utils.VerifiableProperties-68: Verifying properties
> 2014-11-05 23:39:02,836 pool-13-thread-2-SendThread() INFO
> org.apache.zookeeper.ClientCnxn-1061: Opening socket connection to server /
> 172.31.25.198:2181
> 2014-11-05 23:39:02,836 pool-13-thread-1 WARN
> kafka.utils.VerifiableProperties-83: Property batch.size is not valid
> 2014-11-05 23:39:02,832 pool-13-thread-2 DEBUG
> org.I0Itec.zkclient.ZkClient-878: Awaiting connection to Zookeeper server
> 2014-11-05 23:39:02,836 pool-13-thread-1 INFO
> kafka.utils.VerifiableProperties-68: Property message.send.max.retries is
> overridden to 10
> 2014-11-05 23:39:02,836 pool-13-thread-2 DEBUG
> org.I0Itec.zkclient.ZkClient-628: Waiting for keeper state SyncConnected
> 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> kafka.utils.VerifiableProperties-68: Property metadata.broker.list is
> overridden to 172.31.25.198:9092
> 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> kafka.utils.VerifiableProperties-68: Property retry.backoff.ms is
> overridden to 1000
> 2014-11-05 23:39:02,837 pool-13-thread-1 INFO
> kafka.utils.VerifiableProperties-68: Property serializer.class is
> overridden to kafka.serializer.StringEncoder
> 2014-11-05 23:39:02,837
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> INFO org.apache.zookeeper.ClientCnxn-950: Socket connection established to
> ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181, initiating
> session
> 2014-11-05 23:39:02,838
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-999: Session establishment request
> sent on ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181
> 2014-11-05 23:39:02,837 pool-13-thread-1 WARN
> kafka.utils.VerifiableProperties-83: Property zk.connectiontimeout.ms is
> not valid
> 2014-11-05 23:39:02,841
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> INFO org.apache.zookeeper.ClientCnxn-739: Session establishment complete on
> server ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
> sessionid = 0x1498251e8680003, negotiated timeout = 6000
> 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> state:SyncConnected type:None path:null
> 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread INFO
> org.I0Itec.zkclient.ZkClient-449: zookeeper state changed (SyncConnected)
> 2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
> org.I0Itec.zkclient.ZkClient-395: Leaving process event
> 2014-11-05 23:39:02,841 pool-13-thread-2 DEBUG
> org.I0Itec.zkclient.ZkClient-638: State is SyncConnected
> 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> kafka.utils.KafkaScheduler-52: Initializing task scheduler.
> 2014-11-05 23:39:02,842 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting auto
> committer every 1000 ms
> 2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
> kafka.utils.KafkaScheduler-52: Scheduling task kafka-consumer-autocommit
> with initial delay 1000 ms and period 1000 ms.
> 2014-11-05 23:39:02,843 pool-13-thread-2 DEBUG
> kafka.consumer.ZookeeperConsumerConnector-52:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], entering consume
> 2014-11-05 23:39:02,843 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin registering
> consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> 2014-11-05 23:39:02,846
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 1,1 replyHeader:: 1,29,-101 request::
>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> response::
> 2014-11-05 23:39:02,854
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 2,1 replyHeader:: 2,30,-101 request::
> '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> 2014-11-05 23:39:02,858
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 3,1 replyHeader:: 3,31,0 request::
> '/consumers/TestCheck,,v{s{31,s{'world,'anyone}}},0 response::
> '/consumers/TestCheck
> 2014-11-05 23:39:02,861
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 4,1 replyHeader:: 4,32,0 request::
> '/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
> '/consumers/TestCheck/ids
> 2014-11-05 23:39:02,866
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 5,1 replyHeader:: 5,33,0 request::
>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
> response::
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> 2014-11-05 23:39:02,867 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end registering
> consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
> 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> kafka.consumer.ZookeeperConsumerConnector-52:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Adding topicThreadId
> (Test,TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0) and queue [] to
> topicThreadIdAndQueues data structure
> 2014-11-05 23:39:02,868
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting watcher
> executor thread for consumer
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
> 2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
> kafka.consumer.ZookeeperConsumerConnector-52:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], adding topic Test and
> 1 streams to map.
> 2014-11-05 23:39:02,870
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 6,3 replyHeader:: 6,33,0 request::
> '/consumers/TestCheck/ids,T response::
> s{32,32,1415230742859,1415230742859,0,1,0,0,0,1,33}
> 2014-11-05 23:39:02,872
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 7,8 replyHeader:: 7,33,0 request::
> '/consumers/TestCheck/ids,T response::
> v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> 2014-11-05 23:39:02,873
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 8,3 replyHeader:: 8,33,-101 request::
> '/brokers/topics/Test,T response::
> 2014-11-05 23:39:02,873 pool-13-thread-2 DEBUG
> org.I0Itec.zkclient.ZkClient-137: Subscribed data changes for
> /brokers/topics/Test
> 2014-11-05 23:39:02,873 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
> consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> 2014-11-05 23:39:02,875 pool-13-thread-1 DEBUG
> kafka.producer.async.DefaultEventHandler-52: Handling 1 events
> 2014-11-05 23:39:02,877
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 9,8 replyHeader:: 9,33,0 request::
> '/brokers/ids,F response:: v{'0}
> 2014-11-05 23:39:02,879
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 10,4 replyHeader:: 10,33,0 request::
> '/brokers/ids/0,F response::
>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>
> 2014-11-05 23:39:02,887
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 11,4 replyHeader:: 11,33,0 request::
>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> response::
>
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>
> 2014-11-05 23:39:02,896
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 12,8 replyHeader:: 12,33,0 request::
> '/consumers/TestCheck/ids,T response::
> v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> 2014-11-05 23:39:02,898
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 13,4 replyHeader:: 13,33,0 request::
>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> response::
>
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>
> 2014-11-05 23:39:02,908
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 14,8 replyHeader:: 14,33,0 request::
> '/brokers/ids,F response:: v{'0}
> 2014-11-05 23:39:02,910
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 15,4 replyHeader:: 15,33,0 request::
> '/brokers/ids/0,F response::
>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>
> 2014-11-05 23:39:02,917 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 0 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:02,917
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 16,4 replyHeader:: 16,33,-101 request::
> '/brokers/topics/Test,T response::
> 2014-11-05 23:39:02,920 pool-13-thread-2 DEBUG kafka.utils.ZkUtils$-52:
> Partition map for /brokers/topics/Test is Map()
> 2014-11-05 23:39:02,920 pool-13-thread-2 INFO
> kafka.consumer.ConsumerFetcherManager-68:
> [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
> 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> kafka.consumer.ConsumerFetcherManager-68:
> [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> kafka.consumer.ConsumerFetcherManager-68:
> [ConsumerFetcherManager-1415230742842] All connections stopped
> 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all relevant
> queues for this fetcher
> 2014-11-05 23:39:02,921 pool-13-thread-2 DEBUG
> kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
> this consumer iterator
> 2014-11-05 23:39:02,921 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
> chunks in all the consumer message iterators
> 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all offsets
> after clearing the fetcher queues
> 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing partition
> ownership
> 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the following
> partitions: ArrayBuffer() for topic Test with consumers:
> List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> 2014-11-05 23:39:02,922 pool-13-thread-2 WARN
> kafka.consumer.ZookeeperConsumerConnector-83:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], No broker partitions
> consumed by consumer thread
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 for topic Test
> 2014-11-05 23:39:02,922 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
> 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> kafka.consumer.ZookeeperConsumerConnector-52:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per topic
> cache Map(Test -> ArrayBuffer())
> 2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
> kafka.consumer.ZookeeperConsumerConnector-52:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per topic
> cache Map(Test ->
> List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> 2014-11-05 23:39:02,923 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
> 2014-11-05 23:39:02,927 pool-13-thread-2 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
> consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> 2014-11-05 23:39:02,928
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> Starting
> 2014-11-05 23:39:02,930 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:02,930 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:02,985
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-783: Got notification
> sessionid:0x1498251e8680003
> 2014-11-05 23:39:02,986
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-800: Got WatchedEvent
> state:SyncConnected type:NodeCreated path:/brokers/topics/Test for
> sessionid 0x1498251e8680003
> 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
> state:SyncConnected type:NodeCreated path:/brokers/topics/Test
> 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> org.I0Itec.zkclient.ZkEventThread-88: New event: ZkEvent[Data of
> /brokers/topics/Test changed sent to
>
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> ]
> 2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
> org.I0Itec.zkclient.ZkClient-395: Leaving process event
> 2014-11-05 23:39:02,988 ZkClient-EventThread-29-172.31.25.198:2181 DEBUG
> org.I0Itec.zkclient.ZkEventThread-69: Delivering event #1 ZkEvent[Data of
> /brokers/topics/Test changed sent to
>
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
> ]
> 2014-11-05 23:39:02,991
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 17,3 replyHeader:: 17,37,0 request::
> '/brokers/topics/Test,T response::
> s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
> 2014-11-05 23:39:02,993
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 18,4 replyHeader:: 18,37,0 request::
> '/brokers/topics/Test,T response::
>
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
>
> 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181 INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Topic info for path
> /brokers/topics/Test changed to
> {"version":1,"partitions":{"1":[0],"0":[0]}}, triggering rebalance
> 2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181 DEBUG
> org.I0Itec.zkclient.ZkEventThread-79: Delivering event #1 done
> 2014-11-05 23:39:02,995
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
> consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> 2014-11-05 23:39:02,996
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 19,8 replyHeader:: 19,37,0 request::
> '/brokers/ids,F response:: v{'0}
> 2014-11-05 23:39:02,998
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 20,4 replyHeader:: 20,37,0 request::
> '/brokers/ids/0,F response::
>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>
> 2014-11-05 23:39:03,008
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 21,4 replyHeader:: 21,37,0 request::
>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> response::
>
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>
> 2014-11-05 23:39:03,014
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 22,8 replyHeader:: 22,37,0 request::
> '/consumers/TestCheck/ids,T response::
> v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
> 2014-11-05 23:39:03,018
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 23,4 replyHeader:: 23,37,0 request::
>
> '/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
> response::
>
> #7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
>
> 2014-11-05 23:39:03,041
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 24,8 replyHeader:: 24,37,0 request::
> '/brokers/ids,F response:: v{'0}
> 2014-11-05 23:39:03,043
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 25,4 replyHeader:: 25,37,0 request::
> '/brokers/ids/0,F response::
>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>
> 2014-11-05 23:39:03,058
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 26,4 replyHeader:: 26,37,0 request::
> '/brokers/topics/Test,T response::
>
> #7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
>
> 2014-11-05 23:39:03,074
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.utils.ZkUtils$-52: Partition map for /brokers/topics/Test is Map(1 ->
> List(0), 0 -> List(0))
> 2014-11-05 23:39:03,075
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ConsumerFetcherManager-68:
> [ConsumerFetcherManager-1415230742842] Stopping leader finder thread
> 2014-11-05 23:39:03,082
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> Shutting down
> 2014-11-05 23:39:03,082 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:03,089
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> Stopped
> 2014-11-05 23:39:03,089
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> Shutdown completed
> 2014-11-05 23:39:03,089
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ConsumerFetcherManager-68:
> [ConsumerFetcherManager-1415230742842] Stopping all fetchers
> 2014-11-05 23:39:03,090
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ConsumerFetcherManager-68:
> [ConsumerFetcherManager-1415230742842] All connections stopped
> 2014-11-05 23:39:03,090
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all relevant
> queues for this fetcher
> 2014-11-05 23:39:03,090
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
> this consumer iterator
> 2014-11-05 23:39:03,091
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
> chunks in all the consumer message iterators
> 2014-11-05 23:39:03,091
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all offsets
> after clearing the fetcher queues
> 2014-11-05 23:39:03,091
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing partition
> ownership
> 2014-11-05 23:39:03,094 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:03,096
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the following
> partitions: ArrayBuffer(0, 1) for topic Test with consumers:
> List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
> 2014-11-05 23:39:03,098
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
> partition 0
> 2014-11-05 23:39:03,100
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 27,4 replyHeader:: 27,37,-101 request::
> '/consumers/TestCheck/offsets/Test/0,F response::
> 2014-11-05 23:39:03,116
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.consumer.PartitionTopicInfo-52: initial consumer offset of Test:0:
> fetched offset = -1: consumed offset = -1 is -1
> 2014-11-05 23:39:03,117
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:0:
> fetched offset = -1: consumed offset = -1 is -1
> 2014-11-05 23:39:03,118
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.consumer.ZookeeperConsumerConnector-52:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:0: fetched offset
> = -1: consumed offset = -1 selected new offset -1
> 2014-11-05 23:39:03,169
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
> partition 1
> 2014-11-05 23:39:03,171
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 28,4 replyHeader:: 28,37,-101 request::
> '/consumers/TestCheck/offsets/Test/1,F response::
> 2014-11-05 23:39:03,172
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.consumer.PartitionTopicInfo-52: initial consumer offset of Test:1:
> fetched offset = -1: consumed offset = -1 is -1
> 2014-11-05 23:39:03,172
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:1:
> fetched offset = -1: consumed offset = -1 is -1
> 2014-11-05 23:39:03,172
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.consumer.ZookeeperConsumerConnector-52:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:1: fetched offset
> = -1: consumed offset = -1 selected new offset -1
> 2014-11-05 23:39:03,175 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:03,175
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 29,1 replyHeader:: 29,38,-101 request::
>
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> response::
> 2014-11-05 23:39:03,179
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 30,1 replyHeader:: 30,39,-101 request::
> '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
> response::
> 2014-11-05 23:39:03,180 pool-13-thread-1 DEBUG
> kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> topic Test
> 2014-11-05 23:39:03,181 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 1 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:03,182 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:03,182 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:03,186
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 31,1 replyHeader:: 31,40,0 request::
> '/consumers/TestCheck/owners,,v{s{31,s{'world,'anyone}}},0 response::
> '/consumers/TestCheck/owners
> 2014-11-05 23:39:03,189 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:03,190
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 32,1 replyHeader:: 32,41,0 request::
> '/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0 response::
> '/consumers/TestCheck/owners/Test
> 2014-11-05 23:39:03,190 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:03,191 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:03,192 pool-13-thread-1 ERROR
> kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: Test
> 2014-11-05 23:39:03,193
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 33,1 replyHeader:: 33,42,0 request::
>
> '/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> response:: '/consumers/TestCheck/owners/Test/1
> 2014-11-05 23:39:03,193 pool-13-thread-1 INFO
> kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> retrying send. Remaining retries = 10
> 2014-11-05 23:39:03,193
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
> partition 1 for topic Test
> 2014-11-05 23:39:03,196
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 34,1 replyHeader:: 34,43,0 request::
>
> '/consumers/TestCheck/owners/Test/0,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
> response:: '/consumers/TestCheck/owners/Test/0
> 2014-11-05 23:39:03,196
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
> partition 0 for topic Test
> 2014-11-05 23:39:03,196
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
> 2014-11-05 23:39:03,197
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.consumer.ZookeeperConsumerConnector-52:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per topic
> cache Map(Test -> ArrayBuffer(0, 1))
> 2014-11-05 23:39:03,197
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
> kafka.consumer.ZookeeperConsumerConnector-52:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per topic
> cache Map(Test ->
> List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
> 2014-11-05 23:39:03,197
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
> Test:0: fetched offset = -1: consumed offset = -1,Test:1: fetched offset =
> -1: consumed offset = -1
> 2014-11-05 23:39:03,199
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
> Starting
> 2014-11-05 23:39:03,199
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
> kafka.consumer.ZookeeperConsumerConnector-68:
> [TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
> consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
> 2014-11-05 23:39:03,202
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 35,8 replyHeader:: 35,43,0 request::
> '/brokers/ids,F response:: v{'0}
> 2014-11-05 23:39:03,204
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
> sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
> finished:false header:: 36,4 replyHeader:: 36,43,0 request::
> '/brokers/ids/0,F response::
>
> #7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
>
> 2014-11-05 23:39:03,214
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> INFO kafka.utils.VerifiableProperties-68: Verifying properties
> 2014-11-05 23:39:03,215
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> INFO kafka.utils.VerifiableProperties-68: Property client.id is overridden
> to TestCheck
> 2014-11-05 23:39:03,215
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> INFO kafka.utils.VerifiableProperties-68: Property metadata.broker.list is
> overridden to x.x.x.x:9092
> 2014-11-05 23:39:03,215
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> INFO kafka.utils.VerifiableProperties-68: Property request.timeout.ms is
> overridden to 30000
> 2014-11-05 23:39:03,217
> TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
> INFO kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:x.x.x.x,port:9092 with correlation id 0 for 1 topic(s) Set(Test)
> 2014-11-05 23:39:04,195 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 2 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:04,196 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:04,196 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:04,199 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:04,199 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:04,200 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:04,200 pool-13-thread-1 DEBUG
> kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> topic Test
> 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 3 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:04,201 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:04,201 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:04,205 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:04,205 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:04,206 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:04,206 pool-13-thread-1 ERROR
> kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: Test
> 2014-11-05 23:39:04,206 pool-13-thread-1 INFO
> kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> retrying send. Remaining retries = 9
> 2014-11-05 23:39:04,698
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
> 0x1498251e8680002 after 0ms
> 2014-11-05 23:39:05,205
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
> 0x1498251e8680003 after 0ms
> 2014-11-05 23:39:05,207 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 4 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:05,208 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:05,208 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:05,211 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:05,211 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:05,212 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:05,212 pool-13-thread-1 DEBUG
> kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> topic Test
> 2014-11-05 23:39:05,212 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 5 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:05,213 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:05,213 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:05,217 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:05,217 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:05,217 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:05,217 pool-13-thread-1 ERROR
> kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: Test
> 2014-11-05 23:39:05,218 pool-13-thread-1 INFO
> kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> retrying send. Remaining retries = 8
> 2014-11-05 23:39:06,218 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 6 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:06,219 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:06,219 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:06,224 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:06,224 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:06,225 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:06,225 pool-13-thread-1 DEBUG
> kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> topic Test
> 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 7 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:06,226 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:06,226 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:06,230 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:06,230 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:06,231 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:06,231 pool-13-thread-1 ERROR
> kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: Test
> 2014-11-05 23:39:06,231 pool-13-thread-1 INFO
> kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> retrying send. Remaining retries = 7
> 2014-11-05 23:39:06,699
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
> 0x1498251e8680002 after 0ms
> 2014-11-05 23:39:07,208
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
> 0x1498251e8680003 after 0ms
> 2014-11-05 23:39:07,232 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 8 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:07,233 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:07,233 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:07,236 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:07,236 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:07,237 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:07,237 pool-13-thread-1 DEBUG
> kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> topic Test
> 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 9 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:07,238 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:07,238 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:07,241 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:07,241 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:07,242 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:07,242 pool-13-thread-1 ERROR
> kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: Test
> 2014-11-05 23:39:07,242 pool-13-thread-1 INFO
> kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> retrying send. Remaining retries = 6
> 2014-11-05 23:39:08,243 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 10 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:08,243 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:08,245 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:08,249 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:08,250 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:08,250 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> topic Test
> 2014-11-05 23:39:08,251 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 11 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:08,252 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:08,255 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:08,256 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:08,256 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:08,257 pool-13-thread-1 ERROR
> kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: Test
> 2014-11-05 23:39:08,257 pool-13-thread-1 INFO
> kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> retrying send. Remaining retries = 5
> 2014-11-05 23:39:08,702
> Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
> 0x1498251e8680002 after 0ms
> 2014-11-05 23:39:09,211
>
> pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
> DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
> 0x1498251e8680003 after 0ms
> 2014-11-05 23:39:09,257 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 12 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:09,258 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:09,258 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:09,261 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:09,262 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
> kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> topic Test
> 2014-11-05 23:39:09,263 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 13 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:09,264 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:09,264 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:09,268 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:09,268 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:09,269 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:09,269 pool-13-thread-1 ERROR
> kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: Test
> 2014-11-05 23:39:09,269 pool-13-thread-1 INFO
> kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> retrying send. Remaining retries = 4
> 2014-11-05 23:39:10,270 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 14 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:10,271 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:10,271 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:10,275 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:10,275 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:10,275 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:10,276 pool-13-thread-1 DEBUG
> kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
> topic Test
> 2014-11-05 23:39:10,276 pool-13-thread-1 INFO
> kafka.client.ClientUtils$-68: Fetching metadata from broker
> id:0,host:172.31.25.198,port:9092 with correlation id 15 for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:10,277 pool-13-thread-1 DEBUG
> kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
> (requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
> (requested 102400).
> 2014-11-05 23:39:10,277 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
> producing
> 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
> 2014-11-05 23:39:10,280 pool-13-thread-1 DEBUG
> kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
> Set(Test)
> 2014-11-05 23:39:10,280 pool-13-thread-1 WARN
> kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
> [{TopicMetadata for topic Test ->
> No partition metadata for topic Test due to
> kafka.common.LeaderNotAvailableException}] for topic [Test]: class
> kafka.common.LeaderNotAvailableException
> 2014-11-05 23:39:10,280 pool-13-thread-1 ERROR
> kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: Test
> 2014-11-05 23:39:10,280 pool-13-thread-1 INFO
> kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
> retrying send. Remaining retries = 3
>
>
> Brokers Logs -
>
> [2014-11-05 23:39:02,983] INFO Topic creation
> {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
> [2014-11-05 23:39:02,988] INFO [KafkaApi-0] Auto creation of topic Test
> with 2 partitions and replication factor 1 is successful!
> (kafka.server.KafkaApis)
> [2014-11-05 23:39:03,092] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-05 23:39:03,191] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-05 23:39:04,200] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-05 23:39:04,205] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-05 23:39:05,211] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-05 23:39:05,218] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-05 23:39:06,224] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-05 23:39:06,230] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-05 23:39:07,236] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-05 23:39:07,241] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
>
>
>
>
> On Wed, Nov 5, 2014 at 3:05 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Sameer,
> >
> > Yes, this is the server log. But there seems no abnormal entries in it,
> and
> > it does not cover the same time range as the producer client throwing
> > LeaderNotAvailableException
> > (it was 10/24, 14:30).
> >
> > The reason that I want to check the server log at that same reason is
> > that LeaderNotAvailableException
> > is set by the brokers while handling requests, and its server log will
> show
> > what is the cause it set this error code.
> >
> > Guozhang
> >
> > On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net> wrote:
> >
> > > Hi Guozhang,
> > >
> > > This is the server.log -
> > >
> > > [2014-11-04 20:21:57,510] INFO Verifying properties
> > > (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,545] INFO Property advertised.host.name is
> > overridden
> > > to x.x.x.x (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,545] INFO Property broker.id is overridden to 0
> > > (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,545] INFO Property controlled.shutdown.enable is
> > > overridden to true (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,545] INFO Property host.name is overridden to
> > > 172.31.25.198 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is
> overridden
> > to
> > > false (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden to
> > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,546] INFO Property
> log.retention.check.interval.ms
> > is
> > > overridden to 60000 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,546] INFO Property log.retention.hours is
> overridden
> > > to 168 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is overridden
> > to
> > > 536870912 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,546] INFO Property num.io.threads is overridden
> to 8
> > > (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,547] INFO Property num.network.threads is
> overridden
> > > to 2 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,547] INFO Property num.partitions is overridden
> to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,547] INFO Property port is overridden to 9092
> > > (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,547] INFO Property socket.receive.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,547] INFO Property socket.request.max.bytes is
> > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,547] INFO Property socket.send.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is overridden
> > to
> > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,548] INFO Property
> zookeeper.connection.timeout.ms
> > is
> > > overridden to 1000000 (kafka.utils.VerifiableProperties)
> > > [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
> > > (kafka.server.KafkaServer)
> > > [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to
> zookeeper
> > on
> > > localhost:2181 (kafka.server.KafkaServer)
> > > [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs' not
> found,
> > > creating it. (kafka.log.LogManager)
> > > [2014-11-04 20:21:57,748] INFO Starting log cleanup with a period of
> > 60000
> > > ms. (kafka.log.LogManager)
> > > [2014-11-04 20:21:57,752] INFO Starting log flusher with a default
> period
> > > of 9223372036854775807 ms. (kafka.log.LogManager)
> > > [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
> > > ip-172-31-25-198.us-west-1.compute.internal:9092.
> > (kafka.network.Acceptor)
> > > [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0], Started
> > > (kafka.network.SocketServer)
> > > [2014-11-04 20:21:57,852] INFO Will not load MX4J, mx4j-tools.jar is
> not
> > in
> > > the classpath (kafka.utils.Mx4jLoader$)
> > > [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
> > > (kafka.server.ZookeeperLeaderElector)
> > > [2014-11-04 20:21:57,967] INFO Registered broker 0 at path
> /brokers/ids/0
> > > with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
> > > [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
> > > (kafka.server.KafkaServer)
> > > [2014-11-04 20:21:58,050] INFO New leader is 0
> > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > > [2014-11-04 20:22:01,809] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:22:11,812] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:29,084] INFO Topic creation
> > > {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
> > > [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of topic Test
> > > with 2 partitions and replication factor 1 is successful!
> > > (kafka.server.KafkaApis)
> > > [2014-11-04 20:30:29,280] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:29,301] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:30,310] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:30,315] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:31,320] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:31,325] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:32,331] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:32,336] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:33,343] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:33,347] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:34,356] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:34,360] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:35,366] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:35,372] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:36,377] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:36,382] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:37,387] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:37,394] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:38,399] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:38,403] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:39,410] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:39,414] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:40,419] INFO Closing socket connection to /
> > 172.31.25.198
> > > .
> > > (kafka.network.Processor)
> > > [2014-11-04 20:30:41,602] INFO Topic creation
> > > {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
> > >
> > >
> > >
> > > thanks
> > >
> > >
> > > On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > The server.log should be the broker logs.
> > > >
> > > > Guozhang
> > > >
> > > > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <sy...@ailive.net>
> > wrote:
> > > >
> > > > > I do not see any other logs besides these.
> > > > > I checked the log4j.properties and where the logs are getting
> stored.
> > > > > The only other logs (apart from the controller logs) in the
> directory
> > > are
> > > > > zookeeper-gc.log,kafkaServer-gc.log, server.log and
> state-change.log
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <wangguoz@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Sorry for getting late on this Sameer.
> > > > > >
> > > > > > The logs you showed above are from controller logs, do you see
> > > brokers
> > > > > logs
> > > > > > besides that?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <sy...@ailive.net>
> > > wrote:
> > > > > >
> > > > > > > Hi Guozhang,
> > > > > > >
> > > > > > > Is there any clue in the above data as to what is wrong?
> > > > > > >
> > > > > > > thanks
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <syami@ailive.net
> >
> > > > wrote:
> > > > > > >
> > > > > > > > This is the one from the file specified in log4j.properties -
> > > > > > > >
> > > > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on
> > > Controller
> > > > > 0]:
> > > > > > > > Topic change listener fired for path /brokers/topics with
> > > children
> > > > > Test
> > > > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener on
> > Controller
> > > > 0]:
> > > > > > New
> > > > > > > > topics: [Set(Test)], deleted topics: [Set()], new partition
> > > replica
> > > > > > > > assignment [Map([Test,0] -> List(0), [Test,1] -> List(0))]
> > > > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New topic
> > creation
> > > > > > > callback
> > > > > > > > for [Test,0],[Test,1] (kafka.controller.KafkaController)
> > > > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New partition
> > > > creation
> > > > > > > > callback for [Test,0],[Test,1]
> > (kafka.controller.KafkaController)
> > > > > > > > [2014-10-28 22:38:36,095] INFO [Partition state machine on
> > > > Controller
> > > > > > 0]:
> > > > > > > > Invoking state change to NewPartition for partitions
> > > > > [Test,0],[Test,1]
> > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine on
> > > controller
> > > > > 0]:
> > > > > > > > Invoking state change to NewReplica for replicas
> > > > > > > >
> > > > [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > > > [2014-10-28 22:38:36,111] INFO [Partition state machine on
> > > > Controller
> > > > > > 0]:
> > > > > > > > Invoking state change to OnlinePartition for partitions
> > > > > > [Test,0],[Test,1]
> > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state machine on
> > > > > Controller
> > > > > > > 0]:
> > > > > > > > Live assigned replicas for partition [Test,0] are: [List(0)]
> > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state machine on
> > > > > Controller
> > > > > > > 0]:
> > > > > > > > Initializing leader and isr for partition [Test,0] to
> > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state machine on
> > > > > Controller
> > > > > > > 0]:
> > > > > > > > Live assigned replicas for partition [Test,1] are: [List(0)]
> > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state machine on
> > > > > Controller
> > > > > > > 0]:
> > > > > > > > Initializing leader and isr for partition [Test,1] to
> > > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine on
> > > controller
> > > > > 0]:
> > > > > > > > Invoking state change to OnlineReplica for replicas
> > > > > > > >
> > > > [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > > > [2014-10-28 22:39:28,160] ERROR
> > > > > [Controller-0-to-broker-0-send-thread],
> > > > > > > > Controller 0's connection to broker
> id:0,host:x.x.x.x,port:9092
> > > was
> > > > > > > > unsuccessful (kafka.controller.RequestSendThread)
> > > > > > > > java.net.ConnectException: Connection timed out
> > > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > > > at sun.nio.ch.Net.connect(Net.java:449)
> > > > > > > > at
> > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> > > > > > > > at
> > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > > > at
> > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > > > [2014-10-28 22:39:28,460] ERROR
> > > > > [Controller-0-to-broker-0-send-thread],
> > > > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr request with
> > > > > > correlation
> > > > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092. Reconnecting to
> > > broker.
> > > > > > > > (kafka.controller.RequestSendThread)
> > > > > > > > java.nio.channels.ClosedChannelException
> > > > > > > > at
> > > kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > > > at
> > > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > > >
> > > > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
> > > wangguoz@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Sameer, the one you sent me is from the producer logs, not
> the
> > > > > broker
> > > > > > > >> logs.
> > > > > > > >> And /tmp/kafka-logs actually store the message log (i.e. the
> > > > "data")
> > > > > > > >> instead of the server logs. Their locations should be
> > specified
> > > in
> > > > > > your
> > > > > > > >> log4j.properties file.
> > > > > > > >>
> > > > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <
> > syami@ailive.net>
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Is this what you are looking for -
> > > > > > > >> >
> > > > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > > 172.31.16.31:9092
> > > > > for
> > > > > > > >> > producing
> > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > > 172.31.16.31:9092
> > > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
> > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> metadata
> > > for
> > > > 1
> > > > > > > >> topic(s)
> > > > > > > >> > Set(Test)
> > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
> > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> fetching
> > > > > metadata
> > > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > > >> > No partition metadata for topic Test due to
> > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> [Test]:
> > > > class
> > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
> > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> > > partition
> > > > > info
> > > > > > > for
> > > > > > > >> > topic Test
> > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
> > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
> broker
> > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 9
> for 1
> > > > > > topic(s)
> > > > > > > >> > Set(Test)
> > > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
> > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > > > SO_TIMEOUT =
> > > > > > > 10000
> > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> > > SO_SNDBUF
> > > > =
> > > > > > > 102400
> > > > > > > >> > (requested 102400).
> > > > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > > 172.31.16.31:9092
> > > > > for
> > > > > > > >> > producing
> > > > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > > 172.31.16.31:9092
> > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
> > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> metadata
> > > for
> > > > 1
> > > > > > > >> topic(s)
> > > > > > > >> > Set(Test)
> > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
> > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> fetching
> > > > > metadata
> > > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > > >> > No partition metadata for topic Test due to
> > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> [Test]:
> > > > class
> > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
> > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
> > collate
> > > > > > > messages
> > > > > > > >> by
> > > > > > > >> > topic, partition due to: Failed to fetch topic metadata
> for
> > > > topic:
> > > > > > > Test
> > > > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for
> > 1000
> > > > ms
> > > > > > > before
> > > > > > > >> > retrying send. Remaining retries = 6
> > > > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
> > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
> broker
> > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 10
> for
> > 1
> > > > > > topic(s)
> > > > > > > >> > Set(Test)
> > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
> > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > > > SO_TIMEOUT =
> > > > > > > 10000
> > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> > > SO_SNDBUF
> > > > =
> > > > > > > 102400
> > > > > > > >> > (requested 102400).
> > > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > > 172.31.16.31:9092
> > > > > for
> > > > > > > >> > producing
> > > > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > > 172.31.16.31:9092
> > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> metadata
> > > for
> > > > 1
> > > > > > > >> topic(s)
> > > > > > > >> > Set(Test)
> > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
> > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> fetching
> > > > > metadata
> > > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > > >> > No partition metadata for topic Test due to
> > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> [Test]:
> > > > class
> > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> > > partition
> > > > > info
> > > > > > > for
> > > > > > > >> > topic Test
> > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
> > > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from
> broker
> > > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 11
> for
> > 1
> > > > > > topic(s)
> > > > > > > >> > Set(Test)
> > > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
> > > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > > > SO_TIMEOUT =
> > > > > > > 10000
> > > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> > > SO_SNDBUF
> > > > =
> > > > > > > 102400
> > > > > > > >> > (requested 102400).
> > > > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > > 172.31.16.31:9092
> > > > > for
> > > > > > > >> > producing
> > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > > 172.31.16.31:9092
> > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
> > > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched
> metadata
> > > for
> > > > 1
> > > > > > > >> topic(s)
> > > > > > > >> > Set(Test)
> > > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
> > > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while
> fetching
> > > > > metadata
> > > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > > >> > No partition metadata for topic Test due to
> > > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic
> [Test]:
> > > > class
> > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
> > > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
> > collate
> > > > > > > messages
> > > > > > > >> by
> > > > > > > >> > topic, partition due to: Failed to fetch topic metadata
> for
> > > > topic:
> > > > > > > Test
> > > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
> > > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for
> > 1000
> > > > ms
> > > > > > > before
> > > > > > > >> > retrying send. Remaining retries = 5
> > > > > > > >> >
> > > > > > > >> > As for /tmp/kafka-logs, I can see only these files -
> > > > > > > >> > recovery-point-offset-checkpoint
> > > replication-offset-checkpoint
> > > > > > > >> >
> > > > > > > >> > I do not see the usual Kafka log files here.
> > > > > > > >> >
> > > > > > > >> > thanks
> > > > > > > >> > Sameer
> > > > > > > >> >
> > > > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
> > > > > wangguoz@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > > Sameer, could you check broker logs and see if it has
> > > similar
> > > > > > > >> > > LeaderNotAvailable exceptions?
> > > > > > > >> > >
> > > > > > > >> > > Guozhang
> > > > > > > >> > >
> > > > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
> > > > syami@ailive.net
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Yes. I can telnet from outside using the public ip of
> > the
> > > > > > instance
> > > > > > > >> and
> > > > > > > >> > > port
> > > > > > > >> > > > of 9092
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <
> > > > > > > >> roger.hoover@gmail.com
> > > > > > > >> > >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Just a guess but could it be a firewall issue? Did
> > you
> > > > > enable
> > > > > > > >> > > > connections
> > > > > > > >> > > > > to port 9092 from outside EC2 in a security group?
> > Can
> > > > you
> > > > > > > >> telnet to
> > > > > > > >> > > > each
> > > > > > > >> > > > > broker IP and port?
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami <
> > > > > > syami@ailive.net
> > > > > > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > There was a typo earlier.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > This is the output -
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Topic:Test PartitionCount:2 ReplicationFactor:1
> > > > > > Configs:
> > > > > > > >> > > > > > Topic: Test Partition: 0 Leader: 0
> > > > > > Replicas:
> > > > > > > 0
> > > > > > > >> > > > Isr: 0
> > > > > > > >> > > > > > Topic: Test Partition: 1 Leader: 0
> > > > > > Replicas:
> > > > > > > 0
> > > > > > > >> > > > Isr: 0
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > thanks
> > > > > > > >> > > > > > Sameer
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang Wang <
> > > > > > > >> wangguoz@gmail.com
> > > > > > > >> > >
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Sameer,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > The topic name is not consistent from your
> > describe
> > > > > > outputs.
> > > > > > > >> > > Normally
> > > > > > > >> > > > > > they
> > > > > > > >> > > > > > > should be either
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Topic:*Test* PartitionCount:2
> > > > ReplicationFactor:1
> > > > > > > >> > > Configs:
> > > > > > > >> > > > > > > Topic: *Test* Partition: 0
> Leader: 0
> > > > > > > >> Replicas: 0
> > > > > > > >> > > > > > Isr: 0
> > > > > > > >> > > > > > > Topic: *Test* Partition: 1
> Leader: 0
> > > > > > > >> Replicas: 0
> > > > > > > >> > > > > > Isr: 0
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Or
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
> > > > > > > ReplicationFactor:1
> > > > > > > >> > > > > Configs:
> > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 0
> > > Leader:
> > > > 0
> > > > > > > >> > > Replicas:
> > > > > > > >> > > > 0
> > > > > > > >> > > > > > > Isr: 0
> > > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 1
> > > Leader:
> > > > 0
> > > > > > > >> > > Replicas:
> > > > > > > >> > > > 0
> > > > > > > >> > > > > > > Isr: 0
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Did you manually modify the topic name in ZK?
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Guozhang
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer Yami <
> > > > > > > >> syami@ailive.net>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > Hi Guozhang,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > I was wondering if there is anything wrong in
> > the
> > > > > above
> > > > > > > >> > > --describe
> > > > > > > >> > > > > > data.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > thanks
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer Yami <
> > > > > > > >> syami@ailive.net
> > > > > > > >> > >
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > This is what I get -
> > > > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper
> > localhost:2181
> > > > > > > --describe
> > > > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
> > > > > > > >> ReplicationFactor:1
> > > > > > > >> > > > > > Configs:
> > > > > > > >> > > > > > > > > Topic: Test Partition: 0
> > Leader: 0
> > > > > > > >> > Replicas: 0
> > > > > > > >> > > > > > > Isr: 0
> > > > > > > >> > > > > > > > > Topic: Test Partition: 1
> > Leader: 0
> > > > > > > >> > Replicas: 0
> > > > > > > >> > > > > > > Isr: 0
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > The topics are being created at run-time.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > thanks
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM, Guozhang
> > Wang <
> > > > > > > >> > > > wangguoz@gmail.com
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >> Could you use the topic tools "--describe"
> to
> > > > check
> > > > > > if
> > > > > > > >> the
> > > > > > > >> > > topic
> > > > > > > >> > > > > > > Test's
> > > > > > > >> > > > > > > > >> partitions leaders are available?
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > >
> > > > http://kafka.apache.org/documentation.html#basic_ops_add_topic
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > > > > > > > >> Guozhang
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM, Sameer
> Yami
> > <
> > > > > > > >> > > syami@ailive.net>
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > > > > > > > >> > That is not the case.
> > > > > > > >> > > > > > > > >> > This detailed error is below. Due to this
> > > > error ,
> > > > > > the
> > > > > > > >> > > producer
> > > > > > > >> > > > > is
> > > > > > > >> > > > > > > not
> > > > > > > >> > > > > > > > >> able
> > > > > > > >> > > > > > > > >> > to send messages
> > > > > > > >> > > > > > > > >> >
> > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG
> > > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-52:
> > > > > > > >> > > > > > > > >> > Getting broker partition info for topic
> > Test
> > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main INFO
> > > > > > > >> > > > > kafka.client.ClientUtils$-68:
> > > > > > > >> > > > > > > > >> Fetching
> > > > > > > >> > > > > > > > >> > metadata from broker
> > > > > > > id:0,host:54.215.200.61,port:9092
> > > > > > > >> > with
> > > > > > > >> > > > > > > > correlation
> > > > > > > >> > > > > > > > >> id
> > > > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
> > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG
> > > > > > > >> > > > > > > kafka.network.BlockingChannel-52:
> > > > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT = 10000
> > > > (requested
> > > > > > > >> 10000),
> > > > > > > >> > > > > > SO_RCVBUF
> > > > > > > >> > > > > > > =
> > > > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF = 102400
> > > > > > (requested
> > > > > > > >> > > 102400).
> > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main INFO
> > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092 for
> > > producing
> > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main INFO
> > > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > > > >> > > > > > > > >> > Disconnecting from 54.215.200.61:9092
> > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG
> > > > > > > >> > > > > kafka.client.ClientUtils$-52:
> > > > > > > >> > > > > > > > >> > Successfully fetched metadata for 1
> > topic(s)
> > > > > > > Set(Test)
> > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main WARN
> > > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-83:
> > > > > > > >> > > > > > > > >> > Error while fetching metadata
> > [{TopicMetadata
> > > > for
> > > > > > > topic
> > > > > > > >> > Test
> > > > > > > >> > > > ->
> > > > > > > >> > > > > > > > >> > No partition metadata for topic Test due
> to
> > > > > > > >> > > > > > > > >> >
> kafka.common.LeaderNotAvailableException}]
> > > for
> > > > > > topic
> > > > > > > >> > [Test]:
> > > > > > > >> > > > > class
> > > > > > > >> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR
> > > > > > > >> > > > > > > > >> >
> > kafka.producer.async.DefaultEventHandler-97:
> > > > > Failed
> > > > > > > to
> > > > > > > >> > > collate
> > > > > > > >> > > > > > > > messages
> > > > > > > >> > > > > > > > >> by
> > > > > > > >> > > > > > > > >> > topic, partition due to: Failed to fetch
> > > topic
> > > > > > > metadata
> > > > > > > >> > for
> > > > > > > >> > > > > topic:
> > > > > > > >> > > > > > > > Test
> > > > > > > >> > > > > > > > >> >
> > > > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM, Guozhang
> > > Wang
> > > > <
> > > > > > > >> > > > > > wangguoz@gmail.com>
> > > > > > > >> > > > > > > > >> wrote:
> > > > > > > >> > > > > > > > >> >
> > > > > > > >> > > > > > > > >> > > Could you see if this is the case:
> > > > > > > >> > > > > > > > >> > >
> > > > > > > >> > > > > > > > >> > >
> > > > > > > >> > > > > > > > >> > >
> > > > > > > >> > > > > > > > >> >
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> > > > > > > >> > > > > > > > >> > > ?
> > > > > > > >> > > > > > > > >> > >
> > > > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM,
> Sameer
> > > > Yami <
> > > > > > > >> > > > > syami@ailive.net
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > >> wrote:
> > > > > > > >> > > > > > > > >> > >
> > > > > > > >> > > > > > > > >> > > > Hi,
> > > > > > > >> > > > > > > > >> > > >
> > > > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper test
> setup
> > > on a
> > > > > > > single
> > > > > > > >> EC2
> > > > > > > >> > > > > > machine.
> > > > > > > >> > > > > > > > The
> > > > > > > >> > > > > > > > >> > > > host.name is the private ip address
> > and
> > > > the
> > > > > > > >> > > > > > > advertised.host.name
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > >> > the
> > > > > > > >> > > > > > > > >> > > > public address in server.properties.
> > > > > > > >> > > > > > > > >> > > >
> > > > > > > >> > > > > > > > >> > > > We are trying to publish from outside
> > EC2
> > > > > using
> > > > > > > the
> > > > > > > >> > > public
> > > > > > > >> > > > > ip
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > >> > EC2
> > > > > > > >> > > > > > > > >> > > > machine but get this error on the
> > client
> > > > > side -
> > > > > > > >> > > > > > > > >> > > > No partition metadata for topic test
> > due
> > > to
> > > > > > > >> > > > > > > > >> > > >
> > > kafka.common.LeaderNotAvailableException}]
> > > > > for
> > > > > > > >> topic
> > > > > > > >> > > > [test]:
> > > > > > > >> > > > > > > class
> > > > > > > >> > > > > > > > >> > > >
> > kafka.common.LeaderNotAvailableException
> > > > > > > >> > > > > > > > >> > > >
> > > > > > > >> > > > > > > > >> > > > The zookeeper has the public ip
> stored
> > in
> > > > > > > >> > > /brokers/ids/0.
> > > > > > > >> > > > > > > > >> > > > We have tried multiple combinations
> but
> > > it
> > > > > > works
> > > > > > > >> ONLY
> > > > > > > >> > > when
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > > >> > advertised
> > > > > > > >> > > > > > > > >> > > > host name is not used and the client
> is
> > > > > > internal
> > > > > > > to
> > > > > > > >> > EC2.
> > > > > > > >> > > > > > > > >> > > >
> > > > > > > >> > > > > > > > >> > > > Please let me know what the problem
> > could
> > > > be.
> > > > > > > >> > > > > > > > >> > > >
> > > > > > > >> > > > > > > > >> > > > thanks
> > > > > > > >> > > > > > > > >> > > > Sameer
> > > > > > > >> > > > > > > > >> > > >
> > > > > > > >> > > > > > > > >> > >
> > > > > > > >> > > > > > > > >> > >
> > > > > > > >> > > > > > > > >> > >
> > > > > > > >> > > > > > > > >> > > --
> > > > > > > >> > > > > > > > >> > > -- Guozhang
> > > > > > > >> > > > > > > > >> > >
> > > > > > > >> > > > > > > > >> >
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > > > > > > > >> --
> > > > > > > >> > > > > > > > >> -- Guozhang
> > > > > > > >> > > > > > > > >>
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > --
> > > > > > > >> > > > > > > -- Guozhang
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > --
> > > > > > > >> > > -- Guozhang
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> -- Guozhang
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Cannot connect to Kafka from outside of EC2
Posted by Sameer Yami <sy...@ailive.net>.
The server.log was taken separately.
We ran the test again and the server and producer logs are below (to get
same timings).
Thanks!
------------------------------------------------------------------------------------
Producer Logs -
2014-11-05 23:38:58,693
Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
0x1498251e8680002 after 0ms
2014-11-05 23:39:00,695
Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
0x1498251e8680002 after 0ms
2014-11-05 23:39:02,696
Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
0x1498251e8680002 after 0ms
2014-11-05 23:39:02,828 pool-13-thread-2 INFO
kafka.utils.VerifiableProperties-68: Verifying properties
2014-11-05 23:39:02,829 pool-13-thread-2 INFO
kafka.utils.VerifiableProperties-68: Property auto.commit.interval.ms is
overridden to 1000
2014-11-05 23:39:02,829 pool-13-thread-2 INFO
kafka.utils.VerifiableProperties-68: Property auto.offset.reset is
overridden to smallest
2014-11-05 23:39:02,829 pool-13-thread-2 INFO
kafka.utils.VerifiableProperties-68: Property consumer.timeout.ms is
overridden to 10
2014-11-05 23:39:02,829 pool-13-thread-2 INFO
kafka.utils.VerifiableProperties-68: Property group.id is overridden to
TestCheck
2014-11-05 23:39:02,830 pool-13-thread-2 WARN
kafka.utils.VerifiableProperties-83: Property serializer.class is not valid
2014-11-05 23:39:02,830 pool-13-thread-2 INFO
kafka.utils.VerifiableProperties-68: Property zookeeper.connect is
overridden to 172.31.25.198:2181
2014-11-05 23:39:02,831 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Connecting to
zookeeper instance at 172.31.25.198:2181
2014-11-05 23:39:02,831 pool-13-thread-2 DEBUG
org.I0Itec.zkclient.ZkConnection-63: Creating new ZookKeeper instance to
connect to 172.31.25.198:2181.
2014-11-05 23:39:02,831 pool-13-thread-2 INFO
org.apache.zookeeper.ZooKeeper-379: Initiating client connection,
connectString=172.31.25.198:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@3903b165
2014-11-05 23:39:02,831 ZkClient-EventThread-29-172.31.25.198:2181 INFO
org.I0Itec.zkclient.ZkEventThread-64: Starting ZkClient event thread.
2014-11-05 23:39:02,831 pool-13-thread-1 INFO
kafka.utils.VerifiableProperties-68: Verifying properties
2014-11-05 23:39:02,836 pool-13-thread-2-SendThread() INFO
org.apache.zookeeper.ClientCnxn-1061: Opening socket connection to server /
172.31.25.198:2181
2014-11-05 23:39:02,836 pool-13-thread-1 WARN
kafka.utils.VerifiableProperties-83: Property batch.size is not valid
2014-11-05 23:39:02,832 pool-13-thread-2 DEBUG
org.I0Itec.zkclient.ZkClient-878: Awaiting connection to Zookeeper server
2014-11-05 23:39:02,836 pool-13-thread-1 INFO
kafka.utils.VerifiableProperties-68: Property message.send.max.retries is
overridden to 10
2014-11-05 23:39:02,836 pool-13-thread-2 DEBUG
org.I0Itec.zkclient.ZkClient-628: Waiting for keeper state SyncConnected
2014-11-05 23:39:02,837 pool-13-thread-1 INFO
kafka.utils.VerifiableProperties-68: Property metadata.broker.list is
overridden to 172.31.25.198:9092
2014-11-05 23:39:02,837 pool-13-thread-1 INFO
kafka.utils.VerifiableProperties-68: Property retry.backoff.ms is
overridden to 1000
2014-11-05 23:39:02,837 pool-13-thread-1 INFO
kafka.utils.VerifiableProperties-68: Property serializer.class is
overridden to kafka.serializer.StringEncoder
2014-11-05 23:39:02,837
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
INFO org.apache.zookeeper.ClientCnxn-950: Socket connection established to
ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181, initiating
session
2014-11-05 23:39:02,838
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-999: Session establishment request
sent on ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181
2014-11-05 23:39:02,837 pool-13-thread-1 WARN
kafka.utils.VerifiableProperties-83: Property zk.connectiontimeout.ms is
not valid
2014-11-05 23:39:02,841
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
INFO org.apache.zookeeper.ClientCnxn-739: Session establishment complete on
server ip-172-31-25-198.us-west-1.compute.internal/172.31.25.198:2181,
sessionid = 0x1498251e8680003, negotiated timeout = 6000
2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
state:SyncConnected type:None path:null
2014-11-05 23:39:02,841 pool-13-thread-2-EventThread INFO
org.I0Itec.zkclient.ZkClient-449: zookeeper state changed (SyncConnected)
2014-11-05 23:39:02,841 pool-13-thread-2-EventThread DEBUG
org.I0Itec.zkclient.ZkClient-395: Leaving process event
2014-11-05 23:39:02,841 pool-13-thread-2 DEBUG
org.I0Itec.zkclient.ZkClient-638: State is SyncConnected
2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
kafka.utils.KafkaScheduler-52: Initializing task scheduler.
2014-11-05 23:39:02,842 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting auto
committer every 1000 ms
2014-11-05 23:39:02,842 pool-13-thread-2 DEBUG
kafka.utils.KafkaScheduler-52: Scheduling task kafka-consumer-autocommit
with initial delay 1000 ms and period 1000 ms.
2014-11-05 23:39:02,843 pool-13-thread-2 DEBUG
kafka.consumer.ZookeeperConsumerConnector-52:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], entering consume
2014-11-05 23:39:02,843 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin registering
consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
2014-11-05 23:39:02,846
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 1,1 replyHeader:: 1,29,-101 request::
'/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
response::
2014-11-05 23:39:02,854
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 2,1 replyHeader:: 2,30,-101 request::
'/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
2014-11-05 23:39:02,858
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 3,1 replyHeader:: 3,31,0 request::
'/consumers/TestCheck,,v{s{31,s{'world,'anyone}}},0 response::
'/consumers/TestCheck
2014-11-05 23:39:02,861
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 4,1 replyHeader:: 4,32,0 request::
'/consumers/TestCheck/ids,,v{s{31,s{'world,'anyone}}},0 response::
'/consumers/TestCheck/ids
2014-11-05 23:39:02,866
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 5,1 replyHeader:: 5,33,0 request::
'/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,v{s{31,s{'world,'anyone}}},1
response::
'/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
2014-11-05 23:39:02,867 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end registering
consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 in ZK
2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
kafka.consumer.ZookeeperConsumerConnector-52:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Adding topicThreadId
(Test,TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0) and queue [] to
topicThreadIdAndQueues data structure
2014-11-05 23:39:02,868
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], starting watcher
executor thread for consumer
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362
2014-11-05 23:39:02,868 pool-13-thread-2 DEBUG
kafka.consumer.ZookeeperConsumerConnector-52:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], adding topic Test and
1 streams to map.
2014-11-05 23:39:02,870
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 6,3 replyHeader:: 6,33,0 request::
'/consumers/TestCheck/ids,T response::
s{32,32,1415230742859,1415230742859,0,1,0,0,0,1,33}
2014-11-05 23:39:02,872
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 7,8 replyHeader:: 7,33,0 request::
'/consumers/TestCheck/ids,T response::
v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
2014-11-05 23:39:02,873
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 8,3 replyHeader:: 8,33,-101 request::
'/brokers/topics/Test,T response::
2014-11-05 23:39:02,873 pool-13-thread-2 DEBUG
org.I0Itec.zkclient.ZkClient-137: Subscribed data changes for
/brokers/topics/Test
2014-11-05 23:39:02,873 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
2014-11-05 23:39:02,875 pool-13-thread-1 DEBUG
kafka.producer.async.DefaultEventHandler-52: Handling 1 events
2014-11-05 23:39:02,877
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 9,8 replyHeader:: 9,33,0 request::
'/brokers/ids,F response:: v{'0}
2014-11-05 23:39:02,879
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 10,4 replyHeader:: 10,33,0 request::
'/brokers/ids/0,F response::
#7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
2014-11-05 23:39:02,887
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 11,4 replyHeader:: 11,33,0 request::
'/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
response::
#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
2014-11-05 23:39:02,896
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 12,8 replyHeader:: 12,33,0 request::
'/consumers/TestCheck/ids,T response::
v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
2014-11-05 23:39:02,898
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 13,4 replyHeader:: 13,33,0 request::
'/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
response::
#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
2014-11-05 23:39:02,908
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 14,8 replyHeader:: 14,33,0 request::
'/brokers/ids,F response:: v{'0}
2014-11-05 23:39:02,910
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 15,4 replyHeader:: 15,33,0 request::
'/brokers/ids/0,F response::
#7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
2014-11-05 23:39:02,917 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 0 for 1 topic(s)
Set(Test)
2014-11-05 23:39:02,917
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 16,4 replyHeader:: 16,33,-101 request::
'/brokers/topics/Test,T response::
2014-11-05 23:39:02,920 pool-13-thread-2 DEBUG kafka.utils.ZkUtils$-52:
Partition map for /brokers/topics/Test is Map()
2014-11-05 23:39:02,920 pool-13-thread-2 INFO
kafka.consumer.ConsumerFetcherManager-68:
[ConsumerFetcherManager-1415230742842] Stopping leader finder thread
2014-11-05 23:39:02,921 pool-13-thread-2 INFO
kafka.consumer.ConsumerFetcherManager-68:
[ConsumerFetcherManager-1415230742842] Stopping all fetchers
2014-11-05 23:39:02,921 pool-13-thread-2 INFO
kafka.consumer.ConsumerFetcherManager-68:
[ConsumerFetcherManager-1415230742842] All connections stopped
2014-11-05 23:39:02,921 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all relevant
queues for this fetcher
2014-11-05 23:39:02,921 pool-13-thread-2 DEBUG
kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
this consumer iterator
2014-11-05 23:39:02,921 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
chunks in all the consumer message iterators
2014-11-05 23:39:02,922 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all offsets
after clearing the fetcher queues
2014-11-05 23:39:02,922 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing partition
ownership
2014-11-05 23:39:02,922 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the following
partitions: ArrayBuffer() for topic Test with consumers:
List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
2014-11-05 23:39:02,922 pool-13-thread-2 WARN
kafka.consumer.ZookeeperConsumerConnector-83:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], No broker partitions
consumed by consumer thread
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 for topic Test
2014-11-05 23:39:02,922 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
kafka.consumer.ZookeeperConsumerConnector-52:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per topic
cache Map(Test -> ArrayBuffer())
2014-11-05 23:39:02,923 pool-13-thread-2 DEBUG
kafka.consumer.ZookeeperConsumerConnector-52:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per topic
cache Map(Test -> List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
2014-11-05 23:39:02,923 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
2014-11-05 23:39:02,927 pool-13-thread-2 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
2014-11-05 23:39:02,928
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
Starting
2014-11-05 23:39:02,930 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:02,930 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:02,985
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-783: Got notification
sessionid:0x1498251e8680003
2014-11-05 23:39:02,986
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-800: Got WatchedEvent
state:SyncConnected type:NodeCreated path:/brokers/topics/Test for
sessionid 0x1498251e8680003
2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
org.I0Itec.zkclient.ZkClient-351: Received event: WatchedEvent
state:SyncConnected type:NodeCreated path:/brokers/topics/Test
2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
org.I0Itec.zkclient.ZkEventThread-88: New event: ZkEvent[Data of
/brokers/topics/Test changed sent to
kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
]
2014-11-05 23:39:02,987 pool-13-thread-2-EventThread DEBUG
org.I0Itec.zkclient.ZkClient-395: Leaving process event
2014-11-05 23:39:02,988 ZkClient-EventThread-29-172.31.25.198:2181 DEBUG
org.I0Itec.zkclient.ZkEventThread-69: Delivering event #1 ZkEvent[Data of
/brokers/topics/Test changed sent to
kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@523b24ee
]
2014-11-05 23:39:02,991
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 17,3 replyHeader:: 17,37,0 request::
'/brokers/topics/Test,T response::
s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
2014-11-05 23:39:02,993
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 18,4 replyHeader:: 18,37,0 request::
'/brokers/topics/Test,T response::
#7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181 INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Topic info for path
/brokers/topics/Test changed to
{"version":1,"partitions":{"1":[0],"0":[0]}}, triggering rebalance
2014-11-05 23:39:02,994 ZkClient-EventThread-29-172.31.25.198:2181 DEBUG
org.I0Itec.zkclient.ZkEventThread-79: Delivering event #1 done
2014-11-05 23:39:02,995
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], begin rebalancing
consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
2014-11-05 23:39:02,996
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 19,8 replyHeader:: 19,37,0 request::
'/brokers/ids,F response:: v{'0}
2014-11-05 23:39:02,998
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 20,4 replyHeader:: 20,37,0 request::
'/brokers/ids/0,F response::
#7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
2014-11-05 23:39:03,008
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 21,4 replyHeader:: 21,37,0 request::
'/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
response::
#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
2014-11-05 23:39:03,014
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 22,8 replyHeader:: 22,37,0 request::
'/consumers/TestCheck/ids,T response::
v{'TestCheck_ip-172-31-25-198-1415230742830-f3dfc362}
2014-11-05 23:39:03,018
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 23,4 replyHeader:: 23,37,0 request::
'/consumers/TestCheck/ids/TestCheck_ip-172-31-25-198-1415230742830-f3dfc362,F
response::
#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b22486561727462656174223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343135323330373432383433227d,s{33,33,1415230742861,1415230742861,0,0,0,92748555640897539,91,0,33}
2014-11-05 23:39:03,041
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 24,8 replyHeader:: 24,37,0 request::
'/brokers/ids,F response:: v{'0}
2014-11-05 23:39:03,043
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 25,4 replyHeader:: 25,37,0 request::
'/brokers/ids/0,F response::
#7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
2014-11-05 23:39:03,058
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 26,4 replyHeader:: 26,37,0 request::
'/brokers/topics/Test,T response::
#7b2276657273696f6e223a312c22706172746974696f6e73223a7b2231223a5b305d2c2230223a5b305d7d7d,s{37,37,1415230742983,1415230742983,0,0,0,0,44,0,37}
2014-11-05 23:39:03,074
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.utils.ZkUtils$-52: Partition map for /brokers/topics/Test is Map(1 ->
List(0), 0 -> List(0))
2014-11-05 23:39:03,075
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ConsumerFetcherManager-68:
[ConsumerFetcherManager-1415230742842] Stopping leader finder thread
2014-11-05 23:39:03,082
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
Shutting down
2014-11-05 23:39:03,082 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:03,089
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
Stopped
2014-11-05 23:39:03,089
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
Shutdown completed
2014-11-05 23:39:03,089
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ConsumerFetcherManager-68:
[ConsumerFetcherManager-1415230742842] Stopping all fetchers
2014-11-05 23:39:03,090
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ConsumerFetcherManager-68:
[ConsumerFetcherManager-1415230742842] All connections stopped
2014-11-05 23:39:03,090
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared all relevant
queues for this fetcher
2014-11-05 23:39:03,090
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.consumer.ConsumerIterator-52: Clearing the current data chunk for
this consumer iterator
2014-11-05 23:39:03,091
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Cleared the data
chunks in all the consumer message iterators
2014-11-05 23:39:03,091
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Committing all offsets
after clearing the fetcher queues
2014-11-05 23:39:03,091
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Releasing partition
ownership
2014-11-05 23:39:03,094 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:03,096
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 rebalancing the following
partitions: ArrayBuffer(0, 1) for topic Test with consumers:
List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0)
2014-11-05 23:39:03,098
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
partition 0
2014-11-05 23:39:03,100
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 27,4 replyHeader:: 27,37,-101 request::
'/consumers/TestCheck/offsets/Test/0,F response::
2014-11-05 23:39:03,116
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.consumer.PartitionTopicInfo-52: initial consumer offset of Test:0:
fetched offset = -1: consumed offset = -1 is -1
2014-11-05 23:39:03,117
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:0:
fetched offset = -1: consumed offset = -1 is -1
2014-11-05 23:39:03,118
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.consumer.ZookeeperConsumerConnector-52:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:0: fetched offset
= -1: consumed offset = -1 selected new offset -1
2014-11-05 23:39:03,169
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 attempting to claim
partition 1
2014-11-05 23:39:03,171
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 28,4 replyHeader:: 28,37,-101 request::
'/consumers/TestCheck/offsets/Test/1,F response::
2014-11-05 23:39:03,172
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.consumer.PartitionTopicInfo-52: initial consumer offset of Test:1:
fetched offset = -1: consumed offset = -1 is -1
2014-11-05 23:39:03,172
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.consumer.PartitionTopicInfo-52: initial fetch offset of Test:1:
fetched offset = -1: consumed offset = -1 is -1
2014-11-05 23:39:03,172
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.consumer.ZookeeperConsumerConnector-52:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Test:1: fetched offset
= -1: consumed offset = -1 selected new offset -1
2014-11-05 23:39:03,175 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:03,175
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 29,1 replyHeader:: 29,38,-101 request::
'/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
response::
2014-11-05 23:39:03,179
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 30,1 replyHeader:: 30,39,-101 request::
'/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0
response::
2014-11-05 23:39:03,180 pool-13-thread-1 DEBUG
kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
topic Test
2014-11-05 23:39:03,181 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 1 for 1 topic(s)
Set(Test)
2014-11-05 23:39:03,182 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:03,182 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:03,186
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 31,1 replyHeader:: 31,40,0 request::
'/consumers/TestCheck/owners,,v{s{31,s{'world,'anyone}}},0 response::
'/consumers/TestCheck/owners
2014-11-05 23:39:03,189 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:03,190
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 32,1 replyHeader:: 32,41,0 request::
'/consumers/TestCheck/owners/Test,,v{s{31,s{'world,'anyone}}},0 response::
'/consumers/TestCheck/owners/Test
2014-11-05 23:39:03,190 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:03,191 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:03,192 pool-13-thread-1 ERROR
kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
topic, partition due to: Failed to fetch topic metadata for topic: Test
2014-11-05 23:39:03,193
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 33,1 replyHeader:: 33,42,0 request::
'/consumers/TestCheck/owners/Test/1,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
response:: '/consumers/TestCheck/owners/Test/1
2014-11-05 23:39:03,193 pool-13-thread-1 INFO
kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
retrying send. Remaining retries = 10
2014-11-05 23:39:03,193
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
partition 1 for topic Test
2014-11-05 23:39:03,196
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 34,1 replyHeader:: 34,43,0 request::
'/consumers/TestCheck/owners/Test/0,#6865616c7468636865636b5f69702d3137322d33312d32352d3139382d313431353233303734323833302d66336466633336322d30,v{s{31,s{'world,'anyone}}},1
response:: '/consumers/TestCheck/owners/Test/0
2014-11-05 23:39:03,196
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362],
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0 successfully owned
partition 0 for topic Test
2014-11-05 23:39:03,196
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Updating the cache
2014-11-05 23:39:03,197
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.consumer.ZookeeperConsumerConnector-52:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Partitions per topic
cache Map(Test -> ArrayBuffer(0, 1))
2014-11-05 23:39:03,197
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor DEBUG
kafka.consumer.ZookeeperConsumerConnector-52:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumers per topic
cache Map(Test -> List(TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-0))
2014-11-05 23:39:03,197
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], Consumer
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 selected partitions :
Test:0: fetched offset = -1: consumed offset = -1,Test:1: fetched offset =
-1: consumed offset = -1
2014-11-05 23:39:03,199
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
INFO kafka.consumer.ConsumerFetcherManager$LeaderFinderThread-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread],
Starting
2014-11-05 23:39:03,199
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362_watcher_executor INFO
kafka.consumer.ZookeeperConsumerConnector-68:
[TestCheck_ip-172-31-25-198-1415230742830-f3dfc362], end rebalancing
consumer TestCheck_ip-172-31-25-198-1415230742830-f3dfc362 try #0
2014-11-05 23:39:03,202
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 35,8 replyHeader:: 35,43,0 request::
'/brokers/ids,F response:: v{'0}
2014-11-05 23:39:03,204
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-839: Reading reply
sessionid:0x1498251e8680003, packet:: clientPath:null serverPath:null
finished:false header:: 36,4 replyHeader:: 36,43,0 request::
'/brokers/ids/0,F response::
#7b226a6d785f706f7274223a2d312c2274696d657374616d70223a2231343135323330363632383839222c22686f7374223a2235342e3231392e3134332e313639222c2276657273696f6e223a312c22706f7274223a393039327d,s{20,20,1415230662902,1415230662902,0,0,0,92748555640897537,91,0,20}
2014-11-05 23:39:03,214
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
INFO kafka.utils.VerifiableProperties-68: Verifying properties
2014-11-05 23:39:03,215
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
INFO kafka.utils.VerifiableProperties-68: Property client.id is overridden
to TestCheck
2014-11-05 23:39:03,215
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
INFO kafka.utils.VerifiableProperties-68: Property metadata.broker.list is
overridden to x.x.x.x:9092
2014-11-05 23:39:03,215
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
INFO kafka.utils.VerifiableProperties-68: Property request.timeout.ms is
overridden to 30000
2014-11-05 23:39:03,217
TestCheck_ip-172-31-25-198-1415230742830-f3dfc362-leader-finder-thread
INFO kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:x.x.x.x,port:9092 with correlation id 0 for 1 topic(s) Set(Test)
2014-11-05 23:39:04,195 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 2 for 1 topic(s)
Set(Test)
2014-11-05 23:39:04,196 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:04,196 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:04,199 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:04,199 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:04,200 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:04,200 pool-13-thread-1 DEBUG
kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
topic Test
2014-11-05 23:39:04,201 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 3 for 1 topic(s)
Set(Test)
2014-11-05 23:39:04,201 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:04,201 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:04,205 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:04,205 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:04,206 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:04,206 pool-13-thread-1 ERROR
kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
topic, partition due to: Failed to fetch topic metadata for topic: Test
2014-11-05 23:39:04,206 pool-13-thread-1 INFO
kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
retrying send. Remaining retries = 9
2014-11-05 23:39:04,698
Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
0x1498251e8680002 after 0ms
2014-11-05 23:39:05,205
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
0x1498251e8680003 after 0ms
2014-11-05 23:39:05,207 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 4 for 1 topic(s)
Set(Test)
2014-11-05 23:39:05,208 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:05,208 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:05,211 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:05,211 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:05,212 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:05,212 pool-13-thread-1 DEBUG
kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
topic Test
2014-11-05 23:39:05,212 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 5 for 1 topic(s)
Set(Test)
2014-11-05 23:39:05,213 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:05,213 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:05,217 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:05,217 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:05,217 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:05,217 pool-13-thread-1 ERROR
kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
topic, partition due to: Failed to fetch topic metadata for topic: Test
2014-11-05 23:39:05,218 pool-13-thread-1 INFO
kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
retrying send. Remaining retries = 8
2014-11-05 23:39:06,218 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 6 for 1 topic(s)
Set(Test)
2014-11-05 23:39:06,219 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:06,219 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:06,224 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:06,224 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:06,225 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:06,225 pool-13-thread-1 DEBUG
kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
topic Test
2014-11-05 23:39:06,226 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 7 for 1 topic(s)
Set(Test)
2014-11-05 23:39:06,226 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:06,226 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:06,230 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:06,230 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:06,231 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:06,231 pool-13-thread-1 ERROR
kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
topic, partition due to: Failed to fetch topic metadata for topic: Test
2014-11-05 23:39:06,231 pool-13-thread-1 INFO
kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
retrying send. Remaining retries = 7
2014-11-05 23:39:06,699
Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
0x1498251e8680002 after 0ms
2014-11-05 23:39:07,208
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
0x1498251e8680003 after 0ms
2014-11-05 23:39:07,232 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 8 for 1 topic(s)
Set(Test)
2014-11-05 23:39:07,233 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:07,233 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:07,236 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:07,236 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:07,237 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:07,237 pool-13-thread-1 DEBUG
kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
topic Test
2014-11-05 23:39:07,238 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 9 for 1 topic(s)
Set(Test)
2014-11-05 23:39:07,238 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:07,238 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:07,241 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:07,241 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:07,242 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:07,242 pool-13-thread-1 ERROR
kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
topic, partition due to: Failed to fetch topic metadata for topic: Test
2014-11-05 23:39:07,242 pool-13-thread-1 INFO
kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
retrying send. Remaining retries = 6
2014-11-05 23:39:08,243 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 10 for 1 topic(s)
Set(Test)
2014-11-05 23:39:08,243 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:08,245 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:08,249 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:08,250 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:08,250 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
topic Test
2014-11-05 23:39:08,251 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 11 for 1 topic(s)
Set(Test)
2014-11-05 23:39:08,251 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:08,252 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:08,255 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:08,256 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:08,256 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:08,257 pool-13-thread-1 ERROR
kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
topic, partition due to: Failed to fetch topic metadata for topic: Test
2014-11-05 23:39:08,257 pool-13-thread-1 INFO
kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
retrying send. Remaining retries = 5
2014-11-05 23:39:08,702
Thread-3-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
0x1498251e8680002 after 0ms
2014-11-05 23:39:09,211
pool-13-thread-2-SendThread(ip-172-31-25-198.us-west-1.compute.internal:2181)
DEBUG org.apache.zookeeper.ClientCnxn-759: Got ping response for sessionid:
0x1498251e8680003 after 0ms
2014-11-05 23:39:09,257 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 12 for 1 topic(s)
Set(Test)
2014-11-05 23:39:09,258 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:09,258 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:09,261 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:09,262 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:09,262 pool-13-thread-1 DEBUG
kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
topic Test
2014-11-05 23:39:09,263 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 13 for 1 topic(s)
Set(Test)
2014-11-05 23:39:09,264 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:09,264 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:09,268 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:09,268 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:09,269 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:09,269 pool-13-thread-1 ERROR
kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
topic, partition due to: Failed to fetch topic metadata for topic: Test
2014-11-05 23:39:09,269 pool-13-thread-1 INFO
kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
retrying send. Remaining retries = 4
2014-11-05 23:39:10,270 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 14 for 1 topic(s)
Set(Test)
2014-11-05 23:39:10,271 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:10,271 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:10,275 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:10,275 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:10,275 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:10,276 pool-13-thread-1 DEBUG
kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for
topic Test
2014-11-05 23:39:10,276 pool-13-thread-1 INFO
kafka.client.ClientUtils$-68: Fetching metadata from broker
id:0,host:172.31.25.198,port:9092 with correlation id 15 for 1 topic(s)
Set(Test)
2014-11-05 23:39:10,277 pool-13-thread-1 DEBUG
kafka.network.BlockingChannel-52: Created socket with SO_TIMEOUT = 10000
(requested 10000), SO_RCVBUF = 530904 (requested -1), SO_SNDBUF = 102400
(requested 102400).
2014-11-05 23:39:10,277 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.25.198:9092 for
producing
2014-11-05 23:39:10,280 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.25.198:9092
2014-11-05 23:39:10,280 pool-13-thread-1 DEBUG
kafka.client.ClientUtils$-52: Successfully fetched metadata for 1 topic(s)
Set(Test)
2014-11-05 23:39:10,280 pool-13-thread-1 WARN
kafka.producer.BrokerPartitionInfo-83: Error while fetching metadata
[{TopicMetadata for topic Test ->
No partition metadata for topic Test due to
kafka.common.LeaderNotAvailableException}] for topic [Test]: class
kafka.common.LeaderNotAvailableException
2014-11-05 23:39:10,280 pool-13-thread-1 ERROR
kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by
topic, partition due to: Failed to fetch topic metadata for topic: Test
2014-11-05 23:39:10,280 pool-13-thread-1 INFO
kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms before
retrying send. Remaining retries = 3
Brokers Logs -
[2014-11-05 23:39:02,983] INFO Topic creation
{"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
[2014-11-05 23:39:02,988] INFO [KafkaApi-0] Auto creation of topic Test
with 2 partitions and replication factor 1 is successful!
(kafka.server.KafkaApis)
[2014-11-05 23:39:03,092] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-05 23:39:03,191] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-05 23:39:04,200] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-05 23:39:04,205] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-05 23:39:05,211] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-05 23:39:05,218] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-05 23:39:06,224] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-05 23:39:06,230] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-05 23:39:07,236] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
[2014-11-05 23:39:07,241] INFO Closing socket connection to /172.31.25.198.
(kafka.network.Processor)
On Wed, Nov 5, 2014 at 3:05 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Sameer,
>
> Yes, this is the server log. But there seems no abnormal entries in it, and
> it does not cover the same time range as the producer client throwing
> LeaderNotAvailableException
> (it was 10/24, 14:30).
>
> The reason that I want to check the server log at that same reason is
> that LeaderNotAvailableException
> is set by the brokers while handling requests, and its server log will show
> what is the cause it set this error code.
>
> Guozhang
>
> On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net> wrote:
>
> > Hi Guozhang,
> >
> > This is the server.log -
> >
> > [2014-11-04 20:21:57,510] INFO Verifying properties
> > (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,545] INFO Property advertised.host.name is
> overridden
> > to x.x.x.x (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,545] INFO Property broker.id is overridden to 0
> > (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,545] INFO Property controlled.shutdown.enable is
> > overridden to true (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,545] INFO Property host.name is overridden to
> > 172.31.25.198 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is overridden
> to
> > false (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden to
> > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,546] INFO Property log.retention.check.interval.ms
> is
> > overridden to 60000 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,546] INFO Property log.retention.hours is overridden
> > to 168 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is overridden
> to
> > 536870912 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,546] INFO Property num.io.threads is overridden to 8
> > (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,547] INFO Property num.network.threads is overridden
> > to 2 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,547] INFO Property num.partitions is overridden to 2
> > (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,547] INFO Property port is overridden to 9092
> > (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,547] INFO Property socket.receive.buffer.bytes is
> > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,547] INFO Property socket.request.max.bytes is
> > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,547] INFO Property socket.send.buffer.bytes is
> > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is overridden
> to
> > localhost:2181 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,548] INFO Property zookeeper.connection.timeout.ms
> is
> > overridden to 1000000 (kafka.utils.VerifiableProperties)
> > [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
> > (kafka.server.KafkaServer)
> > [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to zookeeper
> on
> > localhost:2181 (kafka.server.KafkaServer)
> > [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs' not found,
> > creating it. (kafka.log.LogManager)
> > [2014-11-04 20:21:57,748] INFO Starting log cleanup with a period of
> 60000
> > ms. (kafka.log.LogManager)
> > [2014-11-04 20:21:57,752] INFO Starting log flusher with a default period
> > of 9223372036854775807 ms. (kafka.log.LogManager)
> > [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
> > ip-172-31-25-198.us-west-1.compute.internal:9092.
> (kafka.network.Acceptor)
> > [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0], Started
> > (kafka.network.SocketServer)
> > [2014-11-04 20:21:57,852] INFO Will not load MX4J, mx4j-tools.jar is not
> in
> > the classpath (kafka.utils.Mx4jLoader$)
> > [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
> > (kafka.server.ZookeeperLeaderElector)
> > [2014-11-04 20:21:57,967] INFO Registered broker 0 at path /brokers/ids/0
> > with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
> > [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
> > (kafka.server.KafkaServer)
> > [2014-11-04 20:21:58,050] INFO New leader is 0
> > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > [2014-11-04 20:22:01,809] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:22:11,812] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:29,084] INFO Topic creation
> > {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
> > [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of topic Test
> > with 2 partitions and replication factor 1 is successful!
> > (kafka.server.KafkaApis)
> > [2014-11-04 20:30:29,280] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:29,301] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:30,310] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:30,315] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:31,320] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:31,325] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:32,331] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:32,336] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:33,343] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:33,347] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:34,356] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:34,360] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:35,366] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:35,372] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:36,377] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:36,382] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:37,387] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:37,394] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:38,399] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:38,403] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:39,410] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:39,414] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:40,419] INFO Closing socket connection to /
> 172.31.25.198
> > .
> > (kafka.network.Processor)
> > [2014-11-04 20:30:41,602] INFO Topic creation
> > {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
> >
> >
> >
> > thanks
> >
> >
> > On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > The server.log should be the broker logs.
> > >
> > > Guozhang
> > >
> > > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <sy...@ailive.net>
> wrote:
> > >
> > > > I do not see any other logs besides these.
> > > > I checked the log4j.properties and where the logs are getting stored.
> > > > The only other logs (apart from the controller logs) in the directory
> > are
> > > > zookeeper-gc.log,kafkaServer-gc.log, server.log and state-change.log
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > > > Sorry for getting late on this Sameer.
> > > > >
> > > > > The logs you showed above are from controller logs, do you see
> > brokers
> > > > logs
> > > > > besides that?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <sy...@ailive.net>
> > wrote:
> > > > >
> > > > > > Hi Guozhang,
> > > > > >
> > > > > > Is there any clue in the above data as to what is wrong?
> > > > > >
> > > > > > thanks
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <sy...@ailive.net>
> > > wrote:
> > > > > >
> > > > > > > This is the one from the file specified in log4j.properties -
> > > > > > >
> > > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on
> > Controller
> > > > 0]:
> > > > > > > Topic change listener fired for path /brokers/topics with
> > children
> > > > Test
> > > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener on
> Controller
> > > 0]:
> > > > > New
> > > > > > > topics: [Set(Test)], deleted topics: [Set()], new partition
> > replica
> > > > > > > assignment [Map([Test,0] -> List(0), [Test,1] -> List(0))]
> > > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New topic
> creation
> > > > > > callback
> > > > > > > for [Test,0],[Test,1] (kafka.controller.KafkaController)
> > > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New partition
> > > creation
> > > > > > > callback for [Test,0],[Test,1]
> (kafka.controller.KafkaController)
> > > > > > > [2014-10-28 22:38:36,095] INFO [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > > Invoking state change to NewPartition for partitions
> > > > [Test,0],[Test,1]
> > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine on
> > controller
> > > > 0]:
> > > > > > > Invoking state change to NewReplica for replicas
> > > > > > >
> > > [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > > [2014-10-28 22:38:36,111] INFO [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > > Invoking state change to OnlinePartition for partitions
> > > > > [Test,0],[Test,1]
> > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state machine on
> > > > Controller
> > > > > > 0]:
> > > > > > > Live assigned replicas for partition [Test,0] are: [List(0)]
> > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state machine on
> > > > Controller
> > > > > > 0]:
> > > > > > > Initializing leader and isr for partition [Test,0] to
> > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state machine on
> > > > Controller
> > > > > > 0]:
> > > > > > > Live assigned replicas for partition [Test,1] are: [List(0)]
> > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state machine on
> > > > Controller
> > > > > > 0]:
> > > > > > > Initializing leader and isr for partition [Test,1] to
> > > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine on
> > controller
> > > > 0]:
> > > > > > > Invoking state change to OnlineReplica for replicas
> > > > > > >
> > > [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > > [2014-10-28 22:39:28,160] ERROR
> > > > [Controller-0-to-broker-0-send-thread],
> > > > > > > Controller 0's connection to broker id:0,host:x.x.x.x,port:9092
> > was
> > > > > > > unsuccessful (kafka.controller.RequestSendThread)
> > > > > > > java.net.ConnectException: Connection timed out
> > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > > at sun.nio.ch.Net.connect(Net.java:449)
> > > > > > > at
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> > > > > > > at
> > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > > at
> > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > > [2014-10-28 22:39:28,460] ERROR
> > > > [Controller-0-to-broker-0-send-thread],
> > > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr request with
> > > > > correlation
> > > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092. Reconnecting to
> > broker.
> > > > > > > (kafka.controller.RequestSendThread)
> > > > > > > java.nio.channels.ClosedChannelException
> > > > > > > at
> > kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > > at
> > > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > >
> > > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
> > wangguoz@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> Sameer, the one you sent me is from the producer logs, not the
> > > > broker
> > > > > > >> logs.
> > > > > > >> And /tmp/kafka-logs actually store the message log (i.e. the
> > > "data")
> > > > > > >> instead of the server logs. Their locations should be
> specified
> > in
> > > > > your
> > > > > > >> log4j.properties file.
> > > > > > >>
> > > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <
> syami@ailive.net>
> > > > > wrote:
> > > > > > >>
> > > > > > >> > Is this what you are looking for -
> > > > > > >> >
> > > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > 172.31.16.31:9092
> > > > for
> > > > > > >> > producing
> > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > 172.31.16.31:9092
> > > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
> > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> > for
> > > 1
> > > > > > >> topic(s)
> > > > > > >> > Set(Test)
> > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
> > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > > metadata
> > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > >> > No partition metadata for topic Test due to
> > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > > class
> > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
> > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> > partition
> > > > info
> > > > > > for
> > > > > > >> > topic Test
> > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
> > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 9 for 1
> > > > > topic(s)
> > > > > > >> > Set(Test)
> > > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
> > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > > SO_TIMEOUT =
> > > > > > 10000
> > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> > SO_SNDBUF
> > > =
> > > > > > 102400
> > > > > > >> > (requested 102400).
> > > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > 172.31.16.31:9092
> > > > for
> > > > > > >> > producing
> > > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > 172.31.16.31:9092
> > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
> > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> > for
> > > 1
> > > > > > >> topic(s)
> > > > > > >> > Set(Test)
> > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
> > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > > metadata
> > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > >> > No partition metadata for topic Test due to
> > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > > class
> > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
> > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
> collate
> > > > > > messages
> > > > > > >> by
> > > > > > >> > topic, partition due to: Failed to fetch topic metadata for
> > > topic:
> > > > > > Test
> > > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for
> 1000
> > > ms
> > > > > > before
> > > > > > >> > retrying send. Remaining retries = 6
> > > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
> > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 10 for
> 1
> > > > > topic(s)
> > > > > > >> > Set(Test)
> > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
> > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > > SO_TIMEOUT =
> > > > > > 10000
> > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> > SO_SNDBUF
> > > =
> > > > > > 102400
> > > > > > >> > (requested 102400).
> > > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > 172.31.16.31:9092
> > > > for
> > > > > > >> > producing
> > > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > 172.31.16.31:9092
> > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> > for
> > > 1
> > > > > > >> topic(s)
> > > > > > >> > Set(Test)
> > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
> > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > > metadata
> > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > >> > No partition metadata for topic Test due to
> > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > > class
> > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> > partition
> > > > info
> > > > > > for
> > > > > > >> > topic Test
> > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
> > > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 11 for
> 1
> > > > > topic(s)
> > > > > > >> > Set(Test)
> > > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
> > > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > > SO_TIMEOUT =
> > > > > > 10000
> > > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> > SO_SNDBUF
> > > =
> > > > > > 102400
> > > > > > >> > (requested 102400).
> > > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.SyncProducer-68: Connected to
> > 172.31.16.31:9092
> > > > for
> > > > > > >> > producing
> > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > > 172.31.16.31:9092
> > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
> > > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> > for
> > > 1
> > > > > > >> topic(s)
> > > > > > >> > Set(Test)
> > > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
> > > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > > metadata
> > > > > > >> > [{TopicMetadata for topic Test ->
> > > > > > >> > No partition metadata for topic Test due to
> > > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > > class
> > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
> > > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to
> collate
> > > > > > messages
> > > > > > >> by
> > > > > > >> > topic, partition due to: Failed to fetch topic metadata for
> > > topic:
> > > > > > Test
> > > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
> > > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for
> 1000
> > > ms
> > > > > > before
> > > > > > >> > retrying send. Remaining retries = 5
> > > > > > >> >
> > > > > > >> > As for /tmp/kafka-logs, I can see only these files -
> > > > > > >> > recovery-point-offset-checkpoint
> > replication-offset-checkpoint
> > > > > > >> >
> > > > > > >> > I do not see the usual Kafka log files here.
> > > > > > >> >
> > > > > > >> > thanks
> > > > > > >> > Sameer
> > > > > > >> >
> > > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
> > > > wangguoz@gmail.com>
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > Sameer, could you check broker logs and see if it has
> > similar
> > > > > > >> > > LeaderNotAvailable exceptions?
> > > > > > >> > >
> > > > > > >> > > Guozhang
> > > > > > >> > >
> > > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
> > > syami@ailive.net
> > > > >
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > > Yes. I can telnet from outside using the public ip of
> the
> > > > > instance
> > > > > > >> and
> > > > > > >> > > port
> > > > > > >> > > > of 9092
> > > > > > >> > > >
> > > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <
> > > > > > >> roger.hoover@gmail.com
> > > > > > >> > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Just a guess but could it be a firewall issue? Did
> you
> > > > enable
> > > > > > >> > > > connections
> > > > > > >> > > > > to port 9092 from outside EC2 in a security group?
> Can
> > > you
> > > > > > >> telnet to
> > > > > > >> > > > each
> > > > > > >> > > > > broker IP and port?
> > > > > > >> > > > >
> > > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami <
> > > > > syami@ailive.net
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > There was a typo earlier.
> > > > > > >> > > > > >
> > > > > > >> > > > > > This is the output -
> > > > > > >> > > > > >
> > > > > > >> > > > > > Topic:Test PartitionCount:2 ReplicationFactor:1
> > > > > Configs:
> > > > > > >> > > > > > Topic: Test Partition: 0 Leader: 0
> > > > > Replicas:
> > > > > > 0
> > > > > > >> > > > Isr: 0
> > > > > > >> > > > > > Topic: Test Partition: 1 Leader: 0
> > > > > Replicas:
> > > > > > 0
> > > > > > >> > > > Isr: 0
> > > > > > >> > > > > >
> > > > > > >> > > > > > thanks
> > > > > > >> > > > > > Sameer
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang Wang <
> > > > > > >> wangguoz@gmail.com
> > > > > > >> > >
> > > > > > >> > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Sameer,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > The topic name is not consistent from your
> describe
> > > > > outputs.
> > > > > > >> > > Normally
> > > > > > >> > > > > > they
> > > > > > >> > > > > > > should be either
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Topic:*Test* PartitionCount:2
> > > ReplicationFactor:1
> > > > > > >> > > Configs:
> > > > > > >> > > > > > > Topic: *Test* Partition: 0 Leader: 0
> > > > > > >> Replicas: 0
> > > > > > >> > > > > > Isr: 0
> > > > > > >> > > > > > > Topic: *Test* Partition: 1 Leader: 0
> > > > > > >> Replicas: 0
> > > > > > >> > > > > > Isr: 0
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Or
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
> > > > > > ReplicationFactor:1
> > > > > > >> > > > > Configs:
> > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 0
> > Leader:
> > > 0
> > > > > > >> > > Replicas:
> > > > > > >> > > > 0
> > > > > > >> > > > > > > Isr: 0
> > > > > > >> > > > > > > Topic: *Heartbeat* Partition: 1
> > Leader:
> > > 0
> > > > > > >> > > Replicas:
> > > > > > >> > > > 0
> > > > > > >> > > > > > > Isr: 0
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Did you manually modify the topic name in ZK?
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Guozhang
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer Yami <
> > > > > > >> syami@ailive.net>
> > > > > > >> > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi Guozhang,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > I was wondering if there is anything wrong in
> the
> > > > above
> > > > > > >> > > --describe
> > > > > > >> > > > > > data.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > thanks
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer Yami <
> > > > > > >> syami@ailive.net
> > > > > > >> > >
> > > > > > >> > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > This is what I get -
> > > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper
> localhost:2181
> > > > > > --describe
> > > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
> > > > > > >> ReplicationFactor:1
> > > > > > >> > > > > > Configs:
> > > > > > >> > > > > > > > > Topic: Test Partition: 0
> Leader: 0
> > > > > > >> > Replicas: 0
> > > > > > >> > > > > > > Isr: 0
> > > > > > >> > > > > > > > > Topic: Test Partition: 1
> Leader: 0
> > > > > > >> > Replicas: 0
> > > > > > >> > > > > > > Isr: 0
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > The topics are being created at run-time.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > thanks
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM, Guozhang
> Wang <
> > > > > > >> > > > wangguoz@gmail.com
> > > > > > >> > > > > >
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >> Could you use the topic tools "--describe" to
> > > check
> > > > > if
> > > > > > >> the
> > > > > > >> > > topic
> > > > > > >> > > > > > > Test's
> > > > > > >> > > > > > > > >> partitions leaders are available?
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >>
> > > > > > >> > >
> > > http://kafka.apache.org/documentation.html#basic_ops_add_topic
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >> Guozhang
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM, Sameer Yami
> <
> > > > > > >> > > syami@ailive.net>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >> > That is not the case.
> > > > > > >> > > > > > > > >> > This detailed error is below. Due to this
> > > error ,
> > > > > the
> > > > > > >> > > producer
> > > > > > >> > > > > is
> > > > > > >> > > > > > > not
> > > > > > >> > > > > > > > >> able
> > > > > > >> > > > > > > > >> > to send messages
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG
> > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-52:
> > > > > > >> > > > > > > > >> > Getting broker partition info for topic
> Test
> > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main INFO
> > > > > > >> > > > > kafka.client.ClientUtils$-68:
> > > > > > >> > > > > > > > >> Fetching
> > > > > > >> > > > > > > > >> > metadata from broker
> > > > > > id:0,host:54.215.200.61,port:9092
> > > > > > >> > with
> > > > > > >> > > > > > > > correlation
> > > > > > >> > > > > > > > >> id
> > > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
> > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG
> > > > > > >> > > > > > > kafka.network.BlockingChannel-52:
> > > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT = 10000
> > > (requested
> > > > > > >> 10000),
> > > > > > >> > > > > > SO_RCVBUF
> > > > > > >> > > > > > > =
> > > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF = 102400
> > > > > (requested
> > > > > > >> > > 102400).
> > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main INFO
> > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092 for
> > producing
> > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main INFO
> > > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > > >> > > > > > > > >> > Disconnecting from 54.215.200.61:9092
> > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG
> > > > > > >> > > > > kafka.client.ClientUtils$-52:
> > > > > > >> > > > > > > > >> > Successfully fetched metadata for 1
> topic(s)
> > > > > > Set(Test)
> > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main WARN
> > > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-83:
> > > > > > >> > > > > > > > >> > Error while fetching metadata
> [{TopicMetadata
> > > for
> > > > > > topic
> > > > > > >> > Test
> > > > > > >> > > > ->
> > > > > > >> > > > > > > > >> > No partition metadata for topic Test due to
> > > > > > >> > > > > > > > >> > kafka.common.LeaderNotAvailableException}]
> > for
> > > > > topic
> > > > > > >> > [Test]:
> > > > > > >> > > > > class
> > > > > > >> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR
> > > > > > >> > > > > > > > >> >
> kafka.producer.async.DefaultEventHandler-97:
> > > > Failed
> > > > > > to
> > > > > > >> > > collate
> > > > > > >> > > > > > > > messages
> > > > > > >> > > > > > > > >> by
> > > > > > >> > > > > > > > >> > topic, partition due to: Failed to fetch
> > topic
> > > > > > metadata
> > > > > > >> > for
> > > > > > >> > > > > topic:
> > > > > > >> > > > > > > > Test
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM, Guozhang
> > Wang
> > > <
> > > > > > >> > > > > > wangguoz@gmail.com>
> > > > > > >> > > > > > > > >> wrote:
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > > Could you see if this is the case:
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> > > > > > >> > > > > > > > >> > > ?
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM, Sameer
> > > Yami <
> > > > > > >> > > > > syami@ailive.net
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > >> wrote:
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > > > Hi,
> > > > > > >> > > > > > > > >> > > >
> > > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper test setup
> > on a
> > > > > > single
> > > > > > >> EC2
> > > > > > >> > > > > > machine.
> > > > > > >> > > > > > > > The
> > > > > > >> > > > > > > > >> > > > host.name is the private ip address
> and
> > > the
> > > > > > >> > > > > > > advertised.host.name
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > >> > the
> > > > > > >> > > > > > > > >> > > > public address in server.properties.
> > > > > > >> > > > > > > > >> > > >
> > > > > > >> > > > > > > > >> > > > We are trying to publish from outside
> EC2
> > > > using
> > > > > > the
> > > > > > >> > > public
> > > > > > >> > > > > ip
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > >> > EC2
> > > > > > >> > > > > > > > >> > > > machine but get this error on the
> client
> > > > side -
> > > > > > >> > > > > > > > >> > > > No partition metadata for topic test
> due
> > to
> > > > > > >> > > > > > > > >> > > >
> > kafka.common.LeaderNotAvailableException}]
> > > > for
> > > > > > >> topic
> > > > > > >> > > > [test]:
> > > > > > >> > > > > > > class
> > > > > > >> > > > > > > > >> > > >
> kafka.common.LeaderNotAvailableException
> > > > > > >> > > > > > > > >> > > >
> > > > > > >> > > > > > > > >> > > > The zookeeper has the public ip stored
> in
> > > > > > >> > > /brokers/ids/0.
> > > > > > >> > > > > > > > >> > > > We have tried multiple combinations but
> > it
> > > > > works
> > > > > > >> ONLY
> > > > > > >> > > when
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > >> > advertised
> > > > > > >> > > > > > > > >> > > > host name is not used and the client is
> > > > > internal
> > > > > > to
> > > > > > >> > EC2.
> > > > > > >> > > > > > > > >> > > >
> > > > > > >> > > > > > > > >> > > > Please let me know what the problem
> could
> > > be.
> > > > > > >> > > > > > > > >> > > >
> > > > > > >> > > > > > > > >> > > > thanks
> > > > > > >> > > > > > > > >> > > > Sameer
> > > > > > >> > > > > > > > >> > > >
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > > --
> > > > > > >> > > > > > > > >> > > -- Guozhang
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >> --
> > > > > > >> > > > > > > > >> -- Guozhang
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > --
> > > > > > >> > > > > > > -- Guozhang
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > --
> > > > > > >> > > -- Guozhang
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> -- Guozhang
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Cannot connect to Kafka from outside of EC2
Posted by Guozhang Wang <wa...@gmail.com>.
Sameer,
Yes, this is the server log. But there seems no abnormal entries in it, and
it does not cover the same time range as the producer client throwing
LeaderNotAvailableException
(it was 10/24, 14:30).
The reason that I want to check the server log at that same reason is
that LeaderNotAvailableException
is set by the brokers while handling requests, and its server log will show
what is the cause it set this error code.
Guozhang
On Tue, Nov 4, 2014 at 12:47 PM, Sameer Yami <sy...@ailive.net> wrote:
> Hi Guozhang,
>
> This is the server.log -
>
> [2014-11-04 20:21:57,510] INFO Verifying properties
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property advertised.host.name is overridden
> to x.x.x.x (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property broker.id is overridden to 0
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property controlled.shutdown.enable is
> overridden to true (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property host.name is overridden to
> 172.31.25.198 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,545] INFO Property log.cleaner.enable is overridden to
> false (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property log.dirs is overridden to
> /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property log.retention.check.interval.ms is
> overridden to 60000 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property log.retention.hours is overridden
> to 168 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property log.segment.bytes is overridden to
> 536870912 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,546] INFO Property num.io.threads is overridden to 8
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property num.network.threads is overridden
> to 2 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property num.partitions is overridden to 2
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property port is overridden to 9092
> (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property socket.receive.buffer.bytes is
> overridden to 1048576 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property socket.request.max.bytes is
> overridden to 104857600 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,547] INFO Property socket.send.buffer.bytes is
> overridden to 1048576 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,548] INFO Property zookeeper.connect is overridden to
> localhost:2181 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,548] INFO Property zookeeper.connection.timeout.ms is
> overridden to 1000000 (kafka.utils.VerifiableProperties)
> [2014-11-04 20:21:57,563] INFO [Kafka Server 0], starting
> (kafka.server.KafkaServer)
> [2014-11-04 20:21:57,565] INFO [Kafka Server 0], Connecting to zookeeper on
> localhost:2181 (kafka.server.KafkaServer)
> [2014-11-04 20:21:57,738] INFO Log directory '/tmp/kafka-logs' not found,
> creating it. (kafka.log.LogManager)
> [2014-11-04 20:21:57,748] INFO Starting log cleanup with a period of 60000
> ms. (kafka.log.LogManager)
> [2014-11-04 20:21:57,752] INFO Starting log flusher with a default period
> of 9223372036854775807 ms. (kafka.log.LogManager)
> [2014-11-04 20:21:57,783] INFO Awaiting socket connections on
> ip-172-31-25-198.us-west-1.compute.internal:9092. (kafka.network.Acceptor)
> [2014-11-04 20:21:57,784] INFO [Socket Server on Broker 0], Started
> (kafka.network.SocketServer)
> [2014-11-04 20:21:57,852] INFO Will not load MX4J, mx4j-tools.jar is not in
> the classpath (kafka.utils.Mx4jLoader$)
> [2014-11-04 20:21:57,884] INFO 0 successfully elected as leader
> (kafka.server.ZookeeperLeaderElector)
> [2014-11-04 20:21:57,967] INFO Registered broker 0 at path /brokers/ids/0
> with address x.x.x.x:9092. (kafka.utils.ZkUtils$)
> [2014-11-04 20:21:57,981] INFO [Kafka Server 0], started
> (kafka.server.KafkaServer)
> [2014-11-04 20:21:58,050] INFO New leader is 0
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2014-11-04 20:22:01,809] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:22:11,812] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:29,084] INFO Topic creation
> {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
> [2014-11-04 20:30:29,088] INFO [KafkaApi-0] Auto creation of topic Test
> with 2 partitions and replication factor 1 is successful!
> (kafka.server.KafkaApis)
> [2014-11-04 20:30:29,280] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:29,301] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:30,310] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:30,315] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:31,320] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:31,325] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:32,331] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:32,336] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:33,343] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:33,347] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:34,356] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:34,360] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:35,366] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:35,372] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:36,377] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:36,382] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:37,387] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:37,394] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:38,399] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:38,403] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:39,410] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:39,414] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:40,419] INFO Closing socket connection to /172.31.25.198
> .
> (kafka.network.Processor)
> [2014-11-04 20:30:41,602] INFO Topic creation
> {"version":1,"partitions":{"1":[0],"0":[0]}} (kafka.admin.AdminUtils$)
>
>
>
> thanks
>
>
> On Fri, Oct 31, 2014 at 9:15 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > The server.log should be the broker logs.
> >
> > Guozhang
> >
> > On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami <sy...@ailive.net> wrote:
> >
> > > I do not see any other logs besides these.
> > > I checked the log4j.properties and where the logs are getting stored.
> > > The only other logs (apart from the controller logs) in the directory
> are
> > > zookeeper-gc.log,kafkaServer-gc.log, server.log and state-change.log
> > >
> > > Thanks
> > >
> > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Sorry for getting late on this Sameer.
> > > >
> > > > The logs you showed above are from controller logs, do you see
> brokers
> > > logs
> > > > besides that?
> > > >
> > > > Guozhang
> > > >
> > > > On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami <sy...@ailive.net>
> wrote:
> > > >
> > > > > Hi Guozhang,
> > > > >
> > > > > Is there any clue in the above data as to what is wrong?
> > > > >
> > > > > thanks
> > > > >
> > > > >
> > > > > On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami <sy...@ailive.net>
> > wrote:
> > > > >
> > > > > > This is the one from the file specified in log4j.properties -
> > > > > >
> > > > > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on
> Controller
> > > 0]:
> > > > > > Topic change listener fired for path /brokers/topics with
> children
> > > Test
> > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > [2014-10-28 22:38:36,094] INFO [TopicChangeListener on Controller
> > 0]:
> > > > New
> > > > > > topics: [Set(Test)], deleted topics: [Set()], new partition
> replica
> > > > > > assignment [Map([Test,0] -> List(0), [Test,1] -> List(0))]
> > > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener)
> > > > > > [2014-10-28 22:38:36,094] INFO [Controller 0]: New topic creation
> > > > > callback
> > > > > > for [Test,0],[Test,1] (kafka.controller.KafkaController)
> > > > > > [2014-10-28 22:38:36,095] INFO [Controller 0]: New partition
> > creation
> > > > > > callback for [Test,0],[Test,1] (kafka.controller.KafkaController)
> > > > > > [2014-10-28 22:38:36,095] INFO [Partition state machine on
> > Controller
> > > > 0]:
> > > > > > Invoking state change to NewPartition for partitions
> > > [Test,0],[Test,1]
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,109] INFO [Replica state machine on
> controller
> > > 0]:
> > > > > > Invoking state change to NewReplica for replicas
> > > > > >
> > [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > [2014-10-28 22:38:36,111] INFO [Partition state machine on
> > Controller
> > > > 0]:
> > > > > > Invoking state change to OnlinePartition for partitions
> > > > [Test,0],[Test,1]
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,111] DEBUG [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > Live assigned replicas for partition [Test,0] are: [List(0)]
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,112] DEBUG [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > Initializing leader and isr for partition [Test,0] to
> > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,141] DEBUG [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > Live assigned replicas for partition [Test,1] are: [List(0)]
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,142] DEBUG [Partition state machine on
> > > Controller
> > > > > 0]:
> > > > > > Initializing leader and isr for partition [Test,1] to
> > > > > > (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> > > > > > (kafka.controller.PartitionStateMachine)
> > > > > > [2014-10-28 22:38:36,150] INFO [Replica state machine on
> controller
> > > 0]:
> > > > > > Invoking state change to OnlineReplica for replicas
> > > > > >
> > [Topic=Test,Partition=0,Replica=0],[Topic=Test,Partition=1,Replica=0]
> > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > [2014-10-28 22:39:28,160] ERROR
> > > [Controller-0-to-broker-0-send-thread],
> > > > > > Controller 0's connection to broker id:0,host:x.x.x.x,port:9092
> was
> > > > > > unsuccessful (kafka.controller.RequestSendThread)
> > > > > > java.net.ConnectException: Connection timed out
> > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > at sun.nio.ch.Net.connect(Net.java:449)
> > > > > > at
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
> > > > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > at
> > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > > [2014-10-28 22:39:28,460] ERROR
> > > [Controller-0-to-broker-0-send-thread],
> > > > > > Controller 0 epoch 1 failed to send LeaderAndIsr request with
> > > > correlation
> > > > > > id 7 to broker id:0,host:x.x.x.x,port:9092. Reconnecting to
> broker.
> > > > > > (kafka.controller.RequestSendThread)
> > > > > > java.nio.channels.ClosedChannelException
> > > > > > at
> kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
> > > > > > at
> > > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > > >
> > > > > > On Tue, Oct 28, 2014 at 3:09 PM, Guozhang Wang <
> wangguoz@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Sameer, the one you sent me is from the producer logs, not the
> > > broker
> > > > > >> logs.
> > > > > >> And /tmp/kafka-logs actually store the message log (i.e. the
> > "data")
> > > > > >> instead of the server logs. Their locations should be specified
> in
> > > > your
> > > > > >> log4j.properties file.
> > > > > >>
> > > > > >> On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami <sy...@ailive.net>
> > > > wrote:
> > > > > >>
> > > > > >> > Is this what you are looking for -
> > > > > >> >
> > > > > >> > 2014-10-28 21:46:24,146 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Connected to
> 172.31.16.31:9092
> > > for
> > > > > >> > producing
> > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > 172.31.16.31:9092
> > > > > >> > 2014-10-28 21:46:24,149 pool-13-thread-1 DEBUG
> > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> for
> > 1
> > > > > >> topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 WARN
> > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > metadata
> > > > > >> > [{TopicMetadata for topic Test ->
> > > > > >> > No partition metadata for topic Test due to
> > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > class
> > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > 2014-10-28 21:46:24,150 pool-13-thread-1 DEBUG
> > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> partition
> > > info
> > > > > for
> > > > > >> > topic Test
> > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 INFO
> > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 9 for 1
> > > > topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:24,151 pool-13-thread-1 DEBUG
> > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > SO_TIMEOUT =
> > > > > 10000
> > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> SO_SNDBUF
> > =
> > > > > 102400
> > > > > >> > (requested 102400).
> > > > > >> > 2014-10-28 21:46:24,152 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Connected to
> 172.31.16.31:9092
> > > for
> > > > > >> > producing
> > > > > >> > 2014-10-28 21:46:24,154 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > 172.31.16.31:9092
> > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 DEBUG
> > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> for
> > 1
> > > > > >> topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 WARN
> > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > metadata
> > > > > >> > [{TopicMetadata for topic Test ->
> > > > > >> > No partition metadata for topic Test due to
> > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > class
> > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > 2014-10-28 21:46:24,155 pool-13-thread-1 ERROR
> > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > > > > messages
> > > > > >> by
> > > > > >> > topic, partition due to: Failed to fetch topic metadata for
> > topic:
> > > > > Test
> > > > > >> > 2014-10-28 21:46:24,156 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000
> > ms
> > > > > before
> > > > > >> > retrying send. Remaining retries = 6
> > > > > >> > 2014-10-28 21:46:25,156 pool-13-thread-1 INFO
> > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 10 for 1
> > > > topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 DEBUG
> > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > SO_TIMEOUT =
> > > > > 10000
> > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> SO_SNDBUF
> > =
> > > > > 102400
> > > > > >> > (requested 102400).
> > > > > >> > 2014-10-28 21:46:25,157 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Connected to
> 172.31.16.31:9092
> > > for
> > > > > >> > producing
> > > > > >> > 2014-10-28 21:46:25,160 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > 172.31.16.31:9092
> > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> for
> > 1
> > > > > >> topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 WARN
> > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > metadata
> > > > > >> > [{TopicMetadata for topic Test ->
> > > > > >> > No partition metadata for topic Test due to
> > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > class
> > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > 2014-10-28 21:46:25,161 pool-13-thread-1 DEBUG
> > > > > >> > kafka.producer.BrokerPartitionInfo-52: Getting broker
> partition
> > > info
> > > > > for
> > > > > >> > topic Test
> > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 INFO
> > > > > >> > kafka.client.ClientUtils$-68: Fetching metadata from broker
> > > > > >> > id:0,host:172.31.16.31,port:9092 with correlation id 11 for 1
> > > > topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:25,162 pool-13-thread-1 DEBUG
> > > > > >> > kafka.network.BlockingChannel-52: Created socket with
> > SO_TIMEOUT =
> > > > > 10000
> > > > > >> > (requested 10000), SO_RCVBUF = 530904 (requested -1),
> SO_SNDBUF
> > =
> > > > > 102400
> > > > > >> > (requested 102400).
> > > > > >> > 2014-10-28 21:46:25,163 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Connected to
> 172.31.16.31:9092
> > > for
> > > > > >> > producing
> > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.SyncProducer-68: Disconnecting from
> > > > 172.31.16.31:9092
> > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 DEBUG
> > > > > >> > kafka.client.ClientUtils$-52: Successfully fetched metadata
> for
> > 1
> > > > > >> topic(s)
> > > > > >> > Set(Test)
> > > > > >> > 2014-10-28 21:46:25,166 pool-13-thread-1 WARN
> > > > > >> > kafka.producer.BrokerPartitionInfo-83: Error while fetching
> > > metadata
> > > > > >> > [{TopicMetadata for topic Test ->
> > > > > >> > No partition metadata for topic Test due to
> > > > > >> > kafka.common.LeaderNotAvailableException}] for topic [Test]:
> > class
> > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 ERROR
> > > > > >> > kafka.producer.async.DefaultEventHandler-97: Failed to collate
> > > > > messages
> > > > > >> by
> > > > > >> > topic, partition due to: Failed to fetch topic metadata for
> > topic:
> > > > > Test
> > > > > >> > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO
> > > > > >> > kafka.producer.async.DefaultEventHandler-68: Back off for 1000
> > ms
> > > > > before
> > > > > >> > retrying send. Remaining retries = 5
> > > > > >> >
> > > > > >> > As for /tmp/kafka-logs, I can see only these files -
> > > > > >> > recovery-point-offset-checkpoint
> replication-offset-checkpoint
> > > > > >> >
> > > > > >> > I do not see the usual Kafka log files here.
> > > > > >> >
> > > > > >> > thanks
> > > > > >> > Sameer
> > > > > >> >
> > > > > >> > On Tue, Oct 28, 2014 at 1:23 PM, Guozhang Wang <
> > > wangguoz@gmail.com>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Sameer, could you check broker logs and see if it has
> similar
> > > > > >> > > LeaderNotAvailable exceptions?
> > > > > >> > >
> > > > > >> > > Guozhang
> > > > > >> > >
> > > > > >> > > On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami <
> > syami@ailive.net
> > > >
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > > Yes. I can telnet from outside using the public ip of the
> > > > instance
> > > > > >> and
> > > > > >> > > port
> > > > > >> > > > of 9092
> > > > > >> > > >
> > > > > >> > > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover <
> > > > > >> roger.hoover@gmail.com
> > > > > >> > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Just a guess but could it be a firewall issue? Did you
> > > enable
> > > > > >> > > > connections
> > > > > >> > > > > to port 9092 from outside EC2 in a security group? Can
> > you
> > > > > >> telnet to
> > > > > >> > > > each
> > > > > >> > > > > broker IP and port?
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami <
> > > > syami@ailive.net
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > There was a typo earlier.
> > > > > >> > > > > >
> > > > > >> > > > > > This is the output -
> > > > > >> > > > > >
> > > > > >> > > > > > Topic:Test PartitionCount:2 ReplicationFactor:1
> > > > Configs:
> > > > > >> > > > > > Topic: Test Partition: 0 Leader: 0
> > > > Replicas:
> > > > > 0
> > > > > >> > > > Isr: 0
> > > > > >> > > > > > Topic: Test Partition: 1 Leader: 0
> > > > Replicas:
> > > > > 0
> > > > > >> > > > Isr: 0
> > > > > >> > > > > >
> > > > > >> > > > > > thanks
> > > > > >> > > > > > Sameer
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang Wang <
> > > > > >> wangguoz@gmail.com
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Sameer,
> > > > > >> > > > > > >
> > > > > >> > > > > > > The topic name is not consistent from your describe
> > > > outputs.
> > > > > >> > > Normally
> > > > > >> > > > > > they
> > > > > >> > > > > > > should be either
> > > > > >> > > > > > >
> > > > > >> > > > > > > Topic:*Test* PartitionCount:2
> > ReplicationFactor:1
> > > > > >> > > Configs:
> > > > > >> > > > > > > Topic: *Test* Partition: 0 Leader: 0
> > > > > >> Replicas: 0
> > > > > >> > > > > > Isr: 0
> > > > > >> > > > > > > Topic: *Test* Partition: 1 Leader: 0
> > > > > >> Replicas: 0
> > > > > >> > > > > > Isr: 0
> > > > > >> > > > > > >
> > > > > >> > > > > > > Or
> > > > > >> > > > > > >
> > > > > >> > > > > > > Topic:*Heartbeat* PartitionCount:2
> > > > > ReplicationFactor:1
> > > > > >> > > > > Configs:
> > > > > >> > > > > > > Topic: *Heartbeat* Partition: 0
> Leader:
> > 0
> > > > > >> > > Replicas:
> > > > > >> > > > 0
> > > > > >> > > > > > > Isr: 0
> > > > > >> > > > > > > Topic: *Heartbeat* Partition: 1
> Leader:
> > 0
> > > > > >> > > Replicas:
> > > > > >> > > > 0
> > > > > >> > > > > > > Isr: 0
> > > > > >> > > > > > >
> > > > > >> > > > > > > Did you manually modify the topic name in ZK?
> > > > > >> > > > > > >
> > > > > >> > > > > > > Guozhang
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Tue, Oct 28, 2014 at 8:53 AM, Sameer Yami <
> > > > > >> syami@ailive.net>
> > > > > >> > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Guozhang,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I was wondering if there is anything wrong in the
> > > above
> > > > > >> > > --describe
> > > > > >> > > > > > data.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > thanks
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Mon, Oct 27, 2014 at 2:56 PM, Sameer Yami <
> > > > > >> syami@ailive.net
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > This is what I get -
> > > > > >> > > > > > > > > bin/kafka-topics.sh --zookeeper localhost:2181
> > > > > --describe
> > > > > >> > > > > > > > > Topic:Heartbeat PartitionCount:2
> > > > > >> ReplicationFactor:1
> > > > > >> > > > > > Configs:
> > > > > >> > > > > > > > > Topic: Test Partition: 0 Leader: 0
> > > > > >> > Replicas: 0
> > > > > >> > > > > > > Isr: 0
> > > > > >> > > > > > > > > Topic: Test Partition: 1 Leader: 0
> > > > > >> > Replicas: 0
> > > > > >> > > > > > > Isr: 0
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > The topics are being created at run-time.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > thanks
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Sun, Oct 26, 2014 at 4:49 PM, Guozhang Wang <
> > > > > >> > > > wangguoz@gmail.com
> > > > > >> > > > > >
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >> Could you use the topic tools "--describe" to
> > check
> > > > if
> > > > > >> the
> > > > > >> > > topic
> > > > > >> > > > > > > Test's
> > > > > >> > > > > > > > >> partitions leaders are available?
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >>
> > > > > >> > >
> > http://kafka.apache.org/documentation.html#basic_ops_add_topic
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> Guozhang
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> On Fri, Oct 24, 2014 at 2:43 PM, Sameer Yami <
> > > > > >> > > syami@ailive.net>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> > That is not the case.
> > > > > >> > > > > > > > >> > This detailed error is below. Due to this
> > error ,
> > > > the
> > > > > >> > > producer
> > > > > >> > > > > is
> > > > > >> > > > > > > not
> > > > > >> > > > > > > > >> able
> > > > > >> > > > > > > > >> > to send messages
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main DEBUG
> > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-52:
> > > > > >> > > > > > > > >> > Getting broker partition info for topic Test
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,209 main INFO
> > > > > >> > > > > kafka.client.ClientUtils$-68:
> > > > > >> > > > > > > > >> Fetching
> > > > > >> > > > > > > > >> > metadata from broker
> > > > > id:0,host:54.215.200.61,port:9092
> > > > > >> > with
> > > > > >> > > > > > > > correlation
> > > > > >> > > > > > > > >> id
> > > > > >> > > > > > > > >> > 1 for 1 topic(s) Set(Test)
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,230 main DEBUG
> > > > > >> > > > > > > kafka.network.BlockingChannel-52:
> > > > > >> > > > > > > > >> > Created socket with SO_TIMEOUT = 10000
> > (requested
> > > > > >> 10000),
> > > > > >> > > > > > SO_RCVBUF
> > > > > >> > > > > > > =
> > > > > >> > > > > > > > >> > 187200 (requested -1), SO_SNDBUF = 102400
> > > > (requested
> > > > > >> > > 102400).
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,231 main INFO
> > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > >> > > > > > > > >> > Connected to 54.215.200.61:9092 for
> producing
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main INFO
> > > > > >> > > > > > kafka.producer.SyncProducer-68:
> > > > > >> > > > > > > > >> > Disconnecting from 54.215.200.61:9092
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,256 main DEBUG
> > > > > >> > > > > kafka.client.ClientUtils$-52:
> > > > > >> > > > > > > > >> > Successfully fetched metadata for 1 topic(s)
> > > > > Set(Test)
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,257 main WARN
> > > > > >> > > > > > > > >> kafka.producer.BrokerPartitionInfo-83:
> > > > > >> > > > > > > > >> > Error while fetching metadata [{TopicMetadata
> > for
> > > > > topic
> > > > > >> > Test
> > > > > >> > > > ->
> > > > > >> > > > > > > > >> > No partition metadata for topic Test due to
> > > > > >> > > > > > > > >> > kafka.common.LeaderNotAvailableException}]
> for
> > > > topic
> > > > > >> > [Test]:
> > > > > >> > > > > class
> > > > > >> > > > > > > > >> > kafka.common.LeaderNotAvailableException
> > > > > >> > > > > > > > >> > 2014-10-24 14:31:47,258 main ERROR
> > > > > >> > > > > > > > >> > kafka.producer.async.DefaultEventHandler-97:
> > > Failed
> > > > > to
> > > > > >> > > collate
> > > > > >> > > > > > > > messages
> > > > > >> > > > > > > > >> by
> > > > > >> > > > > > > > >> > topic, partition due to: Failed to fetch
> topic
> > > > > metadata
> > > > > >> > for
> > > > > >> > > > > topic:
> > > > > >> > > > > > > > Test
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > On Fri, Oct 24, 2014 at 1:11 PM, Guozhang
> Wang
> > <
> > > > > >> > > > > > wangguoz@gmail.com>
> > > > > >> > > > > > > > >> wrote:
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > > Could you see if this is the case:
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> > > > > >> > > > > > > > >> > > ?
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > On Fri, Oct 24, 2014 at 11:35 AM, Sameer
> > Yami <
> > > > > >> > > > > syami@ailive.net
> > > > > >> > > > > > >
> > > > > >> > > > > > > > >> wrote:
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > > Hi,
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > We have a Kafka / Zookeeper test setup
> on a
> > > > > single
> > > > > >> EC2
> > > > > >> > > > > > machine.
> > > > > >> > > > > > > > The
> > > > > >> > > > > > > > >> > > > host.name is the private ip address and
> > the
> > > > > >> > > > > > > advertised.host.name
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > >> > the
> > > > > >> > > > > > > > >> > > > public address in server.properties.
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > We are trying to publish from outside EC2
> > > using
> > > > > the
> > > > > >> > > public
> > > > > >> > > > > ip
> > > > > >> > > > > > of
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > >> > EC2
> > > > > >> > > > > > > > >> > > > machine but get this error on the client
> > > side -
> > > > > >> > > > > > > > >> > > > No partition metadata for topic test due
> to
> > > > > >> > > > > > > > >> > > >
> kafka.common.LeaderNotAvailableException}]
> > > for
> > > > > >> topic
> > > > > >> > > > [test]:
> > > > > >> > > > > > > class
> > > > > >> > > > > > > > >> > > > kafka.common.LeaderNotAvailableException
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > The zookeeper has the public ip stored in
> > > > > >> > > /brokers/ids/0.
> > > > > >> > > > > > > > >> > > > We have tried multiple combinations but
> it
> > > > works
> > > > > >> ONLY
> > > > > >> > > when
> > > > > >> > > > > the
> > > > > >> > > > > > > > >> > advertised
> > > > > >> > > > > > > > >> > > > host name is not used and the client is
> > > > internal
> > > > > to
> > > > > >> > EC2.
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > Please let me know what the problem could
> > be.
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > thanks
> > > > > >> > > > > > > > >> > > > Sameer
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > --
> > > > > >> > > > > > > > >> > > -- Guozhang
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> --
> > > > > >> > > > > > > > >> -- Guozhang
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > --
> > > > > >> > > > > > > -- Guozhang
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > -- Guozhang
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> -- Guozhang
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang