You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mingtao Zhang <va...@gmail.com> on 2014/05/19 17:32:49 UTC
Cause and Recovery
Hi,
I got this from the log:
[exec] 03:50:06.216 [ProducerSendThread-] [1;31mERROR [0;39m
[1;35mk.producer.async.ProducerSendThread [0;39m - Error in handling batch
of 100 events
[exec] kafka.common.FailedToSendMessageException: Failed to send
messages after 3 tries.
[exec] at kafka.producer.async.DefaultEventHandler.handle(Unknown
Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
[exec] at kafka.producer.async.ProducerSendThread.tryToHandle(Unknown
Source) [kafka_2.10-0.8.0.jar:0.8.0]
[exec] at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
[exec] at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
[exec] at scala.collection.immutable.Stream.foreach(Stream.scala:547)
~[scala-library-2.10.1.jar:na]
[exec] at
kafka.producer.async.ProducerSendThread.processEvents(Unknown Source)
[kafka_2.10-0.8.0.jar:0.8.0]
[exec] at kafka.producer.async.ProducerSendThread.run(Unknown Source)
[kafka_2.10-0.8.0.jar:0.8.0]
My question is:
1. What could cause this?
2. Who should deal with the recovery? User or Kafka?
Let me know if more log is needed. (It hangs the integration test, so I
have 3.5G of them ... from a maven build)
--
Best Regards,
Mingtao Zhang
Re: Cause and Recovery
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Mingtao,
>From this log entry it seems the leader for this topic is not available.
Could you use kafka-topics.sh to check if the topic is created successfully
with its partition leader information?
[exec] 03:50:05.849 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.client.
ClientUtils$ [0;39m - Successfully fetched metadata for
1 topic(s) Set(RAW)
[exec] 03:50:05.850 [ProducerSendThread-] [31mWARN [0;39m
[1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
metadata [{TopicMetadata for topic RAW ->
[exec] No partition metadata for topic RAW due to
kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
kafka.common.LeaderNotAvailableException
On Mon, May 19, 2014 at 11:16 AM, Mingtao Zhang <va...@gmail.com>wrote:
> Hi, Guozhang,
>
> I copied a portion of the log file before the failure:
>
> [exec] 03:50:05.809
> [***_meo01055-1400298605173-eed8b129-leader-finder-thread] [34mINFO [0;39m
> [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
> [***_meo01055-1400298605173-eed8b129-leader-finder-thread], Starting
> [exec] 03:50:05.810 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
> id:0,host:localhost,port:9092 with correlation id 4 for 1 topic(s) Set(RAW)
> [exec] 03:50:05.811 [kafka-acceptor] [39mDEBUG [0;39m
> [1;35mkafka.network.Acceptor [0;39m - Accepted connection from
> /127.0.0.1on /
> 127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
> recvBufferSize [actual|requested]: [1048576|1048576]
> [exec] 03:50:05.812 [kafka-processor-9092-1] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Processor 1 listening to new
> connection from /127.0.0.1:23035
> [exec] 03:50:05.812 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
> = 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
> 102400 (requested 102400).
> [exec] 03:50:05.812 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
> producing
> [exec] 03:50:05.814 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getChildren cxid:0x2d
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.814 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getChildren cxid:0x2d
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.815 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 45,8 replyHeader:: 45,30,0 request::
> '/brokers/ids,T response:: v{'0}
> [exec] 03:50:05.816 [kafka-request-handler-0] [39mDEBUG [0;39m
> [1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
> RAW is Map(0 -> List(0))
> [exec] 03:50:05.817 [kafka-request-handler-0] [34mINFO [0;39m
> [1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
> "0":[ 0 ] }, "version":1 }
> [exec] 03:50:05.817 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40000 type:create
> cxid:0x2e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
> /brokers/topics/RAW
> [exec] 03:50:05.818 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x2e zxid:0x1f txntype:-1
> reqpath:n/a
> [exec] 03:50:05.820 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 46,1 replyHeader:: 46,31,-110 request::
>
> '/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response::
> [exec] 03:50:05.821 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from
> localhost:9092
> [exec] 03:50:05.822 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
> 1 topic(s) Set(RAW)
> [exec] 03:50:05.822 [kafka-processor-9092-1] [34mINFO [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing socket connection to /
> 127.0.0.1.
> [exec] 03:50:05.822 [kafka-processor-9092-1] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing connection from /
> 127.0.0.1:23035
> [exec] 03:50:05.822 [ProducerSendThread-] [31mWARN [0;39m
> [1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
> metadata [{TopicMetadata for topic RAW ->
> [exec] No partition metadata for topic RAW due to
> kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
> kafka.common.LeaderNotAvailableException
> [exec] 03:50:05.823 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.producer.BrokerPartitionInfo [0;39m - Getting broker partition
> info for topic RAW
> [exec] 03:50:05.824 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
> id:0,host:localhost,port:9092 with correlation id 5 for 1 topic(s) Set(RAW)
> [exec] 03:50:05.825 [kafka-acceptor] [39mDEBUG [0;39m
> [1;35mkafka.network.Acceptor [0;39m - Accepted connection from
> /127.0.0.1on /
> 127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
> recvBufferSize [actual|requested]: [1048576|1048576]
> [exec] 03:50:05.825 [kafka-processor-9092-0] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Processor 0 listening to new
> connection from /127.0.0.1:23036
> [exec] 03:50:05.826 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
> = 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
> 102400 (requested 102400).
> [exec] 03:50:05.826 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
> producing
> [exec] 03:50:05.828 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getChildren cxid:0x2f
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.828 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getChildren cxid:0x2f
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.828 [***-pool-0] [39mDEBUG [0;39m
> [1;35mConsumerRunnableBuilder [0;39m - Consumption started
> [exec] 03:50:05.830 [***-pool-1] [39mDEBUG [0;39m
> [1;35mConsumerRunnableBuilder [0;39m - Consumption started
> [exec] 03:50:05.829 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 47,8 replyHeader:: 47,31,0 request::
> '/brokers/ids,T response:: v{'0}
> [exec] 03:50:05.834 [kafka-request-handler-1] [39mDEBUG [0;39m
> [1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
> RAW is Map(0 -> List(0))
> [exec] 03:50:05.834 [***-pool-2] [39mDEBUG [0;39m
> [1;35mConsumerRunnableBuilder [0;39m - Consumption started
> [exec] 03:50:05.834 [kafka-request-handler-1] [34mINFO [0;39m
> [1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
> "0":[ 0 ] }, "version":1 }
> [exec] 03:50:05.835 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40000 type:create
> cxid:0x30 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
> /brokers/topics/RAW
> [exec] 03:50:05.836 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x30 zxid:0x20 txntype:-1
> reqpath:n/a
> [exec] 03:50:05.838 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 48,1 replyHeader:: 48,32,-110 request::
>
> '/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response::
> [exec] 03:50:05.839 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from
> localhost:9092
> [exec] 03:50:05.840 [kafka-processor-9092-0] [34mINFO [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing socket connection to /
> 127.0.0.1.
> [exec] 03:50:05.840 [kafka-processor-9092-0] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing connection from /
> 127.0.0.1:23036
> [exec] 03:50:05.849 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
> 1 topic(s) Set(RAW)
> [exec] 03:50:05.850 [ProducerSendThread-] [31mWARN [0;39m
> [1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
> metadata [{TopicMetadata for topic RAW ->
> [exec] No partition metadata for topic RAW due to
> kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
> kafka.common.LeaderNotAvailableException
> [exec] 03:50:05.850 [ProducerSendThread-] [1;31mERROR [0;39m
> [1;35mk.producer.async.DefaultEventHandler [0;39m - Failed to collate
> messages by topic, partition due to: Failed to fetch topic metadata for
> topic: RAW
> [exec] 03:50:05.851 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mk.producer.async.DefaultEventHandler [0;39m - Back off for 100 ms
> before retrying send. Remaining retries = 1
> [exec] 03:50:05.853 [***-pool-3] [39mDEBUG [0;39m
> [1;35mConsumerRunnableBuilder [0;39m - Consumption started
> [exec] 03:50:05.871 [kafka-logflusher-1] [39mDEBUG [0;39m
> [1;35mkafka.log.LogManager [0;39m - [Log Manager on Broker 0] Checking for
> dirty logs to flush...
> [exec] 03:50:05.872 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:getChildren cxid:0x1a
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.872 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40001 type:getChildren cxid:0x1a
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.873 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 26,8 replyHeader:: 26,32,0 request::
> '/brokers/ids,F response:: v{'0}
> [exec] 03:50:05.874 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:getDate cxid:0x1b zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/ids/0
> [exec] 03:50:05.874 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40001 type:getDate cxid:0x1b zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/ids/0
> [exec] 03:50:05.877 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 27,4 replyHeader:: 27,32,0 request::
> '/brokers/ids/0,F response::
>
> #7b2022686f7374223a226d656f30313035352e75732e6f7261636c652e636f6d222c20226a6d785f706f7274223a2d312c2022706f7274223a393039322c202274696d657374616d70223a2231343030323938363033313237222c202276657273696f6e223a31207d,s{6,6,1400298603202,1400298603202,0,0,0,91769969177788416,105,0,6}
> [exec] 03:50:05.908 [ZkClient-EventThread-24-localhost:2181] [34mINFO
> [0;39m [1;35mkafka.controller.ReplicaStateMachine [0;39m - [Replica state
> machine on controller 0]: Invoking state change to NewReplica for replicas
> PartitionAndReplica(RAW,0,0)
> [exec] 03:50:05.918 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x31 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/RAW/partitions/0/state
> [exec] 03:50:05.918 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x31 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/RAW/partitions/0/state
> [exec] 03:50:05.919 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 49,4 replyHeader:: 49,32,-101 request::
> '/brokers/topics/RAW/partitions/0/state,F response::
> [exec] 03:50:05.920 [ZkClient-EventThread-24-localhost:2181] [34mINFO
> [0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
> machine on Controller 0]: Invoking state change to OnlinePartition for
> partitions [RAW,0]
> [exec] 03:50:05.923 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
> machine on Controller 0]: Live assigned replicas for partition [RAW,0] are:
> [List(0)]
> [exec] 03:50:05.926 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
> machine on Controller 0]: Initializing leader and isr for partition [RAW,0]
> to (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> [exec] 03:50:05.930 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkConnection [0;39m - Creating new ZookKeeper
> instance to connect to localhost:2181.
> [exec] 03:50:05.930 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40000 type:create
> cxid:0x32 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/RAW/partitions/0 Error:KeeperErrorCode = NoNode for
> /brokers/topics/RAW/partitions/0
> [exec] 03:50:05.930 [ImmediateThread-1400298600406] [34mINFO [0;39m
> [1;35morg.apache.zookeeper.ZooKeeper [0;39m - Initiating client connection,
> connectString=localhost:2181 sessionTimeout=30000
> watcher=org.I0Itec.zkclient.ZkClient@10a86662
> [exec] 03:50:05.931 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x32 zxid:0x21 txntype:-1
> reqpath:n/a
> [exec] 03:50:05.933 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 50,1 replyHeader:: 50,33,-101 request::
>
> '/brokers/topics/RAW/partitions/0/state,#7b2022636f6e74726f6c6c65725f65706f6368223a312c2022697372223a5b2030205d2c20226c6561646572223a302c20226c65616465725f65706f6368223a302c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response::
> [exec] 03:50:05.938 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40000 type:create
> cxid:0x33 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/RAW/partitions Error:KeeperErrorCode = NoNode for
> /brokers/topics/RAW/partitions
> [exec] 03:50:05.938 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x33 zxid:0x22 txntype:-1
> reqpath:n/a
> [exec] 03:50:05.940 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 51,1 replyHeader:: 51,34,-101 request::
> '/brokers/topics/RAW/partitions/0,,v{s{31,s{'world,'anyone}}},0 response::
>
> [exec] 03:50:05.940 [ZkClient-EventThread-58-localhost:2181] [34mINFO
> [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Starting ZkClient
> event thread.
> [exec] 03:50:05.941 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x34 zxid:0x23 txntype:1
> reqpath:n/a
> [exec] 03:50:05.941 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:create cxid:0x34 zxid:0x23 txntype:1
> reqpath:n/a
> [exec] 03:50:05.943 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 52,1 replyHeader:: 52,35,0 request::
> '/brokers/topics/RAW/partitions,,v{s{31,s{'world,'anyone}}},0 response::
> '/brokers/topics/RAW/partitions
> [exec] 03:50:05.944 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x35 zxid:0x24 txntype:1
> reqpath:n/a
> [exec] 03:50:05.955 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:create cxid:0x35 zxid:0x24 txntype:1
> reqpath:n/a
> [exec] 03:50:05.956 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 53,1 replyHeader:: 53,36,0 request::
> '/brokers/topics/RAW/partitions/0,,v{s{31,s{'world,'anyone}}},0 response::
> '/brokers/topics/RAW/partitions/0
> [exec] 03:50:05.958 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
> id:0,host:localhost,port:9092 with correlation id 6 for 1 topic(s) Set(RAW)
> [exec] 03:50:05.958 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mk.producer.async.ProducerSendThread [0;39m - 1001 ms elapsed. Queue
> time reached. Sending..
> [exec] 03:50:05.958 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mk.producer.async.ProducerSendThread [0;39m - Handling 0 events
> [exec] 03:50:05.959 [kafka-acceptor] [39mDEBUG [0;39m
> [1;35mkafka.network.Acceptor [0;39m - Accepted connection from
> /127.0.0.1on /
> 127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
> recvBufferSize [actual|requested]: [1048576|1048576]
> [exec] 03:50:05.959 [kafka-processor-9092-1] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Processor 1 listening to new
> connection from /127.0.0.1:23037
> [exec] 03:50:05.960 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x36 zxid:0x25 txntype:1
> reqpath:n/a
> [exec] 03:50:05.960 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
> = 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
> 102400 (requested 102400).
> [exec] 03:50:05.961 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
> producing
> [exec] 03:50:05.974 [ImmediateThread-1400298600406-SendThread()]
> [34mINFO [0;39m [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Opening
> socket connection to server localhost/127.0.0.1:2181
> [exec] 03:50:05.975 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> [34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Accepted
> socket connection from /127.0.0.1:59397
> [exec] 03:50:05.982 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Awaiting connection to
> Zookeeper server
> [exec] 03:50:05.982 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Waiting for keeper state
> SyncConnected
> [exec] 03:50:05.987 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:create cxid:0x36 zxid:0x25 txntype:1
> reqpath:n/a
> [exec] 03:50:05.987 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getChildren cxid:0x37
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.987 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getChildren cxid:0x37
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.988
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [34mINFO [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Socket connection
> established to localhost/127.0.0.1:2181, initiating session
> [exec] 03:50:05.988
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Session establishment
> request sent on localhost/127.0.0.1:2181
> [exec] 03:50:05.988 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> [39mDEBUG [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Session
> establishment request from client /127.0.0.1:59397 client's lastZxid is
> 0x0
> [exec] 03:50:05.989 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> [34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Client
> attempting to establish new session at /127.0.0.1:59397
> [exec] 03:50:05.990 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40003 type:createSession cxid:0x0 zxid:0x26
> txntype:-10 reqpath:n/a
> [exec] 03:50:05.990 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40003 type:createSession cxid:0x0 zxid:0x26
> txntype:-10 reqpath:n/a
> [exec] 03:50:05.991 [SyncThread:0] [34mINFO [0;39m
> [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Established session
> 0x146084cfde40003 with negotiated timeout 30000 for client /
> 127.0.0.1:59397
> [exec] 03:50:05.991
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [34mINFO [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Session establishment
> complete on server localhost/127.0.0.1:2181, sessionid =
> 0x146084cfde40003,
> negotiated timeout = 30000
> [exec] 03:50:05.992 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 54,1 replyHeader:: 54,37,0 request::
>
> '/brokers/topics/RAW/partitions/0/state,#7b2022636f6e74726f6c6c65725f65706f6368223a312c2022697372223a5b2030205d2c20226c6561646572223a302c20226c65616465725f65706f6368223a302c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response:: '/brokers/topics/RAW/partitions/0/state
> [exec] 03:50:05.993 [ImmediateThread-1400298600406-EventThread]
> [39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Received
> event: WatchedEvent state:SyncConnected type:None path:null
> [exec] 03:50:05.993 [ImmediateThread-1400298600406-EventThread]
> [34mINFO [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - zookeeper state
> changed (SyncConnected)
> [exec] 03:50:05.994 [ImmediateThread-1400298600406-EventThread]
> [39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Leaving
> process event
> [exec] 03:50:05.994 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - State is SyncConnected
> [exec] 03:50:05.995 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40003 type:getChildren cxid:0x1
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.995 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40003 type:getChildren cxid:0x1
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:05.996
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40003, packet:: clientPath:null serverPath:null
> finished:false header:: 1,8 replyHeader:: 1,38,0 request::
> '/brokers/ids,F response:: v{'0}
> [exec] 03:50:05.997 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
> RAW is Map(2 -> List(0), 1 -> List(0), 3 -> List(0), 0 -> List(0))
> [exec] 03:50:05.998 [ImmediateThread-1400298600406] [34mINFO [0;39m
> [1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
> "0":[ 0 ], "1":[ 0 ], "2":[ 0 ], "3":[ 0 ] }, "version":1 }
> [exec] 03:50:05.999 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 55,8 replyHeader:: 55,37,0 request::
> '/brokers/ids,T response:: v{'0}
> [exec] 03:50:05.999 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40003 type:create
> cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
> /brokers/topics/RAW
> [exec] 03:50:05.999 [kafka-request-handler-0] [39mDEBUG [0;39m
> [1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
> RAW is Map(0 -> List(0))
> [exec] 03:50:06.000 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40003 type:create cxid:0x2 zxid:0x27 txntype:-1
> reqpath:n/a
> [exec] 03:50:06.000 [kafka-request-handler-0] [34mINFO [0;39m
> [1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
> "0":[ 0 ] }, "version":1 }
> [exec] 03:50:06.002
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40003, packet:: clientPath:null serverPath:null
> finished:false header:: 2,1 replyHeader:: 2,39,-110 request::
>
> '/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response::
> [exec] 03:50:06.002 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:getDate cxid:0x1c zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/RAW
> [exec] 03:50:06.003 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40001 type:getDate cxid:0x1c zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/RAW
> [exec] 03:50:06.005 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 28,4 replyHeader:: 28,39,0 request::
> '/brokers/topics/RAW,T response::
>
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,s{27,27,1400298605448,1400298605448,0,1,0,0,43,1,35}
> [exec] 03:50:06.006 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40000 type:create
> cxid:0x38 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
> /brokers/topics/RAW
> [exec] 03:50:06.007 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x38 zxid:0x28 txntype:-1
> reqpath:n/a
> [exec] 03:50:06.008 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 56,1 replyHeader:: 56,40,-110 request::
>
> '/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response::
> [exec] 03:50:06.010 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from
> localhost:9092
> [exec] 03:50:06.010 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
> 1 topic(s) Set(RAW)
> [exec] 03:50:06.010 [kafka-processor-9092-1] [34mINFO [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing socket connection to /
> 127.0.0.1.
> [exec] 03:50:06.011 [kafka-processor-9092-1] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing connection from /
> 127.0.0.1:23037
> [exec] 03:50:06.011 [ProducerSendThread-] [31mWARN [0;39m
> [1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
> metadata [{TopicMetadata for topic RAW ->
> [exec] No partition metadata for topic RAW due to
> kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
> kafka.common.LeaderNotAvailableException
> [exec] 03:50:06.012 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.producer.BrokerPartitionInfo [0;39m - Getting broker partition
> info for topic RAW
> [exec] 03:50:06.012 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
> id:0,host:localhost,port:9092 with correlation id 7 for 1 topic(s) Set(RAW)
> [exec] 03:50:06.013 [kafka-acceptor] [39mDEBUG [0;39m
> [1;35mkafka.network.Acceptor [0;39m - Accepted connection from
> /127.0.0.1on /
> 127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
> recvBufferSize [actual|requested]: [1048576|1048576]
> [exec] 03:50:06.014 [kafka-processor-9092-0] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Processor 0 listening to new
> connection from /127.0.0.1:23039
> [exec] 03:50:06.015 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
> = 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
> 102400 (requested 102400).
> [exec] creation failed because of topic RAW already exists
> [exec] kafka.common.TopicExistsException: topic RAW already exists
> [exec] at
>
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(Unknown
> Source)
> [exec] at kafka.admin.CreateTopicCommand$.createTopic(Unknown Source)
> [exec] at kafka.admin.CreateTopicCommand$.main(Unknown Source)
> [exec] at kafka.admin.CreateTopicCommand.main(Unknown Source)
> [exec] at
>
> com.***.producer.raw.internal.RawTopicCreator.create(RawTopicCreator.java:39)
> [exec] at
>
> com.***.producer.core.internal.topic.TopicPrimer.createTopics(TopicPrimer.java:35)
> [exec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [exec] at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [exec] at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [exec] at java.lang.reflect.Method.invoke(Method.java:606)
> [exec] at
>
> org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)
> [exec] at
> org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388)
> [exec] at
> org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430)
> [exec] at
> org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
> [exec] at
>
> org.glassfish.hk2.internal.ImmediateHelper.findOrCreate(ImmediateHelper.java:164)
> [exec] at
>
> org.glassfish.hk2.utilities.ImmediateContext.findOrCreate(ImmediateContext.java:73)
> [exec] at
> org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
> [exec] at
>
> org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
> [exec] at
>
> org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
> [exec] at
> org.glassfish.hk2.internal.ImmediateHelper.doWork(ImmediateHelper.java:434)
> [exec] at
> org.glassfish.hk2.internal.ImmediateHelper.run(ImmediateHelper.java:317)
> [exec] at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [exec] at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [exec] at java.lang.Thread.run(Thread.java:724)
> [exec]
> [exec] 03:50:06.019 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
> producing
> [exec] 03:50:06.018 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Closing ZkClient...
> [exec] 03:50:06.020 [ZkClient-EventThread-58-localhost:2181] [34mINFO
> [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Terminate ZkClient
> event thread.
> [exec] 03:50:06.020 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getChildren cxid:0x39
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.021 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getChildren cxid:0x39
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.031 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 57,8 replyHeader:: 57,40,0 request::
> '/brokers/ids,T response:: v{'0}
> [exec] 03:50:06.031 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkConnection [0;39m - Closing ZooKeeper connected
> to localhost:2181
> [exec] 03:50:06.031 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ZooKeeper [0;39m - Closing session:
> 0x146084cfde40003
> [exec] 03:50:06.031 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Closing client for session:
> 0x146084cfde40003
> [exec] 03:50:06.033 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Processed session
> termination for sessionid: 0x146084cfde40003
> [exec] 03:50:06.035 [kafka-request-handler-1] [39mDEBUG [0;39m
> [1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
> RAW is Map(0 -> List(0))
> [exec] 03:50:06.035 [kafka-request-handler-1] [34mINFO [0;39m
> [1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
> "0":[ 0 ] }, "version":1 }
> [exec] 03:50:06.037 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40003 type:closeSession cxid:0x3 zxid:0x29
> txntype:-11 reqpath:n/a
> [exec] 03:50:06.037 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40003 type:closeSession cxid:0x3 zxid:0x29
> txntype:-11 reqpath:n/a
> [exec] 03:50:06.038
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40003, packet:: clientPath:null serverPath:null
> finished:false header:: 3,-11 replyHeader:: 3,41,0 request:: null
> response:: null
> [exec] 03:50:06.039 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Disconnecting client for
> session: 0x146084cfde40003
> [exec] 03:50:06.039 [ImmediateThread-1400298600406] [34mINFO [0;39m
> [1;35morg.apache.zookeeper.ZooKeeper [0;39m - Session: 0x146084cfde40003
> closed
> [exec] 03:50:06.039 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Closing ZkClient...done
> [exec] 03:50:06.040 [ImmediateThread-1400298600406-EventThread]
> [34mINFO [0;39m [1;35morg.apache.zookeeper.ClientCnxn [0;39m - EventThread
> shut down
> [exec] 03:50:06.034 [ZkClient-EventThread-24-localhost:2181] [34mINFO
> [0;39m [1;35mkafka.controller.ReplicaStateMachine [0;39m - [Replica state
> machine on controller 0]: Invoking state change to OnlineReplica for
> replicas PartitionAndReplica(RAW,0,0)
> [exec] 03:50:06.046 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Delivering event #3
> done
> [exec] 03:50:06.049 [kafka-request-handler-0] [34mINFO [0;39m
> [1;35mkafka.server.ReplicaManager [0;39m - [Replica Manager on Broker 0]:
> Handling LeaderAndIsr request
>
> Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(RAW,0)
> ->
>
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:meo01055.us.***,port:9092
> [exec] 03:50:06.054 [kafka-request-handler-0] [34mINFO [0;39m
> [1;35mkafka.server.ReplicaFetcherManager [0;39m - [ReplicaFetcherManager on
> broker 0] Removing fetcher for partition [RAW,0]
> [exec] 03:50:06.056 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> [34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Closed
> socket connection for client /127.0.0.1:59397 which had sessionid
> 0x146084cfde40003
> [exec] 03:50:06.057 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40000 type:create
> cxid:0x3a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
> /brokers/topics/RAW
> [exec] 03:50:06.055 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkConnection [0;39m - Creating new ZookKeeper
> instance to connect to localhost:2181.
> [exec] 03:50:06.058 [ImmediateThread-1400298600406] [34mINFO [0;39m
> [1;35morg.apache.zookeeper.ZooKeeper [0;39m - Initiating client connection,
> connectString=localhost:2181 sessionTimeout=30000
> watcher=org.I0Itec.zkclient.ZkClient@74c824b3
> [exec] 03:50:06.059 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x3a zxid:0x2a txntype:-1
> reqpath:n/a
> [exec] 03:50:06.061 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 58,1 replyHeader:: 58,42,-110 request::
>
> '/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response::
> [exec] 03:50:06.055 [ZkClient-EventThread-61-localhost:2181] [34mINFO
> [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Starting ZkClient
> event thread.
> [exec] 03:50:06.064 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from
> localhost:9092
> [exec] 03:50:06.064 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
> 1 topic(s) Set(RAW)
> [exec] 03:50:06.066 [ProducerSendThread-] [31mWARN [0;39m
> [1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
> metadata [{TopicMetadata for topic RAW ->
> [exec] No partition metadata for topic RAW due to
> kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
> kafka.common.LeaderNotAvailableException
> [exec] 03:50:06.066 [ProducerSendThread-] [1;31mERROR [0;39m
> [1;35mk.producer.async.DefaultEventHandler [0;39m - Failed to collate
> messages by topic, partition due to: Failed to fetch topic metadata for
> topic: RAW
> [exec] 03:50:06.066 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mk.producer.async.DefaultEventHandler [0;39m - Back off for 100 ms
> before retrying send. Remaining retries = 0
> [exec] 03:50:06.067 [kafka-processor-9092-0] [34mINFO [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing socket connection to /
> 127.0.0.1.
> [exec] 03:50:06.068 [kafka-processor-9092-0] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing connection from /
> 127.0.0.1:23039
> [exec] 03:50:06.070 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Partition map for
> /brokers/topics/RAW is Map(0 -> List(0))
> [exec] 03:50:06.075 [ImmediateThread-1400298600406-SendThread()]
> [34mINFO [0;39m [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Opening
> socket connection to server localhost/127.0.0.1:2181
> [exec] 03:50:06.076 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> [34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Accepted
> socket connection from /127.0.0.1:59399
> [exec] 03:50:06.077 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
> [ConsumerFetcherManager-1400298603576] Stopping leader finder thread
> [exec] 03:50:06.078 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], Shutting down
> [exec] 03:50:06.086 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Awaiting connection to
> Zookeeper server
> [exec] 03:50:06.086 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Waiting for keeper state
> SyncConnected
> [exec] 03:50:06.087
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], Stopped
> [exec] 03:50:06.088
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [34mINFO [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Socket connection
> established to localhost/127.0.0.1:2181, initiating session
> [exec] 03:50:06.089
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Session establishment
> request sent on localhost/127.0.0.1:2181
> [exec] 03:50:06.089 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> [39mDEBUG [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Session
> establishment request from client /127.0.0.1:59399 client's lastZxid is
> 0x0
> [exec] 03:50:06.089 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> [34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Client
> attempting to establish new session at /127.0.0.1:59399
> [exec] 03:50:06.090 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], Shutdown
> completed
> [exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
> [ConsumerFetcherManager-1400298603576] Stopping all fetchers
> [exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
> [ConsumerFetcherManager-1400298603576] All connections stopped
> [exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], Cleared all relevant queues for this
> fetcher
> [exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
> data chunk for this consumer iterator
> [exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
> data chunk for this consumer iterator
> [exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
> data chunk for this consumer iterator
> [exec] 03:50:06.092 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
> data chunk for this consumer iterator
> [exec] 03:50:06.092 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], Cleared the data chunks in all the
> consumer message iterators
> [exec] 03:50:06.092 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], Committing all offsets after
> clearing the fetcher queues
> [exec] 03:50:06.093 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], Releasing partition ownership
> [exec] 03:50:06.093 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40004 type:createSession cxid:0x0 zxid:0x2b
> txntype:-10 reqpath:n/a
> [exec] 03:50:06.093 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40004 type:createSession cxid:0x0 zxid:0x2b
> txntype:-10 reqpath:n/a
> [exec] 03:50:06.097 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], Consumer
> RAW_meo01055-1400298603522-1cb22725 rebalancing the following partitions:
> ArrayBuffer(0) for topic RAW with consumers:
> List(RAW_meo01055-1400298603522-1cb22725-0,
> RAW_meo01055-1400298603522-1cb22725-1,
> RAW_meo01055-1400298603522-1cb22725-2,
> RAW_meo01055-1400298603522-1cb22725-3)
> [exec] 03:50:06.101 [SyncThread:0] [34mINFO [0;39m
> [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Established session
> 0x146084cfde40004 with negotiated timeout 30000 for client /
> 127.0.0.1:59399
> [exec] 03:50:06.101
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [34mINFO [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Session establishment
> complete on server localhost/127.0.0.1:2181, sessionid =
> 0x146084cfde40004,
> negotiated timeout = 30000
> [exec] 03:50:06.102 [ImmediateThread-1400298600406-EventThread]
> [39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Received
> event: WatchedEvent state:SyncConnected type:None path:null
> [exec] 03:50:06.102 [ImmediateThread-1400298600406-EventThread]
> [34mINFO [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - zookeeper state
> changed (SyncConnected)
> [exec] 03:50:06.102 [ImmediateThread-1400298600406-EventThread]
> [39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Leaving
> process event
> [exec] 03:50:06.102 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - State is SyncConnected
> [exec] 03:50:06.102 [ZkClient-EventThread-34-localhost:2181] [31mWARN
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], No broker partitions consumed by
> consumer thread RAW_meo01055-1400298603522-1cb22725-1 for topic RAW
> [exec] 03:50:06.108 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40004 type:getChildren cxid:0x1
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.109 [ZkClient-EventThread-34-localhost:2181] [31mWARN
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], No broker partitions consumed by
> consumer thread RAW_meo01055-1400298603522-1cb22725-2 for topic RAW
> [exec] 03:50:06.104 [kafka-request-handler-0] [39mDEBUG [0;39m
> [1;35mkafka.log.FileMessageSet [0;39m - Creating or reloading log segment
>
> /scratch/home/Buildlab/12.1/branches/12.1.4.0.3/webcentersites/siteanalytics/integration-test/consumer-itest/target/kafka-log/RAW-0/00000000000000000000.log
> [exec] 03:50:06.109 [ZkClient-EventThread-34-localhost:2181] [31mWARN
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], No broker partitions consumed by
> consumer thread RAW_meo01055-1400298603522-1cb22725-3 for topic RAW
> [exec] 03:50:06.111 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725],
> RAW_meo01055-1400298603522-1cb22725-0 attempting to claim partition 0
> [exec] 03:50:06.109 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40004 type:getChildren cxid:0x1
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.113 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:getDate cxid:0x1d zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/consumers/RAW/offsets/RAW/0
> [exec] 03:50:06.113 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40001 type:getDate cxid:0x1d zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/consumers/RAW/offsets/RAW/0
> [exec] 03:50:06.114
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40004, packet:: clientPath:null serverPath:null
> finished:false header:: 1,8 replyHeader:: 1,43,0 request::
> '/brokers/ids,F response:: v{'0}
> [exec] 03:50:06.114 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 29,4 replyHeader:: 29,43,-101 request::
> '/consumers/RAW/offsets/RAW/0,F response::
> [exec] 03:50:06.115 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
> ENRICHED is Map(2 -> List(0), 1 -> List(0), 3 -> List(0), 0 -> List(0))
> [exec] 03:50:06.116 [ImmediateThread-1400298600406] [34mINFO [0;39m
> [1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
> "0":[ 0 ], "1":[ 0 ], "2":[ 0 ], "3":[ 0 ] }, "version":1 }
> [exec] 03:50:06.118 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40004 type:create cxid:0x2 zxid:0x2c txntype:1
> reqpath:n/a
> [exec] 03:50:06.118 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.consumer.PartitionTopicInfo [0;39m - initial consumer
> offset of RAW:0: fetched offset = -1: consumed offset = -1 is -1
> [exec] 03:50:06.119 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Got notification
> sessionid:0x146084cfde40002
> [exec] 03:50:06.119 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Got WatchedEvent
> state:SyncConnected type:NodeCreated path:/brokers/topics/ENRICHED for
> sessionid 0x146084cfde40002
> [exec] 03:50:06.119 [ConsumerServiceImpl STARTING-EventThread]
> [39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Received
> event: WatchedEvent state:SyncConnected type:NodeCreated
> path:/brokers/topics/ENRICHED
> [exec] 03:50:06.119 [ConsumerServiceImpl STARTING-EventThread]
> [39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - New
> event: ZkEvent[Data of /brokers/topics/ENRICHED changed sent to
>
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@7703f0d2
> ]
> [exec] 03:50:06.119 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40004 type:create cxid:0x2 zxid:0x2c txntype:1
> reqpath:n/a
> [exec] 03:50:06.119 [ConsumerServiceImpl STARTING-EventThread]
> [39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Leaving
> process event
> [exec] 03:50:06.120 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
> [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Delivering event #5
> ZkEvent[Data of /brokers/topics/ENRICHED changed sent to
>
> kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@7703f0d2
> ]
> [exec] 03:50:06.120 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Got notification
> sessionid:0x146084cfde40000
> [exec] 03:50:06.120 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Got WatchedEvent
> state:SyncConnected type:NodeChildrenChanged path:/brokers/topics for
> sessionid 0x146084cfde40000
> [exec] 03:50:06.121 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.consumer.PartitionTopicInfo [0;39m - initial fetch
> offset of RAW:0: fetched offset = -1: consumed offset = -1 is -1
> [exec] 03:50:06.121 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:exists cxid:0x11 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.122 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:exists cxid:0x11 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.123
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40004, packet:: clientPath:null serverPath:null
> finished:false header:: 2,1 replyHeader:: 2,44,0 request::
>
> '/brokers/topics/ENRICHED,#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response:: '/brokers/topics/ENRICHED
> [exec] 03:50:06.123 [main-EventThread] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Received event: WatchedEvent
> state:SyncConnected type:NodeChildrenChanged path:/brokers/topics
> [exec] 03:50:06.123 [main-EventThread] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - New event:
> ZkEvent[Children of /brokers/topics changed sent to
> kafka.controller.PartitionStateMachine$TopicChangeListener@365b27dc]
> [exec] 03:50:06.123 [main-EventThread] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Leaving process event
> [exec] 03:50:06.123 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Delivering event #6
> ZkEvent[Children of /brokers/topics changed sent to
> kafka.controller.PartitionStateMachine$TopicChangeListener@365b27dc]
> [exec] 03:50:06.124 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:exists cxid:0x3b zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics
> [exec] 03:50:06.124 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:exists cxid:0x3b zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics
> [exec] 03:50:06.124 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 17,3 replyHeader:: 17,44,0 request::
> '/brokers/topics/ENRICHED,T response::
> s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
> [exec] 03:50:06.125 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 59,3 replyHeader:: 59,44,0 request::
> '/brokers/topics,T response::
> s{26,26,1400298605436,1400298605436,0,2,0,0,0,2,44}
> [exec] 03:50:06.118 [kafka-request-handler-0] [39mDEBUG [0;39m
> [1;35mkafka.log.OffsetIndex [0;39m - Loaded index file
>
> /scratch/home/Buildlab/12.1/branches/12.1.4.0.3/webcentersites/siteanalytics/integration-test/consumer-itest/target/kafka-log/RAW-0/00000000000000000000.index
> with maxEntries = 1310720, maxIndexSize = 10485760, entries = 0, lastOffset
> = 0, file position = 0
> [exec] 03:50:06.126 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35mkafka.admin.AdminUtils$ [0;39m - Updated path
> /brokers/topics/ENRICHED with { "partitions":{ "0":[ 0 ], "1":[ 0 ], "2":[
> 0 ], "3":[ 0 ] }, "version":1 } for replica assignment
> [exec] creation succeeded!
> [exec] 03:50:06.126 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Closing ZkClient...
> [exec] 03:50:06.127 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:getDate cxid:0x12 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.127 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:getDate cxid:0x12 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.127 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getChildren cxid:0x3c
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/topics
> [exec] 03:50:06.127 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getChildren cxid:0x3c
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/topics
> [exec] 03:50:06.129 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 60,8 replyHeader:: 60,44,0 request::
> '/brokers/topics,T response:: v{'RAW,'ENRICHED}
> [exec] 03:50:06.129 [ZkClient-EventThread-61-localhost:2181] [34mINFO
> [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Terminate ZkClient
> event thread.
> [exec] 03:50:06.133 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 18,4 replyHeader:: 18,44,0 request::
> '/brokers/topics/ENRICHED,T response::
>
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
> [exec] 03:50:06.133 [ZkClient-EventThread-48-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [***_meo01055-1400298605173-eed8b129], Topic info for path
> /brokers/topics/ENRICHED changed to { "partitions":{ "0":[ 0 ], "1":[ 0 ],
> "2":[ 0 ], "3":[ 0 ] }, "version":1 }, triggering rebalance
> [exec] 03:50:06.133 [ZkClient-EventThread-48-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [***_meo01055-1400298605173-eed8b129], begin rebalancing consumer
> ***_meo01055-1400298605173-eed8b129 try #0
> [exec] 03:50:06.134 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkConnection [0;39m - Closing ZooKeeper connected
> to localhost:2181
> [exec] 03:50:06.134 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ZooKeeper [0;39m - Closing session:
> 0x146084cfde40004
> [exec] 03:50:06.134 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Closing client for session:
> 0x146084cfde40004
> [exec] 03:50:06.135 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mk.c.PartitionStateMachine$TopicChangeListener [0;39m -
> [TopicChangeListener on Controller 0]: Topic change listener fired for path
> /brokers/topics with children RAW,ENRICHED
> [exec] 03:50:06.136 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x3d zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.136 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x3d zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.137 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Processed session
> termination for sessionid: 0x146084cfde40004
> [exec] 03:50:06.138 [kafka-request-handler-0] [34mINFO [0;39m
> [1;35mkafka.log.Log [0;39m - [Kafka Log on Broker 0], Completed load of log
> RAW-0 with log end offset 0
> [exec] 03:50:06.139 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 61,4 replyHeader:: 61,44,0 request::
> '/brokers/topics/ENRICHED,F response::
>
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
> [exec] 03:50:06.139 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40004 type:closeSession cxid:0x3 zxid:0x2d
> txntype:-11 reqpath:n/a
> [exec] 03:50:06.140 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40004 type:closeSession cxid:0x3 zxid:0x2d
> txntype:-11 reqpath:n/a
> [exec] 03:50:06.140 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:getChildren cxid:0x13
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.140 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:getChildren cxid:0x13
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.141
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40004, packet:: clientPath:null serverPath:null
> finished:false header:: 3,-11 replyHeader:: 3,45,0 request:: null
> response:: null
> [exec] 03:50:06.141 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> [34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Closed
> socket connection for client /127.0.0.1:59399 which had sessionid
> 0x146084cfde40004
> [exec] 03:50:06.141 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Disconnecting client for
> session: 0x146084cfde40004
> [exec] 03:50:06.142 [ImmediateThread-1400298600406] [34mINFO [0;39m
> [1;35morg.apache.zookeeper.ZooKeeper [0;39m - Session: 0x146084cfde40004
> closed
> [exec] 03:50:06.142 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
> [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Closing ZkClient...done
> [exec] 03:50:06.143 [ImmediateThread-1400298600406-EventThread]
> [34mINFO [0;39m [1;35morg.apache.zookeeper.ClientCnxn [0;39m - EventThread
> shut down
> [exec] 03:50:06.150 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 19,8 replyHeader:: 19,45,0 request::
> '/brokers/ids,F response:: v{'0}
> [exec] 03:50:06.151 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], RAW:0: fetched offset = -1: consumed
> offset = -1 selected new offset -1
> [exec] 03:50:06.152 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:getDate cxid:0x14 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/ids/0
> [exec] 03:50:06.152 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:getDate cxid:0x14 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/ids/0
> [exec] 03:50:06.152 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40001 type:create
> cxid:0x1e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/consumers/RAW/owners/RAW Error:KeeperErrorCode = NoNode for
> /consumers/RAW/owners/RAW
> [exec] 03:50:06.153 [kafka-request-handler-0] [34mINFO [0;39m
> [1;35mkafka.log.LogManager [0;39m - [Log Manager on Broker 0] Created log
> for partition [RAW,0] in
>
> /scratch/home/Buildlab/12.1/branches/12.1.4.0.3/webcentersites/siteanalytics/integration-test/consumer-itest/target/kafka-log.
> [exec] 03:50:06.154 [kafka-request-handler-0] [31mWARN [0;39m
> [1;35mkafka.server.HighwaterMarkCheckpoint [0;39m - No highwatermark file
> is found. Returning 0 as the highwatermark for partition [RAW,0]
> [exec] 03:50:06.155 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 20,4 replyHeader:: 20,45,0 request::
> '/brokers/ids/0,F response::
>
> #7b2022686f7374223a226d656f30313035352e75732e6f7261636c652e636f6d222c20226a6d785f706f7274223a2d312c2022706f7274223a393039322c202274696d657374616d70223a2231343030323938363033313237222c202276657273696f6e223a31207d,s{6,6,1400298603202,1400298603202,0,0,0,91769969177788416,105,0,6}
> [exec] 03:50:06.156
> [ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - An exception was thrown
> while closing send thread for session 0x146084cfde40004 : Unable to read
> additional data from server sessionid 0x146084cfde40004, likely server has
> closed socket
> [exec] 03:50:06.159 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:create cxid:0x1e zxid:0x2e txntype:-1
> reqpath:n/a
> [exec] 03:50:06.160 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 30,1 replyHeader:: 30,46,-101 request::
>
> '/consumers/RAW/owners/RAW/0,#5241575f6d656f30313035352d313430303239383630333532322d31636232323732352d30,v{s{31,s{'world,'anyone}}},1
> response::
> [exec] 03:50:06.161 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40001 type:create
> cxid:0x1f zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/consumers/RAW/owners Error:KeeperErrorCode = NoNode for
> /consumers/RAW/owners
> [exec] 03:50:06.163 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:create cxid:0x1f zxid:0x2f txntype:-1
> reqpath:n/a
> [exec] 03:50:06.165 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 31,1 replyHeader:: 31,47,-101 request::
> '/consumers/RAW/owners/RAW,,v{s{31,s{'world,'anyone}}},0 response::
> [exec] 03:50:06.168 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
> id:0,host:localhost,port:9092 with correlation id 8 for 1 topic(s) Set(RAW)
> [exec] 03:50:06.169 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:create cxid:0x20 zxid:0x30 txntype:1
> reqpath:n/a
> [exec] 03:50:06.169 [kafka-acceptor] [39mDEBUG [0;39m
> [1;35mkafka.network.Acceptor [0;39m - Accepted connection from
> /127.0.0.1on /
> 127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
> recvBufferSize [actual|requested]: [1048576|1048576]
> [exec] 03:50:06.169 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40001 type:create cxid:0x20 zxid:0x30 txntype:1
> reqpath:n/a
> [exec] 03:50:06.170 [kafka-processor-9092-1] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Processor 1 listening to new
> connection from /127.0.0.1:23041
> [exec] 03:50:06.170 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
> = 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
> 102400 (requested 102400).
> [exec] 03:50:06.170 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
> producing
> [exec] 03:50:06.172 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getChildren cxid:0x3e
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.172 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getChildren cxid:0x3e
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.173 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 62,8 replyHeader:: 62,48,0 request::
> '/brokers/ids,T response:: v{'0}
> [exec] 03:50:06.174 [kafka-request-handler-1] [39mDEBUG [0;39m
> [1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
> RAW is Map(0 -> List(0))
> [exec] 03:50:06.174 [kafka-request-handler-1] [34mINFO [0;39m
> [1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
> "0":[ 0 ] }, "version":1 }
> [exec] 03:50:06.175 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40000 type:create
> cxid:0x3f zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
> /brokers/topics/RAW
> [exec] 03:50:06.176 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 32,1 replyHeader:: 32,48,0 request::
> '/consumers/RAW/owners,,v{s{31,s{'world,'anyone}}},0 response::
> '/consumers/RAW/owners
> [exec] 03:50:06.176 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x3f zxid:0x31 txntype:-1
> reqpath:n/a
> [exec] 03:50:06.178 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 63,1 replyHeader:: 63,49,-110 request::
>
> '/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response::
> [exec] 03:50:06.179 [ProducerSendThread-] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from
> localhost:9092
> [exec] 03:50:06.179 [kafka-processor-9092-1] [34mINFO [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing socket connection to /
> 127.0.0.1.
> [exec] 03:50:06.180 [kafka-processor-9092-1] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing connection from /
> 127.0.0.1:23041
> [exec] 03:50:06.179 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
> 1 topic(s) Set(RAW)
> [exec] 03:50:06.180 [ProducerSendThread-] [31mWARN [0;39m
> [1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
> metadata [{TopicMetadata for topic RAW ->
> [exec] No partition metadata for topic RAW due to
> kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
> kafka.common.LeaderNotAvailableException
> [exec] 03:50:06.183 [ProducerSendThread-] [1;31mERROR [0;39m
> [1;35mk.producer.async.DefaultEventHandler [0;39m - Failed to send requests
> for topics RAW with correlation ids in [0,8]
> [exec] 03:50:06.183 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:create cxid:0x21 zxid:0x32 txntype:1
> reqpath:n/a
> [exec] 03:50:06.198 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40001 type:create cxid:0x21 zxid:0x32 txntype:1
> reqpath:n/a
> [exec] 03:50:06.200 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 33,1 replyHeader:: 33,50,0 request::
> '/consumers/RAW/owners/RAW,,v{s{31,s{'world,'anyone}}},0 response::
> '/consumers/RAW/owners/RAW
> [exec] 03:50:06.201 [kafka-request-handler-0] [39mDEBUG [0;39m
> [1;35mkafka.cluster.Partition [0;39m - Partition [RAW,0] on broker 0: Old
> hw for partition [RAW,0] is 0. New hw is 0. All leo's are 0
> [exec] 03:50:06.206 [kafka-request-handler-0] [34mINFO [0;39m
> [1;35mkafka.server.ReplicaManager [0;39m - [Replica Manager on Broker 0]:
> Handled leader and isr request
>
> Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(RAW,0)
> ->
>
> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:meo01055.us.***,port:9092
> [exec] 03:50:06.208 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:create cxid:0x22 zxid:0x33 txntype:1
> reqpath:n/a
> [exec] 03:50:06.209 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40001 type:create cxid:0x22 zxid:0x33 txntype:1
> reqpath:n/a
> [exec] 03:50:06.210 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 34,1 replyHeader:: 34,51,0 request::
>
> '/consumers/RAW/owners/RAW/0,#5241575f6d656f30313035352d313430303239383630333532322d31636232323732352d30,v{s{31,s{'world,'anyone}}},1
> response:: '/consumers/RAW/owners/RAW/0
> [exec] 03:50:06.215 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725],
> RAW_meo01055-1400298603522-1cb22725-0 successfully owned partition 0 for
> topic RAW
> [exec] 03:50:06.215 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], Updating the cache
> [exec] 03:50:06.215 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], Partitions per topic cache Map(RAW
> -> ArrayBuffer(0))
> [exec] 03:50:06.216 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], Consumers per topic cache Map(RAW ->
> List(RAW_meo01055-1400298603522-1cb22725-0,
> RAW_meo01055-1400298603522-1cb22725-1,
> RAW_meo01055-1400298603522-1cb22725-2,
> RAW_meo01055-1400298603522-1cb22725-3))
> [exec] 03:50:06.216 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], Consumer
> RAW_meo01055-1400298603522-1cb22725 selected partitions : RAW:0: fetched
> offset = -1: consumed offset = -1
>
> After the failure, the logs looks like a 'loop':
>
> [exec] 03:50:06.227 [ZkClient-EventThread-34-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [RAW_meo01055-1400298603522-1cb22725], end rebalancing consumer
> RAW_meo01055-1400298603522-1cb22725 try #0
> [exec] 03:50:06.228 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
> [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Delivering event #4
> done
> [exec] 03:50:06.243
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], Starting
> [exec] 03:50:06.245 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:getChildren cxid:0x23
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.245 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40001 type:getChildren cxid:0x23
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.246 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 35,8 replyHeader:: 35,51,0 request::
> '/brokers/ids,F response:: v{'0}
> [exec] 03:50:06.251 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40001 type:getDate cxid:0x24 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/ids/0
> [exec] 03:50:06.251 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40001 type:getDate cxid:0x24 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/ids/0
> [exec] 03:50:06.253 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
> finished:false header:: 36,4 replyHeader:: 36,51,0 request::
> '/brokers/ids/0,F response::
>
> #7b2022686f7374223a226d656f30313035352e75732e6f7261636c652e636f6d222c20226a6d785f706f7274223a2d312c2022706f7274223a393039322c202274696d657374616d70223a2231343030323938363033313237222c202276657273696f6e223a31207d,s{6,6,1400298603202,1400298603202,0,0,0,91769969177788416,105,0,6}
> [exec] 03:50:06.294 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Replicas assigned to topic
> [ENRICHED], partition [0] are [List(0)]
> [exec] 03:50:06.295 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Replicas assigned to topic
> [ENRICHED], partition [1] are [List(0)]
> [exec] 03:50:06.296 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Replicas assigned to topic
> [ENRICHED], partition [2] are [List(0)]
> [exec] 03:50:06.296 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Replicas assigned to topic
> [ENRICHED], partition [3] are [List(0)]
> [exec] 03:50:06.298 [ZkClient-EventThread-24-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.PartitionStateMachine$TopicChangeListener [0;39m -
> [TopicChangeListener on Controller 0]: New topics: [Set(ENRICHED)], deleted
> topics: [Set()], new partition replica assignment [Map([ENRICHED,3] ->
> List(0), [ENRICHED,2] -> List(0), [ENRICHED,0] -> List(0), [ENRICHED,1] ->
> List(0))]
> [exec] 03:50:06.307 [ZkClient-EventThread-24-localhost:2181] [34mINFO
> [0;39m [1;35mkafka.controller.KafkaController [0;39m - [Controller 0]: New
> topic creation callback for
> [ENRICHED,3],[ENRICHED,2],[ENRICHED,0],[ENRICHED,1]
> [exec] 03:50:06.309 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:exists cxid:0x40 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.309 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:exists cxid:0x40 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.311 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 64,3 replyHeader:: 64,51,0 request::
> '/brokers/topics/ENRICHED,T response::
> s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
> [exec] 03:50:06.311 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Subscribed data changes
> for /brokers/topics/ENRICHED
> [exec] 03:50:06.312 [ZkClient-EventThread-24-localhost:2181] [34mINFO
> [0;39m [1;35mkafka.controller.KafkaController [0;39m - [Controller 0]: New
> partition creation callback for
> [ENRICHED,3],[ENRICHED,2],[ENRICHED,0],[ENRICHED,1]
> [exec] 03:50:06.313 [ZkClient-EventThread-24-localhost:2181] [34mINFO
> [0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
> machine on Controller 0]: Invoking state change to NewPartition for
> partitions [ENRICHED,3],[ENRICHED,2],[ENRICHED,0],[ENRICHED,1]
> [exec] 03:50:06.314 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x41 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.314 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x41 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.316 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 65,4 replyHeader:: 65,51,0 request::
> '/brokers/topics/ENRICHED,T response::
>
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
> [exec] 03:50:06.326 [ImmediateThread-1400298600406] [34mINFO [0;39m
> [1;35mconfig.core [0;39m - Loading ***/***_schema.yaml
> [exec] 03:50:06.335 [ImmediateThread-1400298600406] [34mINFO [0;39m
> [1;35mconfig.core [0;39m - Found resource [***/***_schema.yaml] at
>
> [jar:file:/scratch/home/Buildlab/12.1/branches/12.1.4.0.3/webcenter***/***/app/config/config-dev/target/config-dev-1.0-SNAPSHOT.jar!/***/***_schema.yaml]
> [exec] 03:50:06.387
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mkafka.utils.VerifiableProperties [0;39m - Verifying properties
> [exec] 03:50:06.387
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mkafka.utils.VerifiableProperties [0;39m - Property
> metadata.broker.list is overridden to meo01055.us.***.com:9092
> [exec] 03:50:06.388
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mkafka.utils.VerifiableProperties [0;39m - Property
> request.timeout.msis overridden to 30000
> [exec] 03:50:06.388
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mkafka.utils.VerifiableProperties [0;39m - Property client.id is
> overridden to RAW
> [exec] 03:50:06.394
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
> id:0,host:meo01055.us.***.com,port:9092 with correlation id 0 for 1
> topic(s) Set(RAW)
> [exec] 03:50:06.395 [kafka-acceptor] [39mDEBUG [0;39m
> [1;35mkafka.network.Acceptor [0;39m - Accepted connection from /
> 10.152.196.47 on /10.152.196.47:9092. sendBufferSize [actual|requested]:
> [1048576|1048576] recvBufferSize [actual|requested]: [1048576|1048576]
> [exec] 03:50:06.395 [kafka-processor-9092-0] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Processor 0 listening to new
> connection from /10.152.196.47:62293
> [exec] 03:50:06.396
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
> [1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
> = 30000 (requested 30000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
> 102400 (requested 102400).
> [exec] 03:50:06.400
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Connected to
> meo01055.us.***.com:9092 for producing
> [exec] 03:50:06.406 [kafka-request-handler-0] [39mDEBUG [0;39m
> [1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] [RAW,0];replicas =
> Set(0), in sync replicas = List(0), leader = 0
> [exec] 03:50:06.422
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from
> meo01055.us.***.com:9092
> [exec] 03:50:06.422 [kafka-processor-9092-0] [34mINFO [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing socket connection to /
> 10.152.196.47.
> [exec] 03:50:06.423 [kafka-processor-9092-0] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Closing connection from /
> 10.152.196.47:62293
> [exec] 03:50:06.430
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
> [1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
> 1 topic(s) Set(RAW)
> [exec] 03:50:06.434
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
> [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], {TopicMetadata
> for topic RAW ->
> [exec] Metadata for partition [RAW,0] is partition 0 leader: 0
> (meo01055.us.***.com:9092) replicas: 0 (meo01055.us.***.com:9092) isr: 0
> (meo01055.us.***.com:9092) isUnderReplicated: false}
> [exec] 03:50:06.463 [kafka-acceptor] [39mDEBUG [0;39m
> [1;35mkafka.network.Acceptor [0;39m - Accepted connection from /
> 10.152.196.47 on /10.152.196.47:9092. sendBufferSize [actual|requested]:
> [1048576|1048576] recvBufferSize [actual|requested]: [1048576|1048576]
> [exec] 03:50:06.463 [kafka-processor-9092-1] [39mDEBUG [0;39m
> [1;35mkafka.network.Processor [0;39m - Processor 1 listening to new
> connection from /10.152.196.47:62294
> [exec] 03:50:06.464
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
> [1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
> = 30000 (requested 30000), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF =
> 26010 (requested -1).
> [exec] 03:50:06.469
> [ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0] [34mINFO
> [0;39m [1;35mkafka.consumer.ConsumerFetcherThread [0;39m -
> [ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0], Starting
> [exec] 03:50:06.502 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:getDate cxid:0x15 zxid:0xfffffffffffffffe
> txntype:unknown
> reqpath:/consumers/***/ids/***_meo01055-1400298605173-eed8b129
> [exec] 03:50:06.503 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:getDate cxid:0x15 zxid:0xfffffffffffffffe
> txntype:unknown
> reqpath:/consumers/***/ids/***_meo01055-1400298605173-eed8b129
> [exec] 03:50:06.505 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 21,4 replyHeader:: 21,51,0 request::
> '/consumers/***/ids/***_meo01055-1400298605173-eed8b129,F response::
>
> #7b20227061747465726e223a22737461746963222c2022737562736372697074696f6e223a7b2022454e524943484544223a2034207d2c202274696d657374616d70223a2231343030323938363035323133222c202276657273696f6e223a31207d,s{24,24,1400298605244,1400298605244,0,0,0,91769969177788418,98,0,24}
> [exec] 03:50:06.533
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
> [1;35mkafka.consumer.PartitionTopicInfo [0;39m - reset fetch offset of (
> RAW:0: fetched offset = 0: consumed offset = -1 ) to 0
> [exec] 03:50:06.535
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
> [1;35mkafka.consumer.PartitionTopicInfo [0;39m - reset consume offset of
> RAW:0: fetched offset = 0: consumed offset = 0 to 0
> [exec] 03:50:06.543
> [RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
> [1;35mk.consumer.ConsumerFetcherManager [0;39m -
> [ConsumerFetcherManager-1400298603576] Adding fetcher for partition
> [RAW,0], initOffset -1 to broker 0 with fetcherId 0
> [exec] 03:50:06.572 [kafka-request-handler-0] [39mDEBUG [0;39m
> [1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
> with correlation id 0 from client
> RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
> purgatory
> [exec] 03:50:06.595 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x42 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.595 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x42 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.598 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 66,4 replyHeader:: 66,51,0 request::
> '/brokers/topics/ENRICHED,T response::
>
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
> [exec] 03:50:06.647 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:getChildren cxid:0x16
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/consumers/***/ids
> [exec] 03:50:06.647 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:getChildren cxid:0x16
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/consumers/***/ids
> [exec] 03:50:06.649 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 22,8 replyHeader:: 22,51,0 request::
> '/consumers/***/ids,T response:: v{'***_meo01055-1400298605173-eed8b129}
> [exec] 03:50:06.650 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:getDate cxid:0x17 zxid:0xfffffffffffffffe
> txntype:unknown
> reqpath:/consumers/***/ids/***_meo01055-1400298605173-eed8b129
> [exec] 03:50:06.650 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:getDate cxid:0x17 zxid:0xfffffffffffffffe
> txntype:unknown
> reqpath:/consumers/***/ids/***_meo01055-1400298605173-eed8b129
> [exec] 03:50:06.653 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 23,4 replyHeader:: 23,51,0 request::
> '/consumers/***/ids/***_meo01055-1400298605173-eed8b129,F response::
>
> #7b20227061747465726e223a22737461746963222c2022737562736372697074696f6e223a7b2022454e524943484544223a2034207d2c202274696d657374616d70223a2231343030323938363035323133222c202276657273696f6e223a31207d,s{24,24,1400298605244,1400298605244,0,0,0,91769969177788418,98,0,24}
> [exec] 03:50:06.686 [request-expiration-task] [39mDEBUG [0;39m
> [1;35mk.s.KafkaApis$FetchRequestPurgatory [0;39m -
> [FetchRequestPurgatory-0] Expiring fetch request Name: FetchRequest;
> Version: 0; CorrelationId: 0; ClientId:
> RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0;
> ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [RAW,0] ->
> PartitionFetchInfo(0,1048576).
> [exec] 03:50:06.810 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x43 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.811 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x43 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.813 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 67,4 replyHeader:: 67,51,0 request::
> '/brokers/topics/ENRICHED,T response::
>
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
> [exec] 03:50:06.818 [kafka-request-handler-1] [39mDEBUG [0;39m
> [1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
> with correlation id 1 from client
> RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
> purgatory
> [exec] 03:50:06.887 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:getChildren cxid:0x18
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.887 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:getChildren cxid:0x18
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
> [exec] 03:50:06.889 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 24,8 replyHeader:: 24,51,0 request::
> '/brokers/ids,F response:: v{'0}
> [exec] 03:50:06.890 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:getDate cxid:0x19 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/ids/0
> [exec] 03:50:06.890 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:getDate cxid:0x19 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/ids/0
> [exec] 03:50:06.893 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 25,4 replyHeader:: 25,51,0 request::
> '/brokers/ids/0,F response::
>
> #7b2022686f7374223a226d656f30313035352e75732e6f7261636c652e636f6d222c20226a6d785f706f7274223a2d312c2022706f7274223a393039322c202274696d657374616d70223a2231343030323938363033313237222c202276657273696f6e223a31207d,s{6,6,1400298603202,1400298603202,0,0,0,91769969177788416,105,0,6}
> [exec] 03:50:06.918 [request-expiration-task] [39mDEBUG [0;39m
> [1;35mk.s.KafkaApis$FetchRequestPurgatory [0;39m -
> [FetchRequestPurgatory-0] Expiring fetch request Name: FetchRequest;
> Version: 0; CorrelationId: 1; ClientId:
> RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0;
> ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [RAW,0] ->
> PartitionFetchInfo(0,1048576).
> [exec] 03:50:06.922 [kafka-request-handler-0] [39mDEBUG [0;39m
> [1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
> with correlation id 2 from client
> RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
> purgatory
> [exec] 03:50:06.956 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x44 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.957 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x44 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:06.960 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 68,4 replyHeader:: 68,51,0 request::
> '/brokers/topics/ENRICHED,T response::
>
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
> [exec] 03:50:06.960 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mk.producer.async.ProducerSendThread [0;39m - 1002 ms elapsed. Queue
> time reached. Sending..
> [exec] 03:50:06.960 [ProducerSendThread-] [39mDEBUG [0;39m
> [1;35mk.producer.async.ProducerSendThread [0;39m - Handling 0 events
> [exec] 03:50:07.025 [request-expiration-task] [39mDEBUG [0;39m
> [1;35mk.s.KafkaApis$FetchRequestPurgatory [0;39m -
> [FetchRequestPurgatory-0] Expiring fetch request Name: FetchRequest;
> Version: 0; CorrelationId: 2; ClientId:
> RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0;
> ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [RAW,0] ->
> PartitionFetchInfo(0,1048576).
> [exec] 03:50:07.028 [kafka-request-handler-1] [39mDEBUG [0;39m
> [1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
> with correlation id 3 from client
> RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
> purgatory
> [exec] 03:50:07.052 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40002 type:getDate cxid:0x1a zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:07.052 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40002 type:getDate cxid:0x1a zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED
> [exec] 03:50:07.054 [ConsumerServiceImpl
> STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
> finished:false header:: 26,4 replyHeader:: 26,51,0 request::
> '/brokers/topics/ENRICHED,T response::
>
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
> [exec] 03:50:07.097 [ZkClient-EventThread-24-localhost:2181] [34mINFO
> [0;39m [1;35mkafka.controller.ReplicaStateMachine [0;39m - [Replica state
> machine on controller 0]: Invoking state change to NewReplica for replicas
>
> PartitionAndReplica(ENRICHED,3,0),PartitionAndReplica(ENRICHED,2,0),PartitionAndReplica(ENRICHED,0,0),PartitionAndReplica(ENRICHED,1,0)
> [exec] 03:50:07.100 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x45 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/3/state
> [exec] 03:50:07.100 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x45 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/3/state
> [exec] 03:50:07.102 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 69,4 replyHeader:: 69,51,-101 request::
> '/brokers/topics/ENRICHED/partitions/3/state,F response::
> [exec] 03:50:07.103 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x46 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/2/state
> [exec] 03:50:07.103 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x46 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/2/state
> [exec] 03:50:07.105 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 70,4 replyHeader:: 70,51,-101 request::
> '/brokers/topics/ENRICHED/partitions/2/state,F response::
> [exec] 03:50:07.107 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x47 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/0/state
> [exec] 03:50:07.107 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x47 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/0/state
> [exec] 03:50:07.109 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 71,4 replyHeader:: 71,51,-101 request::
> '/brokers/topics/ENRICHED/partitions/0/state,F response::
> [exec] 03:50:07.110 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:getDate cxid:0x48 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/1/state
> [exec] 03:50:07.110 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:getDate cxid:0x48 zxid:0xfffffffffffffffe
> txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/1/state
> [exec] 03:50:07.112 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 72,4 replyHeader:: 72,51,-101 request::
> '/brokers/topics/ENRICHED/partitions/1/state,F response::
> [exec] 03:50:07.114 [ZkClient-EventThread-24-localhost:2181] [34mINFO
> [0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
> machine on Controller 0]: Invoking state change to OnlinePartition for
> partitions [ENRICHED,3],[ENRICHED,2],[ENRICHED,0],[ENRICHED,1]
> [exec] 03:50:07.121 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
> machine on Controller 0]: Live assigned replicas for partition [ENRICHED,3]
> are: [List(0)]
> [exec] 03:50:07.122 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
> machine on Controller 0]: Initializing leader and isr for partition
> [ENRICHED,3] to (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
> [exec] 03:50:07.124 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40000 type:create
> cxid:0x49 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/ENRICHED/partitions/3 Error:KeeperErrorCode = NoNode
> for /brokers/topics/ENRICHED/partitions/3
> [exec] 03:50:07.125 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x49 zxid:0x34 txntype:-1
> reqpath:n/a
> [exec] 03:50:07.127 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 73,1 replyHeader:: 73,52,-101 request::
>
> '/brokers/topics/ENRICHED/partitions/3/state,#7b2022636f6e74726f6c6c65725f65706f6368223a312c2022697372223a5b2030205d2c20226c6561646572223a302c20226c65616465725f65706f6368223a302c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response::
> [exec] 03:50:07.128 [ProcessThread:-1] [34mINFO [0;39m
> [1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
> KeeperException when processing sessionid:0x146084cfde40000 type:create
> cxid:0x4a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/ENRICHED/partitions Error:KeeperErrorCode = NoNode for
> /brokers/topics/ENRICHED/partitions
> [exec] 03:50:07.129 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x4a zxid:0x35 txntype:-1
> reqpath:n/a
> [exec] 03:50:07.129 [request-expiration-task] [39mDEBUG [0;39m
> [1;35mk.s.KafkaApis$FetchRequestPurgatory [0;39m -
> [FetchRequestPurgatory-0] Expiring fetch request Name: FetchRequest;
> Version: 0; CorrelationId: 3; ClientId:
> RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0;
> ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [RAW,0] ->
> PartitionFetchInfo(0,1048576).
> [exec] 03:50:07.133 [kafka-request-handler-0] [39mDEBUG [0;39m
> [1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
> with correlation id 4 from client
> RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
> purgatory
> [exec] 03:50:07.134 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 74,1 replyHeader:: 74,53,-101 request::
> '/brokers/topics/ENRICHED/partitions/3,,v{s{31,s{'world,'anyone}}},0
> response::
> [exec] 03:50:07.136 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x4b zxid:0x36 txntype:1
> reqpath:n/a
> [exec] 03:50:07.136 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:create cxid:0x4b zxid:0x36 txntype:1
> reqpath:n/a
> [exec] 03:50:07.146 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 75,1 replyHeader:: 75,54,0 request::
> '/brokers/topics/ENRICHED/partitions,,v{s{31,s{'world,'anyone}}},0
> response:: '/brokers/topics/ENRICHED/partitions
> [exec] 03:50:07.152 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x4c zxid:0x37 txntype:1
> reqpath:n/a
> [exec] 03:50:07.153 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:create cxid:0x4c zxid:0x37 txntype:1
> reqpath:n/a
> [exec] 03:50:07.157 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Partition map for
> /brokers/topics/ENRICHED is Map(0 -> List(0), 1 -> List(0), 2 -> List(0), 3
> -> List(0))
> [exec] 03:50:07.162 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 76,1 replyHeader:: 76,55,0 request::
> '/brokers/topics/ENRICHED/partitions/3,,v{s{31,s{'world,'anyone}}},0
> response:: '/brokers/topics/ENRICHED/partitions/3
> [exec] 03:50:07.164 [ZkClient-EventThread-48-localhost:2181] [34mINFO
> [0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
> [ConsumerFetcherManager-1400298605202] Stopping leader finder thread
> [exec] 03:50:07.165 [ZkClient-EventThread-48-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
> [***_meo01055-1400298605173-eed8b129-leader-finder-thread], Shutting down
> [exec] 03:50:07.165 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
> sessionid:0x146084cfde40000 type:create cxid:0x4d zxid:0x38 txntype:1
> reqpath:n/a
> [exec] 03:50:07.170
> [***_meo01055-1400298605173-eed8b129-leader-finder-thread] [34mINFO [0;39m
> [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
> [***_meo01055-1400298605173-eed8b129-leader-finder-thread], Stopped
> [exec] 03:50:07.170 [ZkClient-EventThread-48-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
> [***_meo01055-1400298605173-eed8b129-leader-finder-thread], Shutdown
> completed
> [exec] 03:50:07.171 [SyncThread:0] [39mDEBUG [0;39m
> [1;35mo.a.z.server.FinalRequestProcessor [0;39m -
> sessionid:0x146084cfde40000 type:create cxid:0x4d zxid:0x38 txntype:1
> reqpath:n/a
> [exec] 03:50:07.171 [ZkClient-EventThread-48-localhost:2181] [34mINFO
> [0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
> [ConsumerFetcherManager-1400298605202] Stopping all fetchers
> [exec] 03:50:07.172 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
> [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
> sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
> finished:false header:: 77,1 replyHeader:: 77,56,0 request::
>
> '/brokers/topics/ENRICHED/partitions/3/state,#7b2022636f6e74726f6c6c65725f65706f6368223a312c2022697372223a5b2030205d2c20226c6561646572223a302c20226c65616465725f65706f6368223a302c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
> response:: '/brokers/topics/ENRICHED/partitions/3/state
> [exec] 03:50:07.173 [ZkClient-EventThread-48-localhost:2181] [34mINFO
> [0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
> [ConsumerFetcherManager-1400298605202] All connections stopped
> [exec] 03:50:07.173 [ZkClient-EventThread-48-localhost:2181] [34mINFO
> [0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
> [***_meo01055-1400298605173-eed8b129], Cleared all relevant queues for this
> fetcher
> [exec] 03:50:07.173 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
> data chunk for this consumer iterator
> [exec] 03:50:07.174 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
> [0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
> machine on Controller 0]: Live assigned replicas for partition [ENRICHED,2]
> are: [List(0)]
> [exec] 03:50:07.174 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
> data chunk for this consumer iterator
> [exec] 03:50:07.174 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
> [0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
> data chunk for this consumer iterator
>
> The CorrelationId is increasing until we killed the build. (we have a
> mostly running deployment, the above failure is from integration test)
>
> Best Regards,
> Mingtao
>
>
> On Mon, May 19, 2014 at 2:06 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Mingtao,
> >
> > 1. Do you see any error/warn log before this entry?
> > 2. The producer will re-try on sending when the previous trial is not
> > successful. In your case it has exhausted all the retries and hence
> > messages are dropped on this floor and lost.
> >
> > Guozhang
> >
> >
> > On Mon, May 19, 2014 at 8:32 AM, Mingtao Zhang <vansriddles@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > I got this from the log:
> > >
> > > [exec] 03:50:06.216 [ProducerSendThread-] [1;31mERROR [0;39m
> > > [1;35mk.producer.async.ProducerSendThread [0;39m - Error in handling
> > batch
> > > of 100 events
> > > [exec] kafka.common.FailedToSendMessageException: Failed to send
> > > messages after 3 tries.
> > > [exec] at kafka.producer.async.DefaultEventHandler.handle(Unknown
> > > Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
> > > [exec] at
> > kafka.producer.async.ProducerSendThread.tryToHandle(Unknown
> > > Source) [kafka_2.10-0.8.0.jar:0.8.0]
> > > [exec] at
> > >
> > >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
> > > Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
> > > [exec] at
> > >
> > >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
> > > Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
> > > [exec] at
> > scala.collection.immutable.Stream.foreach(Stream.scala:547)
> > > ~[scala-library-2.10.1.jar:na]
> > > [exec] at
> > > kafka.producer.async.ProducerSendThread.processEvents(Unknown Source)
> > > [kafka_2.10-0.8.0.jar:0.8.0]
> > > [exec] at kafka.producer.async.ProducerSendThread.run(Unknown
> > Source)
> > > [kafka_2.10-0.8.0.jar:0.8.0]
> > >
> > > My question is:
> > >
> > > 1. What could cause this?
> > > 2. Who should deal with the recovery? User or Kafka?
> > >
> > > Let me know if more log is needed. (It hangs the integration test, so I
> > > have 3.5G of them ... from a maven build)
> > >
> > > --
> > >
> > > Best Regards,
> > > Mingtao Zhang
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
>
> Best Regards,
> Mingtao Zhang
>
--
-- Guozhang
Re: Cause and Recovery
Posted by Mingtao Zhang <va...@gmail.com>.
Hi, Guozhang,
I copied a portion of the log file before the failure:
[exec] 03:50:05.809
[***_meo01055-1400298605173-eed8b129-leader-finder-thread] [34mINFO [0;39m
[1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
[***_meo01055-1400298605173-eed8b129-leader-finder-thread], Starting
[exec] 03:50:05.810 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
id:0,host:localhost,port:9092 with correlation id 4 for 1 topic(s) Set(RAW)
[exec] 03:50:05.811 [kafka-acceptor] [39mDEBUG [0;39m
[1;35mkafka.network.Acceptor [0;39m - Accepted connection from /127.0.0.1on /
127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
recvBufferSize [actual|requested]: [1048576|1048576]
[exec] 03:50:05.812 [kafka-processor-9092-1] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Processor 1 listening to new
connection from /127.0.0.1:23035
[exec] 03:50:05.812 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
= 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
102400 (requested 102400).
[exec] 03:50:05.812 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
producing
[exec] 03:50:05.814 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getChildren cxid:0x2d
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.814 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getChildren cxid:0x2d
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.815 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 45,8 replyHeader:: 45,30,0 request::
'/brokers/ids,T response:: v{'0}
[exec] 03:50:05.816 [kafka-request-handler-0] [39mDEBUG [0;39m
[1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
RAW is Map(0 -> List(0))
[exec] 03:50:05.817 [kafka-request-handler-0] [34mINFO [0;39m
[1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
"0":[ 0 ] }, "version":1 }
[exec] 03:50:05.817 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40000 type:create
cxid:0x2e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
/brokers/topics/RAW
[exec] 03:50:05.818 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x2e zxid:0x1f txntype:-1
reqpath:n/a
[exec] 03:50:05.820 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 46,1 replyHeader:: 46,31,-110 request::
'/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response::
[exec] 03:50:05.821 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from localhost:9092
[exec] 03:50:05.822 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
1 topic(s) Set(RAW)
[exec] 03:50:05.822 [kafka-processor-9092-1] [34mINFO [0;39m
[1;35mkafka.network.Processor [0;39m - Closing socket connection to /
127.0.0.1.
[exec] 03:50:05.822 [kafka-processor-9092-1] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Closing connection from /
127.0.0.1:23035
[exec] 03:50:05.822 [ProducerSendThread-] [31mWARN [0;39m
[1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
metadata [{TopicMetadata for topic RAW ->
[exec] No partition metadata for topic RAW due to
kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
kafka.common.LeaderNotAvailableException
[exec] 03:50:05.823 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.producer.BrokerPartitionInfo [0;39m - Getting broker partition
info for topic RAW
[exec] 03:50:05.824 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
id:0,host:localhost,port:9092 with correlation id 5 for 1 topic(s) Set(RAW)
[exec] 03:50:05.825 [kafka-acceptor] [39mDEBUG [0;39m
[1;35mkafka.network.Acceptor [0;39m - Accepted connection from /127.0.0.1on /
127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
recvBufferSize [actual|requested]: [1048576|1048576]
[exec] 03:50:05.825 [kafka-processor-9092-0] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Processor 0 listening to new
connection from /127.0.0.1:23036
[exec] 03:50:05.826 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
= 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
102400 (requested 102400).
[exec] 03:50:05.826 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
producing
[exec] 03:50:05.828 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getChildren cxid:0x2f
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.828 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getChildren cxid:0x2f
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.828 [***-pool-0] [39mDEBUG [0;39m
[1;35mConsumerRunnableBuilder [0;39m - Consumption started
[exec] 03:50:05.830 [***-pool-1] [39mDEBUG [0;39m
[1;35mConsumerRunnableBuilder [0;39m - Consumption started
[exec] 03:50:05.829 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 47,8 replyHeader:: 47,31,0 request::
'/brokers/ids,T response:: v{'0}
[exec] 03:50:05.834 [kafka-request-handler-1] [39mDEBUG [0;39m
[1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
RAW is Map(0 -> List(0))
[exec] 03:50:05.834 [***-pool-2] [39mDEBUG [0;39m
[1;35mConsumerRunnableBuilder [0;39m - Consumption started
[exec] 03:50:05.834 [kafka-request-handler-1] [34mINFO [0;39m
[1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
"0":[ 0 ] }, "version":1 }
[exec] 03:50:05.835 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40000 type:create
cxid:0x30 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
/brokers/topics/RAW
[exec] 03:50:05.836 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x30 zxid:0x20 txntype:-1
reqpath:n/a
[exec] 03:50:05.838 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 48,1 replyHeader:: 48,32,-110 request::
'/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response::
[exec] 03:50:05.839 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from localhost:9092
[exec] 03:50:05.840 [kafka-processor-9092-0] [34mINFO [0;39m
[1;35mkafka.network.Processor [0;39m - Closing socket connection to /
127.0.0.1.
[exec] 03:50:05.840 [kafka-processor-9092-0] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Closing connection from /
127.0.0.1:23036
[exec] 03:50:05.849 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
1 topic(s) Set(RAW)
[exec] 03:50:05.850 [ProducerSendThread-] [31mWARN [0;39m
[1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
metadata [{TopicMetadata for topic RAW ->
[exec] No partition metadata for topic RAW due to
kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
kafka.common.LeaderNotAvailableException
[exec] 03:50:05.850 [ProducerSendThread-] [1;31mERROR [0;39m
[1;35mk.producer.async.DefaultEventHandler [0;39m - Failed to collate
messages by topic, partition due to: Failed to fetch topic metadata for
topic: RAW
[exec] 03:50:05.851 [ProducerSendThread-] [34mINFO [0;39m
[1;35mk.producer.async.DefaultEventHandler [0;39m - Back off for 100 ms
before retrying send. Remaining retries = 1
[exec] 03:50:05.853 [***-pool-3] [39mDEBUG [0;39m
[1;35mConsumerRunnableBuilder [0;39m - Consumption started
[exec] 03:50:05.871 [kafka-logflusher-1] [39mDEBUG [0;39m
[1;35mkafka.log.LogManager [0;39m - [Log Manager on Broker 0] Checking for
dirty logs to flush...
[exec] 03:50:05.872 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:getChildren cxid:0x1a
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.872 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40001 type:getChildren cxid:0x1a
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.873 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 26,8 replyHeader:: 26,32,0 request::
'/brokers/ids,F response:: v{'0}
[exec] 03:50:05.874 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:getDate cxid:0x1b zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/ids/0
[exec] 03:50:05.874 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40001 type:getDate cxid:0x1b zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/ids/0
[exec] 03:50:05.877 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 27,4 replyHeader:: 27,32,0 request::
'/brokers/ids/0,F response::
#7b2022686f7374223a226d656f30313035352e75732e6f7261636c652e636f6d222c20226a6d785f706f7274223a2d312c2022706f7274223a393039322c202274696d657374616d70223a2231343030323938363033313237222c202276657273696f6e223a31207d,s{6,6,1400298603202,1400298603202,0,0,0,91769969177788416,105,0,6}
[exec] 03:50:05.908 [ZkClient-EventThread-24-localhost:2181] [34mINFO
[0;39m [1;35mkafka.controller.ReplicaStateMachine [0;39m - [Replica state
machine on controller 0]: Invoking state change to NewReplica for replicas
PartitionAndReplica(RAW,0,0)
[exec] 03:50:05.918 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x31 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/RAW/partitions/0/state
[exec] 03:50:05.918 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x31 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/RAW/partitions/0/state
[exec] 03:50:05.919 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 49,4 replyHeader:: 49,32,-101 request::
'/brokers/topics/RAW/partitions/0/state,F response::
[exec] 03:50:05.920 [ZkClient-EventThread-24-localhost:2181] [34mINFO
[0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
machine on Controller 0]: Invoking state change to OnlinePartition for
partitions [RAW,0]
[exec] 03:50:05.923 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
machine on Controller 0]: Live assigned replicas for partition [RAW,0] are:
[List(0)]
[exec] 03:50:05.926 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
machine on Controller 0]: Initializing leader and isr for partition [RAW,0]
to (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
[exec] 03:50:05.930 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkConnection [0;39m - Creating new ZookKeeper
instance to connect to localhost:2181.
[exec] 03:50:05.930 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40000 type:create
cxid:0x32 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/RAW/partitions/0 Error:KeeperErrorCode = NoNode for
/brokers/topics/RAW/partitions/0
[exec] 03:50:05.930 [ImmediateThread-1400298600406] [34mINFO [0;39m
[1;35morg.apache.zookeeper.ZooKeeper [0;39m - Initiating client connection,
connectString=localhost:2181 sessionTimeout=30000
watcher=org.I0Itec.zkclient.ZkClient@10a86662
[exec] 03:50:05.931 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x32 zxid:0x21 txntype:-1
reqpath:n/a
[exec] 03:50:05.933 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 50,1 replyHeader:: 50,33,-101 request::
'/brokers/topics/RAW/partitions/0/state,#7b2022636f6e74726f6c6c65725f65706f6368223a312c2022697372223a5b2030205d2c20226c6561646572223a302c20226c65616465725f65706f6368223a302c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response::
[exec] 03:50:05.938 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40000 type:create
cxid:0x33 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/RAW/partitions Error:KeeperErrorCode = NoNode for
/brokers/topics/RAW/partitions
[exec] 03:50:05.938 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x33 zxid:0x22 txntype:-1
reqpath:n/a
[exec] 03:50:05.940 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 51,1 replyHeader:: 51,34,-101 request::
'/brokers/topics/RAW/partitions/0,,v{s{31,s{'world,'anyone}}},0 response::
[exec] 03:50:05.940 [ZkClient-EventThread-58-localhost:2181] [34mINFO
[0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Starting ZkClient
event thread.
[exec] 03:50:05.941 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x34 zxid:0x23 txntype:1
reqpath:n/a
[exec] 03:50:05.941 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:create cxid:0x34 zxid:0x23 txntype:1
reqpath:n/a
[exec] 03:50:05.943 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 52,1 replyHeader:: 52,35,0 request::
'/brokers/topics/RAW/partitions,,v{s{31,s{'world,'anyone}}},0 response::
'/brokers/topics/RAW/partitions
[exec] 03:50:05.944 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x35 zxid:0x24 txntype:1
reqpath:n/a
[exec] 03:50:05.955 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:create cxid:0x35 zxid:0x24 txntype:1
reqpath:n/a
[exec] 03:50:05.956 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 53,1 replyHeader:: 53,36,0 request::
'/brokers/topics/RAW/partitions/0,,v{s{31,s{'world,'anyone}}},0 response::
'/brokers/topics/RAW/partitions/0
[exec] 03:50:05.958 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
id:0,host:localhost,port:9092 with correlation id 6 for 1 topic(s) Set(RAW)
[exec] 03:50:05.958 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mk.producer.async.ProducerSendThread [0;39m - 1001 ms elapsed. Queue
time reached. Sending..
[exec] 03:50:05.958 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mk.producer.async.ProducerSendThread [0;39m - Handling 0 events
[exec] 03:50:05.959 [kafka-acceptor] [39mDEBUG [0;39m
[1;35mkafka.network.Acceptor [0;39m - Accepted connection from /127.0.0.1on /
127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
recvBufferSize [actual|requested]: [1048576|1048576]
[exec] 03:50:05.959 [kafka-processor-9092-1] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Processor 1 listening to new
connection from /127.0.0.1:23037
[exec] 03:50:05.960 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x36 zxid:0x25 txntype:1
reqpath:n/a
[exec] 03:50:05.960 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
= 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
102400 (requested 102400).
[exec] 03:50:05.961 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
producing
[exec] 03:50:05.974 [ImmediateThread-1400298600406-SendThread()]
[34mINFO [0;39m [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Opening
socket connection to server localhost/127.0.0.1:2181
[exec] 03:50:05.975 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
[34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Accepted
socket connection from /127.0.0.1:59397
[exec] 03:50:05.982 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Awaiting connection to
Zookeeper server
[exec] 03:50:05.982 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Waiting for keeper state
SyncConnected
[exec] 03:50:05.987 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:create cxid:0x36 zxid:0x25 txntype:1
reqpath:n/a
[exec] 03:50:05.987 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getChildren cxid:0x37
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.987 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getChildren cxid:0x37
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.988
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [34mINFO [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Socket connection
established to localhost/127.0.0.1:2181, initiating session
[exec] 03:50:05.988
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Session establishment
request sent on localhost/127.0.0.1:2181
[exec] 03:50:05.988 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
[39mDEBUG [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Session
establishment request from client /127.0.0.1:59397 client's lastZxid is 0x0
[exec] 03:50:05.989 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
[34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Client
attempting to establish new session at /127.0.0.1:59397
[exec] 03:50:05.990 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40003 type:createSession cxid:0x0 zxid:0x26
txntype:-10 reqpath:n/a
[exec] 03:50:05.990 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40003 type:createSession cxid:0x0 zxid:0x26
txntype:-10 reqpath:n/a
[exec] 03:50:05.991 [SyncThread:0] [34mINFO [0;39m
[1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Established session
0x146084cfde40003 with negotiated timeout 30000 for client /127.0.0.1:59397
[exec] 03:50:05.991
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [34mINFO [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Session establishment
complete on server localhost/127.0.0.1:2181, sessionid = 0x146084cfde40003,
negotiated timeout = 30000
[exec] 03:50:05.992 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 54,1 replyHeader:: 54,37,0 request::
'/brokers/topics/RAW/partitions/0/state,#7b2022636f6e74726f6c6c65725f65706f6368223a312c2022697372223a5b2030205d2c20226c6561646572223a302c20226c65616465725f65706f6368223a302c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response:: '/brokers/topics/RAW/partitions/0/state
[exec] 03:50:05.993 [ImmediateThread-1400298600406-EventThread]
[39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Received
event: WatchedEvent state:SyncConnected type:None path:null
[exec] 03:50:05.993 [ImmediateThread-1400298600406-EventThread]
[34mINFO [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - zookeeper state
changed (SyncConnected)
[exec] 03:50:05.994 [ImmediateThread-1400298600406-EventThread]
[39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Leaving
process event
[exec] 03:50:05.994 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - State is SyncConnected
[exec] 03:50:05.995 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40003 type:getChildren cxid:0x1
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.995 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40003 type:getChildren cxid:0x1
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:05.996
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40003, packet:: clientPath:null serverPath:null
finished:false header:: 1,8 replyHeader:: 1,38,0 request::
'/brokers/ids,F response:: v{'0}
[exec] 03:50:05.997 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
RAW is Map(2 -> List(0), 1 -> List(0), 3 -> List(0), 0 -> List(0))
[exec] 03:50:05.998 [ImmediateThread-1400298600406] [34mINFO [0;39m
[1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
"0":[ 0 ], "1":[ 0 ], "2":[ 0 ], "3":[ 0 ] }, "version":1 }
[exec] 03:50:05.999 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 55,8 replyHeader:: 55,37,0 request::
'/brokers/ids,T response:: v{'0}
[exec] 03:50:05.999 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40003 type:create
cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
/brokers/topics/RAW
[exec] 03:50:05.999 [kafka-request-handler-0] [39mDEBUG [0;39m
[1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
RAW is Map(0 -> List(0))
[exec] 03:50:06.000 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40003 type:create cxid:0x2 zxid:0x27 txntype:-1
reqpath:n/a
[exec] 03:50:06.000 [kafka-request-handler-0] [34mINFO [0;39m
[1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
"0":[ 0 ] }, "version":1 }
[exec] 03:50:06.002
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40003, packet:: clientPath:null serverPath:null
finished:false header:: 2,1 replyHeader:: 2,39,-110 request::
'/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response::
[exec] 03:50:06.002 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:getDate cxid:0x1c zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/RAW
[exec] 03:50:06.003 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40001 type:getDate cxid:0x1c zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/RAW
[exec] 03:50:06.005 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 28,4 replyHeader:: 28,39,0 request::
'/brokers/topics/RAW,T response::
#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,s{27,27,1400298605448,1400298605448,0,1,0,0,43,1,35}
[exec] 03:50:06.006 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40000 type:create
cxid:0x38 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
/brokers/topics/RAW
[exec] 03:50:06.007 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x38 zxid:0x28 txntype:-1
reqpath:n/a
[exec] 03:50:06.008 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 56,1 replyHeader:: 56,40,-110 request::
'/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response::
[exec] 03:50:06.010 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from localhost:9092
[exec] 03:50:06.010 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
1 topic(s) Set(RAW)
[exec] 03:50:06.010 [kafka-processor-9092-1] [34mINFO [0;39m
[1;35mkafka.network.Processor [0;39m - Closing socket connection to /
127.0.0.1.
[exec] 03:50:06.011 [kafka-processor-9092-1] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Closing connection from /
127.0.0.1:23037
[exec] 03:50:06.011 [ProducerSendThread-] [31mWARN [0;39m
[1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
metadata [{TopicMetadata for topic RAW ->
[exec] No partition metadata for topic RAW due to
kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
kafka.common.LeaderNotAvailableException
[exec] 03:50:06.012 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.producer.BrokerPartitionInfo [0;39m - Getting broker partition
info for topic RAW
[exec] 03:50:06.012 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
id:0,host:localhost,port:9092 with correlation id 7 for 1 topic(s) Set(RAW)
[exec] 03:50:06.013 [kafka-acceptor] [39mDEBUG [0;39m
[1;35mkafka.network.Acceptor [0;39m - Accepted connection from /127.0.0.1on /
127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
recvBufferSize [actual|requested]: [1048576|1048576]
[exec] 03:50:06.014 [kafka-processor-9092-0] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Processor 0 listening to new
connection from /127.0.0.1:23039
[exec] 03:50:06.015 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
= 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
102400 (requested 102400).
[exec] creation failed because of topic RAW already exists
[exec] kafka.common.TopicExistsException: topic RAW already exists
[exec] at
kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(Unknown
Source)
[exec] at kafka.admin.CreateTopicCommand$.createTopic(Unknown Source)
[exec] at kafka.admin.CreateTopicCommand$.main(Unknown Source)
[exec] at kafka.admin.CreateTopicCommand.main(Unknown Source)
[exec] at
com.***.producer.raw.internal.RawTopicCreator.create(RawTopicCreator.java:39)
[exec] at
com.***.producer.core.internal.topic.TopicPrimer.createTopics(TopicPrimer.java:35)
[exec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[exec] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[exec] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[exec] at java.lang.reflect.Method.invoke(Method.java:606)
[exec] at
org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)
[exec] at
org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388)
[exec] at
org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430)
[exec] at
org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
[exec] at
org.glassfish.hk2.internal.ImmediateHelper.findOrCreate(ImmediateHelper.java:164)
[exec] at
org.glassfish.hk2.utilities.ImmediateContext.findOrCreate(ImmediateContext.java:73)
[exec] at
org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
[exec] at
org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
[exec] at
org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
[exec] at
org.glassfish.hk2.internal.ImmediateHelper.doWork(ImmediateHelper.java:434)
[exec] at
org.glassfish.hk2.internal.ImmediateHelper.run(ImmediateHelper.java:317)
[exec] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[exec] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[exec] at java.lang.Thread.run(Thread.java:724)
[exec]
[exec] 03:50:06.019 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
producing
[exec] 03:50:06.018 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Closing ZkClient...
[exec] 03:50:06.020 [ZkClient-EventThread-58-localhost:2181] [34mINFO
[0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Terminate ZkClient
event thread.
[exec] 03:50:06.020 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getChildren cxid:0x39
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.021 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getChildren cxid:0x39
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.031 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 57,8 replyHeader:: 57,40,0 request::
'/brokers/ids,T response:: v{'0}
[exec] 03:50:06.031 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkConnection [0;39m - Closing ZooKeeper connected
to localhost:2181
[exec] 03:50:06.031 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ZooKeeper [0;39m - Closing session:
0x146084cfde40003
[exec] 03:50:06.031 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Closing client for session:
0x146084cfde40003
[exec] 03:50:06.033 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Processed session
termination for sessionid: 0x146084cfde40003
[exec] 03:50:06.035 [kafka-request-handler-1] [39mDEBUG [0;39m
[1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
RAW is Map(0 -> List(0))
[exec] 03:50:06.035 [kafka-request-handler-1] [34mINFO [0;39m
[1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
"0":[ 0 ] }, "version":1 }
[exec] 03:50:06.037 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40003 type:closeSession cxid:0x3 zxid:0x29
txntype:-11 reqpath:n/a
[exec] 03:50:06.037 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40003 type:closeSession cxid:0x3 zxid:0x29
txntype:-11 reqpath:n/a
[exec] 03:50:06.038
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40003, packet:: clientPath:null serverPath:null
finished:false header:: 3,-11 replyHeader:: 3,41,0 request:: null
response:: null
[exec] 03:50:06.039 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Disconnecting client for
session: 0x146084cfde40003
[exec] 03:50:06.039 [ImmediateThread-1400298600406] [34mINFO [0;39m
[1;35morg.apache.zookeeper.ZooKeeper [0;39m - Session: 0x146084cfde40003
closed
[exec] 03:50:06.039 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Closing ZkClient...done
[exec] 03:50:06.040 [ImmediateThread-1400298600406-EventThread]
[34mINFO [0;39m [1;35morg.apache.zookeeper.ClientCnxn [0;39m - EventThread
shut down
[exec] 03:50:06.034 [ZkClient-EventThread-24-localhost:2181] [34mINFO
[0;39m [1;35mkafka.controller.ReplicaStateMachine [0;39m - [Replica state
machine on controller 0]: Invoking state change to OnlineReplica for
replicas PartitionAndReplica(RAW,0,0)
[exec] 03:50:06.046 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Delivering event #3
done
[exec] 03:50:06.049 [kafka-request-handler-0] [34mINFO [0;39m
[1;35mkafka.server.ReplicaManager [0;39m - [Replica Manager on Broker 0]:
Handling LeaderAndIsr request
Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(RAW,0)
->
(LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:meo01055.us.***,port:9092
[exec] 03:50:06.054 [kafka-request-handler-0] [34mINFO [0;39m
[1;35mkafka.server.ReplicaFetcherManager [0;39m - [ReplicaFetcherManager on
broker 0] Removing fetcher for partition [RAW,0]
[exec] 03:50:06.056 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
[34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Closed
socket connection for client /127.0.0.1:59397 which had sessionid
0x146084cfde40003
[exec] 03:50:06.057 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40000 type:create
cxid:0x3a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
/brokers/topics/RAW
[exec] 03:50:06.055 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkConnection [0;39m - Creating new ZookKeeper
instance to connect to localhost:2181.
[exec] 03:50:06.058 [ImmediateThread-1400298600406] [34mINFO [0;39m
[1;35morg.apache.zookeeper.ZooKeeper [0;39m - Initiating client connection,
connectString=localhost:2181 sessionTimeout=30000
watcher=org.I0Itec.zkclient.ZkClient@74c824b3
[exec] 03:50:06.059 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x3a zxid:0x2a txntype:-1
reqpath:n/a
[exec] 03:50:06.061 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 58,1 replyHeader:: 58,42,-110 request::
'/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response::
[exec] 03:50:06.055 [ZkClient-EventThread-61-localhost:2181] [34mINFO
[0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Starting ZkClient
event thread.
[exec] 03:50:06.064 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from localhost:9092
[exec] 03:50:06.064 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
1 topic(s) Set(RAW)
[exec] 03:50:06.066 [ProducerSendThread-] [31mWARN [0;39m
[1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
metadata [{TopicMetadata for topic RAW ->
[exec] No partition metadata for topic RAW due to
kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
kafka.common.LeaderNotAvailableException
[exec] 03:50:06.066 [ProducerSendThread-] [1;31mERROR [0;39m
[1;35mk.producer.async.DefaultEventHandler [0;39m - Failed to collate
messages by topic, partition due to: Failed to fetch topic metadata for
topic: RAW
[exec] 03:50:06.066 [ProducerSendThread-] [34mINFO [0;39m
[1;35mk.producer.async.DefaultEventHandler [0;39m - Back off for 100 ms
before retrying send. Remaining retries = 0
[exec] 03:50:06.067 [kafka-processor-9092-0] [34mINFO [0;39m
[1;35mkafka.network.Processor [0;39m - Closing socket connection to /
127.0.0.1.
[exec] 03:50:06.068 [kafka-processor-9092-0] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Closing connection from /
127.0.0.1:23039
[exec] 03:50:06.070 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Partition map for
/brokers/topics/RAW is Map(0 -> List(0))
[exec] 03:50:06.075 [ImmediateThread-1400298600406-SendThread()]
[34mINFO [0;39m [1;35morg.apache.zookeeper.ClientCnxn [0;39m - Opening
socket connection to server localhost/127.0.0.1:2181
[exec] 03:50:06.076 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
[34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Accepted
socket connection from /127.0.0.1:59399
[exec] 03:50:06.077 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
[ConsumerFetcherManager-1400298603576] Stopping leader finder thread
[exec] 03:50:06.078 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], Shutting down
[exec] 03:50:06.086 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Awaiting connection to
Zookeeper server
[exec] 03:50:06.086 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Waiting for keeper state
SyncConnected
[exec] 03:50:06.087
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], Stopped
[exec] 03:50:06.088
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [34mINFO [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Socket connection
established to localhost/127.0.0.1:2181, initiating session
[exec] 03:50:06.089
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Session establishment
request sent on localhost/127.0.0.1:2181
[exec] 03:50:06.089 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
[39mDEBUG [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Session
establishment request from client /127.0.0.1:59399 client's lastZxid is 0x0
[exec] 03:50:06.089 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
[34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Client
attempting to establish new session at /127.0.0.1:59399
[exec] 03:50:06.090 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], Shutdown
completed
[exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
[ConsumerFetcherManager-1400298603576] Stopping all fetchers
[exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
[ConsumerFetcherManager-1400298603576] All connections stopped
[exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], Cleared all relevant queues for this
fetcher
[exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
data chunk for this consumer iterator
[exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
data chunk for this consumer iterator
[exec] 03:50:06.091 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
data chunk for this consumer iterator
[exec] 03:50:06.092 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
data chunk for this consumer iterator
[exec] 03:50:06.092 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], Cleared the data chunks in all the
consumer message iterators
[exec] 03:50:06.092 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], Committing all offsets after
clearing the fetcher queues
[exec] 03:50:06.093 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], Releasing partition ownership
[exec] 03:50:06.093 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40004 type:createSession cxid:0x0 zxid:0x2b
txntype:-10 reqpath:n/a
[exec] 03:50:06.093 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40004 type:createSession cxid:0x0 zxid:0x2b
txntype:-10 reqpath:n/a
[exec] 03:50:06.097 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], Consumer
RAW_meo01055-1400298603522-1cb22725 rebalancing the following partitions:
ArrayBuffer(0) for topic RAW with consumers:
List(RAW_meo01055-1400298603522-1cb22725-0,
RAW_meo01055-1400298603522-1cb22725-1,
RAW_meo01055-1400298603522-1cb22725-2,
RAW_meo01055-1400298603522-1cb22725-3)
[exec] 03:50:06.101 [SyncThread:0] [34mINFO [0;39m
[1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Established session
0x146084cfde40004 with negotiated timeout 30000 for client /127.0.0.1:59399
[exec] 03:50:06.101
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [34mINFO [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Session establishment
complete on server localhost/127.0.0.1:2181, sessionid = 0x146084cfde40004,
negotiated timeout = 30000
[exec] 03:50:06.102 [ImmediateThread-1400298600406-EventThread]
[39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Received
event: WatchedEvent state:SyncConnected type:None path:null
[exec] 03:50:06.102 [ImmediateThread-1400298600406-EventThread]
[34mINFO [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - zookeeper state
changed (SyncConnected)
[exec] 03:50:06.102 [ImmediateThread-1400298600406-EventThread]
[39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Leaving
process event
[exec] 03:50:06.102 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - State is SyncConnected
[exec] 03:50:06.102 [ZkClient-EventThread-34-localhost:2181] [31mWARN
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], No broker partitions consumed by
consumer thread RAW_meo01055-1400298603522-1cb22725-1 for topic RAW
[exec] 03:50:06.108 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40004 type:getChildren cxid:0x1
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.109 [ZkClient-EventThread-34-localhost:2181] [31mWARN
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], No broker partitions consumed by
consumer thread RAW_meo01055-1400298603522-1cb22725-2 for topic RAW
[exec] 03:50:06.104 [kafka-request-handler-0] [39mDEBUG [0;39m
[1;35mkafka.log.FileMessageSet [0;39m - Creating or reloading log segment
/scratch/home/Buildlab/12.1/branches/12.1.4.0.3/webcentersites/siteanalytics/integration-test/consumer-itest/target/kafka-log/RAW-0/00000000000000000000.log
[exec] 03:50:06.109 [ZkClient-EventThread-34-localhost:2181] [31mWARN
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], No broker partitions consumed by
consumer thread RAW_meo01055-1400298603522-1cb22725-3 for topic RAW
[exec] 03:50:06.111 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725],
RAW_meo01055-1400298603522-1cb22725-0 attempting to claim partition 0
[exec] 03:50:06.109 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40004 type:getChildren cxid:0x1
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.113 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:getDate cxid:0x1d zxid:0xfffffffffffffffe
txntype:unknown reqpath:/consumers/RAW/offsets/RAW/0
[exec] 03:50:06.113 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40001 type:getDate cxid:0x1d zxid:0xfffffffffffffffe
txntype:unknown reqpath:/consumers/RAW/offsets/RAW/0
[exec] 03:50:06.114
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40004, packet:: clientPath:null serverPath:null
finished:false header:: 1,8 replyHeader:: 1,43,0 request::
'/brokers/ids,F response:: v{'0}
[exec] 03:50:06.114 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 29,4 replyHeader:: 29,43,-101 request::
'/consumers/RAW/offsets/RAW/0,F response::
[exec] 03:50:06.115 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
ENRICHED is Map(2 -> List(0), 1 -> List(0), 3 -> List(0), 0 -> List(0))
[exec] 03:50:06.116 [ImmediateThread-1400298600406] [34mINFO [0;39m
[1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
"0":[ 0 ], "1":[ 0 ], "2":[ 0 ], "3":[ 0 ] }, "version":1 }
[exec] 03:50:06.118 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40004 type:create cxid:0x2 zxid:0x2c txntype:1
reqpath:n/a
[exec] 03:50:06.118 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.consumer.PartitionTopicInfo [0;39m - initial consumer
offset of RAW:0: fetched offset = -1: consumed offset = -1 is -1
[exec] 03:50:06.119 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Got notification
sessionid:0x146084cfde40002
[exec] 03:50:06.119 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Got WatchedEvent
state:SyncConnected type:NodeCreated path:/brokers/topics/ENRICHED for
sessionid 0x146084cfde40002
[exec] 03:50:06.119 [ConsumerServiceImpl STARTING-EventThread]
[39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Received
event: WatchedEvent state:SyncConnected type:NodeCreated
path:/brokers/topics/ENRICHED
[exec] 03:50:06.119 [ConsumerServiceImpl STARTING-EventThread]
[39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - New
event: ZkEvent[Data of /brokers/topics/ENRICHED changed sent to
kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@7703f0d2
]
[exec] 03:50:06.119 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40004 type:create cxid:0x2 zxid:0x2c txntype:1
reqpath:n/a
[exec] 03:50:06.119 [ConsumerServiceImpl STARTING-EventThread]
[39mDEBUG [0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Leaving
process event
[exec] 03:50:06.120 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
[0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Delivering event #5
ZkEvent[Data of /brokers/topics/ENRICHED changed sent to
kafka.consumer.ZookeeperConsumerConnector$ZKTopicPartitionChangeListener@7703f0d2
]
[exec] 03:50:06.120 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Got notification
sessionid:0x146084cfde40000
[exec] 03:50:06.120 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Got WatchedEvent
state:SyncConnected type:NodeChildrenChanged path:/brokers/topics for
sessionid 0x146084cfde40000
[exec] 03:50:06.121 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.consumer.PartitionTopicInfo [0;39m - initial fetch
offset of RAW:0: fetched offset = -1: consumed offset = -1 is -1
[exec] 03:50:06.121 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:exists cxid:0x11 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.122 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:exists cxid:0x11 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.123
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40004, packet:: clientPath:null serverPath:null
finished:false header:: 2,1 replyHeader:: 2,44,0 request::
'/brokers/topics/ENRICHED,#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response:: '/brokers/topics/ENRICHED
[exec] 03:50:06.123 [main-EventThread] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Received event: WatchedEvent
state:SyncConnected type:NodeChildrenChanged path:/brokers/topics
[exec] 03:50:06.123 [main-EventThread] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - New event:
ZkEvent[Children of /brokers/topics changed sent to
kafka.controller.PartitionStateMachine$TopicChangeListener@365b27dc]
[exec] 03:50:06.123 [main-EventThread] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Leaving process event
[exec] 03:50:06.123 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Delivering event #6
ZkEvent[Children of /brokers/topics changed sent to
kafka.controller.PartitionStateMachine$TopicChangeListener@365b27dc]
[exec] 03:50:06.124 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:exists cxid:0x3b zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics
[exec] 03:50:06.124 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:exists cxid:0x3b zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics
[exec] 03:50:06.124 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 17,3 replyHeader:: 17,44,0 request::
'/brokers/topics/ENRICHED,T response::
s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
[exec] 03:50:06.125 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 59,3 replyHeader:: 59,44,0 request::
'/brokers/topics,T response::
s{26,26,1400298605436,1400298605436,0,2,0,0,0,2,44}
[exec] 03:50:06.118 [kafka-request-handler-0] [39mDEBUG [0;39m
[1;35mkafka.log.OffsetIndex [0;39m - Loaded index file
/scratch/home/Buildlab/12.1/branches/12.1.4.0.3/webcentersites/siteanalytics/integration-test/consumer-itest/target/kafka-log/RAW-0/00000000000000000000.index
with maxEntries = 1310720, maxIndexSize = 10485760, entries = 0, lastOffset
= 0, file position = 0
[exec] 03:50:06.126 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35mkafka.admin.AdminUtils$ [0;39m - Updated path
/brokers/topics/ENRICHED with { "partitions":{ "0":[ 0 ], "1":[ 0 ], "2":[
0 ], "3":[ 0 ] }, "version":1 } for replica assignment
[exec] creation succeeded!
[exec] 03:50:06.126 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Closing ZkClient...
[exec] 03:50:06.127 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:getDate cxid:0x12 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.127 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:getDate cxid:0x12 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.127 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getChildren cxid:0x3c
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/topics
[exec] 03:50:06.127 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getChildren cxid:0x3c
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/topics
[exec] 03:50:06.129 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 60,8 replyHeader:: 60,44,0 request::
'/brokers/topics,T response:: v{'RAW,'ENRICHED}
[exec] 03:50:06.129 [ZkClient-EventThread-61-localhost:2181] [34mINFO
[0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Terminate ZkClient
event thread.
[exec] 03:50:06.133 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 18,4 replyHeader:: 18,44,0 request::
'/brokers/topics/ENRICHED,T response::
#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
[exec] 03:50:06.133 [ZkClient-EventThread-48-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[***_meo01055-1400298605173-eed8b129], Topic info for path
/brokers/topics/ENRICHED changed to { "partitions":{ "0":[ 0 ], "1":[ 0 ],
"2":[ 0 ], "3":[ 0 ] }, "version":1 }, triggering rebalance
[exec] 03:50:06.133 [ZkClient-EventThread-48-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[***_meo01055-1400298605173-eed8b129], begin rebalancing consumer
***_meo01055-1400298605173-eed8b129 try #0
[exec] 03:50:06.134 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkConnection [0;39m - Closing ZooKeeper connected
to localhost:2181
[exec] 03:50:06.134 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ZooKeeper [0;39m - Closing session:
0x146084cfde40004
[exec] 03:50:06.134 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Closing client for session:
0x146084cfde40004
[exec] 03:50:06.135 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mk.c.PartitionStateMachine$TopicChangeListener [0;39m -
[TopicChangeListener on Controller 0]: Topic change listener fired for path
/brokers/topics with children RAW,ENRICHED
[exec] 03:50:06.136 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x3d zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.136 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x3d zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.137 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Processed session
termination for sessionid: 0x146084cfde40004
[exec] 03:50:06.138 [kafka-request-handler-0] [34mINFO [0;39m
[1;35mkafka.log.Log [0;39m - [Kafka Log on Broker 0], Completed load of log
RAW-0 with log end offset 0
[exec] 03:50:06.139 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 61,4 replyHeader:: 61,44,0 request::
'/brokers/topics/ENRICHED,F response::
#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
[exec] 03:50:06.139 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40004 type:closeSession cxid:0x3 zxid:0x2d
txntype:-11 reqpath:n/a
[exec] 03:50:06.140 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40004 type:closeSession cxid:0x3 zxid:0x2d
txntype:-11 reqpath:n/a
[exec] 03:50:06.140 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:getChildren cxid:0x13
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.140 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:getChildren cxid:0x13
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.141
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40004, packet:: clientPath:null serverPath:null
finished:false header:: 3,-11 replyHeader:: 3,45,0 request:: null
response:: null
[exec] 03:50:06.141 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
[34mINFO [0;39m [1;35mo.a.zookeeper.server.NIOServerCnxn [0;39m - Closed
socket connection for client /127.0.0.1:59399 which had sessionid
0x146084cfde40004
[exec] 03:50:06.141 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Disconnecting client for
session: 0x146084cfde40004
[exec] 03:50:06.142 [ImmediateThread-1400298600406] [34mINFO [0;39m
[1;35morg.apache.zookeeper.ZooKeeper [0;39m - Session: 0x146084cfde40004
closed
[exec] 03:50:06.142 [ImmediateThread-1400298600406] [39mDEBUG [0;39m
[1;35morg.I0Itec.zkclient.ZkClient [0;39m - Closing ZkClient...done
[exec] 03:50:06.143 [ImmediateThread-1400298600406-EventThread]
[34mINFO [0;39m [1;35morg.apache.zookeeper.ClientCnxn [0;39m - EventThread
shut down
[exec] 03:50:06.150 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 19,8 replyHeader:: 19,45,0 request::
'/brokers/ids,F response:: v{'0}
[exec] 03:50:06.151 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], RAW:0: fetched offset = -1: consumed
offset = -1 selected new offset -1
[exec] 03:50:06.152 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:getDate cxid:0x14 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/ids/0
[exec] 03:50:06.152 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:getDate cxid:0x14 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/ids/0
[exec] 03:50:06.152 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40001 type:create
cxid:0x1e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/consumers/RAW/owners/RAW Error:KeeperErrorCode = NoNode for
/consumers/RAW/owners/RAW
[exec] 03:50:06.153 [kafka-request-handler-0] [34mINFO [0;39m
[1;35mkafka.log.LogManager [0;39m - [Log Manager on Broker 0] Created log
for partition [RAW,0] in
/scratch/home/Buildlab/12.1/branches/12.1.4.0.3/webcentersites/siteanalytics/integration-test/consumer-itest/target/kafka-log.
[exec] 03:50:06.154 [kafka-request-handler-0] [31mWARN [0;39m
[1;35mkafka.server.HighwaterMarkCheckpoint [0;39m - No highwatermark file
is found. Returning 0 as the highwatermark for partition [RAW,0]
[exec] 03:50:06.155 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 20,4 replyHeader:: 20,45,0 request::
'/brokers/ids/0,F response::
#7b2022686f7374223a226d656f30313035352e75732e6f7261636c652e636f6d222c20226a6d785f706f7274223a2d312c2022706f7274223a393039322c202274696d657374616d70223a2231343030323938363033313237222c202276657273696f6e223a31207d,s{6,6,1400298603202,1400298603202,0,0,0,91769969177788416,105,0,6}
[exec] 03:50:06.156
[ImmediateThread-1400298600406-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - An exception was thrown
while closing send thread for session 0x146084cfde40004 : Unable to read
additional data from server sessionid 0x146084cfde40004, likely server has
closed socket
[exec] 03:50:06.159 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:create cxid:0x1e zxid:0x2e txntype:-1
reqpath:n/a
[exec] 03:50:06.160 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 30,1 replyHeader:: 30,46,-101 request::
'/consumers/RAW/owners/RAW/0,#5241575f6d656f30313035352d313430303239383630333532322d31636232323732352d30,v{s{31,s{'world,'anyone}}},1
response::
[exec] 03:50:06.161 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40001 type:create
cxid:0x1f zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/consumers/RAW/owners Error:KeeperErrorCode = NoNode for
/consumers/RAW/owners
[exec] 03:50:06.163 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:create cxid:0x1f zxid:0x2f txntype:-1
reqpath:n/a
[exec] 03:50:06.165 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 31,1 replyHeader:: 31,47,-101 request::
'/consumers/RAW/owners/RAW,,v{s{31,s{'world,'anyone}}},0 response::
[exec] 03:50:06.168 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
id:0,host:localhost,port:9092 with correlation id 8 for 1 topic(s) Set(RAW)
[exec] 03:50:06.169 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:create cxid:0x20 zxid:0x30 txntype:1
reqpath:n/a
[exec] 03:50:06.169 [kafka-acceptor] [39mDEBUG [0;39m
[1;35mkafka.network.Acceptor [0;39m - Accepted connection from /127.0.0.1on /
127.0.0.1:9092. sendBufferSize [actual|requested]: [1048576|1048576]
recvBufferSize [actual|requested]: [1048576|1048576]
[exec] 03:50:06.169 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40001 type:create cxid:0x20 zxid:0x30 txntype:1
reqpath:n/a
[exec] 03:50:06.170 [kafka-processor-9092-1] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Processor 1 listening to new
connection from /127.0.0.1:23041
[exec] 03:50:06.170 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
= 10000 (requested 10000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
102400 (requested 102400).
[exec] 03:50:06.170 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Connected to localhost:9092 for
producing
[exec] 03:50:06.172 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getChildren cxid:0x3e
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.172 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getChildren cxid:0x3e
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.173 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 62,8 replyHeader:: 62,48,0 request::
'/brokers/ids,T response:: v{'0}
[exec] 03:50:06.174 [kafka-request-handler-1] [39mDEBUG [0;39m
[1;35mkafka.admin.CreateTopicCommand$ [0;39m - Replica assignment list for
RAW is Map(0 -> List(0))
[exec] 03:50:06.174 [kafka-request-handler-1] [34mINFO [0;39m
[1;35mkafka.admin.AdminUtils$ [0;39m - Topic creation { "partitions":{
"0":[ 0 ] }, "version":1 }
[exec] 03:50:06.175 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40000 type:create
cxid:0x3f zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/RAW Error:KeeperErrorCode = NodeExists for
/brokers/topics/RAW
[exec] 03:50:06.176 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 32,1 replyHeader:: 32,48,0 request::
'/consumers/RAW/owners,,v{s{31,s{'world,'anyone}}},0 response::
'/consumers/RAW/owners
[exec] 03:50:06.176 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x3f zxid:0x31 txntype:-1
reqpath:n/a
[exec] 03:50:06.178 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 63,1 replyHeader:: 63,49,-110 request::
'/brokers/topics/RAW,#7b2022706172746974696f6e73223a7b202230223a5b2030205d207d2c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response::
[exec] 03:50:06.179 [ProducerSendThread-] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from localhost:9092
[exec] 03:50:06.179 [kafka-processor-9092-1] [34mINFO [0;39m
[1;35mkafka.network.Processor [0;39m - Closing socket connection to /
127.0.0.1.
[exec] 03:50:06.180 [kafka-processor-9092-1] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Closing connection from /
127.0.0.1:23041
[exec] 03:50:06.179 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
1 topic(s) Set(RAW)
[exec] 03:50:06.180 [ProducerSendThread-] [31mWARN [0;39m
[1;35mkafka.producer.BrokerPartitionInfo [0;39m - Error while fetching
metadata [{TopicMetadata for topic RAW ->
[exec] No partition metadata for topic RAW due to
kafka.common.LeaderNotAvailableException}] for topic [RAW]: class
kafka.common.LeaderNotAvailableException
[exec] 03:50:06.183 [ProducerSendThread-] [1;31mERROR [0;39m
[1;35mk.producer.async.DefaultEventHandler [0;39m - Failed to send requests
for topics RAW with correlation ids in [0,8]
[exec] 03:50:06.183 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:create cxid:0x21 zxid:0x32 txntype:1
reqpath:n/a
[exec] 03:50:06.198 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40001 type:create cxid:0x21 zxid:0x32 txntype:1
reqpath:n/a
[exec] 03:50:06.200 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 33,1 replyHeader:: 33,50,0 request::
'/consumers/RAW/owners/RAW,,v{s{31,s{'world,'anyone}}},0 response::
'/consumers/RAW/owners/RAW
[exec] 03:50:06.201 [kafka-request-handler-0] [39mDEBUG [0;39m
[1;35mkafka.cluster.Partition [0;39m - Partition [RAW,0] on broker 0: Old
hw for partition [RAW,0] is 0. New hw is 0. All leo's are 0
[exec] 03:50:06.206 [kafka-request-handler-0] [34mINFO [0;39m
[1;35mkafka.server.ReplicaManager [0;39m - [Replica Manager on Broker 0]:
Handled leader and isr request
Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:6;ClientId:id_0-host_null-port_9092;PartitionState:(RAW,0)
->
(LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:meo01055.us.***,port:9092
[exec] 03:50:06.208 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:create cxid:0x22 zxid:0x33 txntype:1
reqpath:n/a
[exec] 03:50:06.209 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40001 type:create cxid:0x22 zxid:0x33 txntype:1
reqpath:n/a
[exec] 03:50:06.210 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 34,1 replyHeader:: 34,51,0 request::
'/consumers/RAW/owners/RAW/0,#5241575f6d656f30313035352d313430303239383630333532322d31636232323732352d30,v{s{31,s{'world,'anyone}}},1
response:: '/consumers/RAW/owners/RAW/0
[exec] 03:50:06.215 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725],
RAW_meo01055-1400298603522-1cb22725-0 successfully owned partition 0 for
topic RAW
[exec] 03:50:06.215 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], Updating the cache
[exec] 03:50:06.215 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], Partitions per topic cache Map(RAW
-> ArrayBuffer(0))
[exec] 03:50:06.216 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], Consumers per topic cache Map(RAW ->
List(RAW_meo01055-1400298603522-1cb22725-0,
RAW_meo01055-1400298603522-1cb22725-1,
RAW_meo01055-1400298603522-1cb22725-2,
RAW_meo01055-1400298603522-1cb22725-3))
[exec] 03:50:06.216 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], Consumer
RAW_meo01055-1400298603522-1cb22725 selected partitions : RAW:0: fetched
offset = -1: consumed offset = -1
After the failure, the logs looks like a 'loop':
[exec] 03:50:06.227 [ZkClient-EventThread-34-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[RAW_meo01055-1400298603522-1cb22725], end rebalancing consumer
RAW_meo01055-1400298603522-1cb22725 try #0
[exec] 03:50:06.228 [ZkClient-EventThread-34-localhost:2181] [39mDEBUG
[0;39m [1;35morg.I0Itec.zkclient.ZkEventThread [0;39m - Delivering event #4
done
[exec] 03:50:06.243
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], Starting
[exec] 03:50:06.245 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:getChildren cxid:0x23
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.245 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40001 type:getChildren cxid:0x23
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.246 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 35,8 replyHeader:: 35,51,0 request::
'/brokers/ids,F response:: v{'0}
[exec] 03:50:06.251 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40001 type:getDate cxid:0x24 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/ids/0
[exec] 03:50:06.251 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40001 type:getDate cxid:0x24 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/ids/0
[exec] 03:50:06.253 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40001, packet:: clientPath:null serverPath:null
finished:false header:: 36,4 replyHeader:: 36,51,0 request::
'/brokers/ids/0,F response::
#7b2022686f7374223a226d656f30313035352e75732e6f7261636c652e636f6d222c20226a6d785f706f7274223a2d312c2022706f7274223a393039322c202274696d657374616d70223a2231343030323938363033313237222c202276657273696f6e223a31207d,s{6,6,1400298603202,1400298603202,0,0,0,91769969177788416,105,0,6}
[exec] 03:50:06.294 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Replicas assigned to topic
[ENRICHED], partition [0] are [List(0)]
[exec] 03:50:06.295 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Replicas assigned to topic
[ENRICHED], partition [1] are [List(0)]
[exec] 03:50:06.296 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Replicas assigned to topic
[ENRICHED], partition [2] are [List(0)]
[exec] 03:50:06.296 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Replicas assigned to topic
[ENRICHED], partition [3] are [List(0)]
[exec] 03:50:06.298 [ZkClient-EventThread-24-localhost:2181] [34mINFO
[0;39m [1;35mk.c.PartitionStateMachine$TopicChangeListener [0;39m -
[TopicChangeListener on Controller 0]: New topics: [Set(ENRICHED)], deleted
topics: [Set()], new partition replica assignment [Map([ENRICHED,3] ->
List(0), [ENRICHED,2] -> List(0), [ENRICHED,0] -> List(0), [ENRICHED,1] ->
List(0))]
[exec] 03:50:06.307 [ZkClient-EventThread-24-localhost:2181] [34mINFO
[0;39m [1;35mkafka.controller.KafkaController [0;39m - [Controller 0]: New
topic creation callback for
[ENRICHED,3],[ENRICHED,2],[ENRICHED,0],[ENRICHED,1]
[exec] 03:50:06.309 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:exists cxid:0x40 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.309 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:exists cxid:0x40 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.311 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 64,3 replyHeader:: 64,51,0 request::
'/brokers/topics/ENRICHED,T response::
s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
[exec] 03:50:06.311 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35morg.I0Itec.zkclient.ZkClient [0;39m - Subscribed data changes
for /brokers/topics/ENRICHED
[exec] 03:50:06.312 [ZkClient-EventThread-24-localhost:2181] [34mINFO
[0;39m [1;35mkafka.controller.KafkaController [0;39m - [Controller 0]: New
partition creation callback for
[ENRICHED,3],[ENRICHED,2],[ENRICHED,0],[ENRICHED,1]
[exec] 03:50:06.313 [ZkClient-EventThread-24-localhost:2181] [34mINFO
[0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
machine on Controller 0]: Invoking state change to NewPartition for
partitions [ENRICHED,3],[ENRICHED,2],[ENRICHED,0],[ENRICHED,1]
[exec] 03:50:06.314 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x41 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.314 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x41 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.316 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 65,4 replyHeader:: 65,51,0 request::
'/brokers/topics/ENRICHED,T response::
#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
[exec] 03:50:06.326 [ImmediateThread-1400298600406] [34mINFO [0;39m
[1;35mconfig.core [0;39m - Loading ***/***_schema.yaml
[exec] 03:50:06.335 [ImmediateThread-1400298600406] [34mINFO [0;39m
[1;35mconfig.core [0;39m - Found resource [***/***_schema.yaml] at
[jar:file:/scratch/home/Buildlab/12.1/branches/12.1.4.0.3/webcenter***/***/app/config/config-dev/target/config-dev-1.0-SNAPSHOT.jar!/***/***_schema.yaml]
[exec] 03:50:06.387
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mkafka.utils.VerifiableProperties [0;39m - Verifying properties
[exec] 03:50:06.387
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mkafka.utils.VerifiableProperties [0;39m - Property
metadata.broker.list is overridden to meo01055.us.***.com:9092
[exec] 03:50:06.388
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mkafka.utils.VerifiableProperties [0;39m - Property
request.timeout.msis overridden to 30000
[exec] 03:50:06.388
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mkafka.utils.VerifiableProperties [0;39m - Property client.id is
overridden to RAW
[exec] 03:50:06.394
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Fetching metadata from broker
id:0,host:meo01055.us.***.com,port:9092 with correlation id 0 for 1
topic(s) Set(RAW)
[exec] 03:50:06.395 [kafka-acceptor] [39mDEBUG [0;39m
[1;35mkafka.network.Acceptor [0;39m - Accepted connection from /
10.152.196.47 on /10.152.196.47:9092. sendBufferSize [actual|requested]:
[1048576|1048576] recvBufferSize [actual|requested]: [1048576|1048576]
[exec] 03:50:06.395 [kafka-processor-9092-0] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Processor 0 listening to new
connection from /10.152.196.47:62293
[exec] 03:50:06.396
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
[1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
= 30000 (requested 30000), SO_RCVBUF = 65632 (requested -1), SO_SNDBUF =
102400 (requested 102400).
[exec] 03:50:06.400
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Connected to
meo01055.us.***.com:9092 for producing
[exec] 03:50:06.406 [kafka-request-handler-0] [39mDEBUG [0;39m
[1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] [RAW,0];replicas =
Set(0), in sync replicas = List(0), leader = 0
[exec] 03:50:06.422
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mkafka.producer.SyncProducer [0;39m - Disconnecting from
meo01055.us.***.com:9092
[exec] 03:50:06.422 [kafka-processor-9092-0] [34mINFO [0;39m
[1;35mkafka.network.Processor [0;39m - Closing socket connection to /
10.152.196.47.
[exec] 03:50:06.423 [kafka-processor-9092-0] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Closing connection from /
10.152.196.47:62293
[exec] 03:50:06.430
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
[1;35mkafka.client.ClientUtils$ [0;39m - Successfully fetched metadata for
1 topic(s) Set(RAW)
[exec] 03:50:06.434
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
[1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread], {TopicMetadata
for topic RAW ->
[exec] Metadata for partition [RAW,0] is partition 0 leader: 0
(meo01055.us.***.com:9092) replicas: 0 (meo01055.us.***.com:9092) isr: 0
(meo01055.us.***.com:9092) isUnderReplicated: false}
[exec] 03:50:06.463 [kafka-acceptor] [39mDEBUG [0;39m
[1;35mkafka.network.Acceptor [0;39m - Accepted connection from /
10.152.196.47 on /10.152.196.47:9092. sendBufferSize [actual|requested]:
[1048576|1048576] recvBufferSize [actual|requested]: [1048576|1048576]
[exec] 03:50:06.463 [kafka-processor-9092-1] [39mDEBUG [0;39m
[1;35mkafka.network.Processor [0;39m - Processor 1 listening to new
connection from /10.152.196.47:62294
[exec] 03:50:06.464
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
[1;35mkafka.network.BlockingChannel [0;39m - Created socket with SO_TIMEOUT
= 30000 (requested 30000), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF =
26010 (requested -1).
[exec] 03:50:06.469
[ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0] [34mINFO
[0;39m [1;35mkafka.consumer.ConsumerFetcherThread [0;39m -
[ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0], Starting
[exec] 03:50:06.502 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:getDate cxid:0x15 zxid:0xfffffffffffffffe
txntype:unknown
reqpath:/consumers/***/ids/***_meo01055-1400298605173-eed8b129
[exec] 03:50:06.503 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:getDate cxid:0x15 zxid:0xfffffffffffffffe
txntype:unknown
reqpath:/consumers/***/ids/***_meo01055-1400298605173-eed8b129
[exec] 03:50:06.505 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 21,4 replyHeader:: 21,51,0 request::
'/consumers/***/ids/***_meo01055-1400298605173-eed8b129,F response::
#7b20227061747465726e223a22737461746963222c2022737562736372697074696f6e223a7b2022454e524943484544223a2034207d2c202274696d657374616d70223a2231343030323938363035323133222c202276657273696f6e223a31207d,s{24,24,1400298605244,1400298605244,0,0,0,91769969177788418,98,0,24}
[exec] 03:50:06.533
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
[1;35mkafka.consumer.PartitionTopicInfo [0;39m - reset fetch offset of (
RAW:0: fetched offset = 0: consumed offset = -1 ) to 0
[exec] 03:50:06.535
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [39mDEBUG [0;39m
[1;35mkafka.consumer.PartitionTopicInfo [0;39m - reset consume offset of
RAW:0: fetched offset = 0: consumed offset = 0 to 0
[exec] 03:50:06.543
[RAW_meo01055-1400298603522-1cb22725-leader-finder-thread] [34mINFO [0;39m
[1;35mk.consumer.ConsumerFetcherManager [0;39m -
[ConsumerFetcherManager-1400298603576] Adding fetcher for partition
[RAW,0], initOffset -1 to broker 0 with fetcherId 0
[exec] 03:50:06.572 [kafka-request-handler-0] [39mDEBUG [0;39m
[1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
with correlation id 0 from client
RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
purgatory
[exec] 03:50:06.595 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x42 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.595 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x42 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.598 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 66,4 replyHeader:: 66,51,0 request::
'/brokers/topics/ENRICHED,T response::
#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
[exec] 03:50:06.647 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:getChildren cxid:0x16
zxid:0xfffffffffffffffe txntype:unknown reqpath:/consumers/***/ids
[exec] 03:50:06.647 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:getChildren cxid:0x16
zxid:0xfffffffffffffffe txntype:unknown reqpath:/consumers/***/ids
[exec] 03:50:06.649 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 22,8 replyHeader:: 22,51,0 request::
'/consumers/***/ids,T response:: v{'***_meo01055-1400298605173-eed8b129}
[exec] 03:50:06.650 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:getDate cxid:0x17 zxid:0xfffffffffffffffe
txntype:unknown
reqpath:/consumers/***/ids/***_meo01055-1400298605173-eed8b129
[exec] 03:50:06.650 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:getDate cxid:0x17 zxid:0xfffffffffffffffe
txntype:unknown
reqpath:/consumers/***/ids/***_meo01055-1400298605173-eed8b129
[exec] 03:50:06.653 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 23,4 replyHeader:: 23,51,0 request::
'/consumers/***/ids/***_meo01055-1400298605173-eed8b129,F response::
#7b20227061747465726e223a22737461746963222c2022737562736372697074696f6e223a7b2022454e524943484544223a2034207d2c202274696d657374616d70223a2231343030323938363035323133222c202276657273696f6e223a31207d,s{24,24,1400298605244,1400298605244,0,0,0,91769969177788418,98,0,24}
[exec] 03:50:06.686 [request-expiration-task] [39mDEBUG [0;39m
[1;35mk.s.KafkaApis$FetchRequestPurgatory [0;39m -
[FetchRequestPurgatory-0] Expiring fetch request Name: FetchRequest;
Version: 0; CorrelationId: 0; ClientId:
RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0;
ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [RAW,0] ->
PartitionFetchInfo(0,1048576).
[exec] 03:50:06.810 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x43 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.811 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x43 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.813 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 67,4 replyHeader:: 67,51,0 request::
'/brokers/topics/ENRICHED,T response::
#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
[exec] 03:50:06.818 [kafka-request-handler-1] [39mDEBUG [0;39m
[1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
with correlation id 1 from client
RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
purgatory
[exec] 03:50:06.887 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:getChildren cxid:0x18
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.887 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:getChildren cxid:0x18
zxid:0xfffffffffffffffe txntype:unknown reqpath:/brokers/ids
[exec] 03:50:06.889 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 24,8 replyHeader:: 24,51,0 request::
'/brokers/ids,F response:: v{'0}
[exec] 03:50:06.890 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:getDate cxid:0x19 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/ids/0
[exec] 03:50:06.890 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:getDate cxid:0x19 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/ids/0
[exec] 03:50:06.893 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 25,4 replyHeader:: 25,51,0 request::
'/brokers/ids/0,F response::
#7b2022686f7374223a226d656f30313035352e75732e6f7261636c652e636f6d222c20226a6d785f706f7274223a2d312c2022706f7274223a393039322c202274696d657374616d70223a2231343030323938363033313237222c202276657273696f6e223a31207d,s{6,6,1400298603202,1400298603202,0,0,0,91769969177788416,105,0,6}
[exec] 03:50:06.918 [request-expiration-task] [39mDEBUG [0;39m
[1;35mk.s.KafkaApis$FetchRequestPurgatory [0;39m -
[FetchRequestPurgatory-0] Expiring fetch request Name: FetchRequest;
Version: 0; CorrelationId: 1; ClientId:
RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0;
ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [RAW,0] ->
PartitionFetchInfo(0,1048576).
[exec] 03:50:06.922 [kafka-request-handler-0] [39mDEBUG [0;39m
[1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
with correlation id 2 from client
RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
purgatory
[exec] 03:50:06.956 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x44 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.957 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x44 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:06.960 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 68,4 replyHeader:: 68,51,0 request::
'/brokers/topics/ENRICHED,T response::
#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
[exec] 03:50:06.960 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mk.producer.async.ProducerSendThread [0;39m - 1002 ms elapsed. Queue
time reached. Sending..
[exec] 03:50:06.960 [ProducerSendThread-] [39mDEBUG [0;39m
[1;35mk.producer.async.ProducerSendThread [0;39m - Handling 0 events
[exec] 03:50:07.025 [request-expiration-task] [39mDEBUG [0;39m
[1;35mk.s.KafkaApis$FetchRequestPurgatory [0;39m -
[FetchRequestPurgatory-0] Expiring fetch request Name: FetchRequest;
Version: 0; CorrelationId: 2; ClientId:
RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0;
ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [RAW,0] ->
PartitionFetchInfo(0,1048576).
[exec] 03:50:07.028 [kafka-request-handler-1] [39mDEBUG [0;39m
[1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
with correlation id 3 from client
RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
purgatory
[exec] 03:50:07.052 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40002 type:getDate cxid:0x1a zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:07.052 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40002 type:getDate cxid:0x1a zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED
[exec] 03:50:07.054 [ConsumerServiceImpl
STARTING-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40002, packet:: clientPath:null serverPath:null
finished:false header:: 26,4 replyHeader:: 26,51,0 request::
'/brokers/topics/ENRICHED,T response::
#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d2c202232223a5b2030205d2c202233223a5b2030205d207d2c202276657273696f6e223a31207d,s{44,44,1400298606117,1400298606117,0,0,0,0,76,0,44}
[exec] 03:50:07.097 [ZkClient-EventThread-24-localhost:2181] [34mINFO
[0;39m [1;35mkafka.controller.ReplicaStateMachine [0;39m - [Replica state
machine on controller 0]: Invoking state change to NewReplica for replicas
PartitionAndReplica(ENRICHED,3,0),PartitionAndReplica(ENRICHED,2,0),PartitionAndReplica(ENRICHED,0,0),PartitionAndReplica(ENRICHED,1,0)
[exec] 03:50:07.100 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x45 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/3/state
[exec] 03:50:07.100 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x45 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/3/state
[exec] 03:50:07.102 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 69,4 replyHeader:: 69,51,-101 request::
'/brokers/topics/ENRICHED/partitions/3/state,F response::
[exec] 03:50:07.103 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x46 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/2/state
[exec] 03:50:07.103 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x46 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/2/state
[exec] 03:50:07.105 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 70,4 replyHeader:: 70,51,-101 request::
'/brokers/topics/ENRICHED/partitions/2/state,F response::
[exec] 03:50:07.107 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x47 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/0/state
[exec] 03:50:07.107 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x47 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/0/state
[exec] 03:50:07.109 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 71,4 replyHeader:: 71,51,-101 request::
'/brokers/topics/ENRICHED/partitions/0/state,F response::
[exec] 03:50:07.110 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:getDate cxid:0x48 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/1/state
[exec] 03:50:07.110 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:getDate cxid:0x48 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/brokers/topics/ENRICHED/partitions/1/state
[exec] 03:50:07.112 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 72,4 replyHeader:: 72,51,-101 request::
'/brokers/topics/ENRICHED/partitions/1/state,F response::
[exec] 03:50:07.114 [ZkClient-EventThread-24-localhost:2181] [34mINFO
[0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
machine on Controller 0]: Invoking state change to OnlinePartition for
partitions [ENRICHED,3],[ENRICHED,2],[ENRICHED,0],[ENRICHED,1]
[exec] 03:50:07.121 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
machine on Controller 0]: Live assigned replicas for partition [ENRICHED,3]
are: [List(0)]
[exec] 03:50:07.122 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
machine on Controller 0]: Initializing leader and isr for partition
[ENRICHED,3] to (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1)
[exec] 03:50:07.124 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40000 type:create
cxid:0x49 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/ENRICHED/partitions/3 Error:KeeperErrorCode = NoNode
for /brokers/topics/ENRICHED/partitions/3
[exec] 03:50:07.125 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x49 zxid:0x34 txntype:-1
reqpath:n/a
[exec] 03:50:07.127 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 73,1 replyHeader:: 73,52,-101 request::
'/brokers/topics/ENRICHED/partitions/3/state,#7b2022636f6e74726f6c6c65725f65706f6368223a312c2022697372223a5b2030205d2c20226c6561646572223a302c20226c65616465725f65706f6368223a302c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response::
[exec] 03:50:07.128 [ProcessThread:-1] [34mINFO [0;39m
[1;35mo.a.z.server.PrepRequestProcessor [0;39m - Got user-level
KeeperException when processing sessionid:0x146084cfde40000 type:create
cxid:0x4a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/ENRICHED/partitions Error:KeeperErrorCode = NoNode for
/brokers/topics/ENRICHED/partitions
[exec] 03:50:07.129 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x4a zxid:0x35 txntype:-1
reqpath:n/a
[exec] 03:50:07.129 [request-expiration-task] [39mDEBUG [0;39m
[1;35mk.s.KafkaApis$FetchRequestPurgatory [0;39m -
[FetchRequestPurgatory-0] Expiring fetch request Name: FetchRequest;
Version: 0; CorrelationId: 3; ClientId:
RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0;
ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [RAW,0] ->
PartitionFetchInfo(0,1048576).
[exec] 03:50:07.133 [kafka-request-handler-0] [39mDEBUG [0;39m
[1;35mkafka.server.KafkaApis [0;39m - [KafkaApi-0] Putting fetch request
with correlation id 4 from client
RAW-ConsumerFetcherThread-RAW_meo01055-1400298603522-1cb22725-0-0 into
purgatory
[exec] 03:50:07.134 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 74,1 replyHeader:: 74,53,-101 request::
'/brokers/topics/ENRICHED/partitions/3,,v{s{31,s{'world,'anyone}}},0
response::
[exec] 03:50:07.136 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x4b zxid:0x36 txntype:1
reqpath:n/a
[exec] 03:50:07.136 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:create cxid:0x4b zxid:0x36 txntype:1
reqpath:n/a
[exec] 03:50:07.146 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 75,1 replyHeader:: 75,54,0 request::
'/brokers/topics/ENRICHED/partitions,,v{s{31,s{'world,'anyone}}},0
response:: '/brokers/topics/ENRICHED/partitions
[exec] 03:50:07.152 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x4c zxid:0x37 txntype:1
reqpath:n/a
[exec] 03:50:07.153 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:create cxid:0x4c zxid:0x37 txntype:1
reqpath:n/a
[exec] 03:50:07.157 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.utils.ZkUtils$ [0;39m - Partition map for
/brokers/topics/ENRICHED is Map(0 -> List(0), 1 -> List(0), 2 -> List(0), 3
-> List(0))
[exec] 03:50:07.162 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 76,1 replyHeader:: 76,55,0 request::
'/brokers/topics/ENRICHED/partitions/3,,v{s{31,s{'world,'anyone}}},0
response:: '/brokers/topics/ENRICHED/partitions/3
[exec] 03:50:07.164 [ZkClient-EventThread-48-localhost:2181] [34mINFO
[0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
[ConsumerFetcherManager-1400298605202] Stopping leader finder thread
[exec] 03:50:07.165 [ZkClient-EventThread-48-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
[***_meo01055-1400298605173-eed8b129-leader-finder-thread], Shutting down
[exec] 03:50:07.165 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m - Processing request::
sessionid:0x146084cfde40000 type:create cxid:0x4d zxid:0x38 txntype:1
reqpath:n/a
[exec] 03:50:07.170
[***_meo01055-1400298605173-eed8b129-leader-finder-thread] [34mINFO [0;39m
[1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
[***_meo01055-1400298605173-eed8b129-leader-finder-thread], Stopped
[exec] 03:50:07.170 [ZkClient-EventThread-48-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ConsumerFetcherManager$LeaderFinderThread [0;39m -
[***_meo01055-1400298605173-eed8b129-leader-finder-thread], Shutdown
completed
[exec] 03:50:07.171 [SyncThread:0] [39mDEBUG [0;39m
[1;35mo.a.z.server.FinalRequestProcessor [0;39m -
sessionid:0x146084cfde40000 type:create cxid:0x4d zxid:0x38 txntype:1
reqpath:n/a
[exec] 03:50:07.171 [ZkClient-EventThread-48-localhost:2181] [34mINFO
[0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
[ConsumerFetcherManager-1400298605202] Stopping all fetchers
[exec] 03:50:07.172 [main-SendThread(localhost:2181)] [39mDEBUG [0;39m
[1;35morg.apache.zookeeper.ClientCnxn [0;39m - Reading reply
sessionid:0x146084cfde40000, packet:: clientPath:null serverPath:null
finished:false header:: 77,1 replyHeader:: 77,56,0 request::
'/brokers/topics/ENRICHED/partitions/3/state,#7b2022636f6e74726f6c6c65725f65706f6368223a312c2022697372223a5b2030205d2c20226c6561646572223a302c20226c65616465725f65706f6368223a302c202276657273696f6e223a31207d,v{s{31,s{'world,'anyone}}},0
response:: '/brokers/topics/ENRICHED/partitions/3/state
[exec] 03:50:07.173 [ZkClient-EventThread-48-localhost:2181] [34mINFO
[0;39m [1;35mk.consumer.ConsumerFetcherManager [0;39m -
[ConsumerFetcherManager-1400298605202] All connections stopped
[exec] 03:50:07.173 [ZkClient-EventThread-48-localhost:2181] [34mINFO
[0;39m [1;35mk.c.ZookeeperConsumerConnector [0;39m -
[***_meo01055-1400298605173-eed8b129], Cleared all relevant queues for this
fetcher
[exec] 03:50:07.173 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
data chunk for this consumer iterator
[exec] 03:50:07.174 [ZkClient-EventThread-24-localhost:2181] [39mDEBUG
[0;39m [1;35mk.controller.PartitionStateMachine [0;39m - [Partition state
machine on Controller 0]: Live assigned replicas for partition [ENRICHED,2]
are: [List(0)]
[exec] 03:50:07.174 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
data chunk for this consumer iterator
[exec] 03:50:07.174 [ZkClient-EventThread-48-localhost:2181] [39mDEBUG
[0;39m [1;35mkafka.consumer.ConsumerIterator [0;39m - Clearing the current
data chunk for this consumer iterator
The CorrelationId is increasing until we killed the build. (we have a
mostly running deployment, the above failure is from integration test)
Best Regards,
Mingtao
On Mon, May 19, 2014 at 2:06 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Mingtao,
>
> 1. Do you see any error/warn log before this entry?
> 2. The producer will re-try on sending when the previous trial is not
> successful. In your case it has exhausted all the retries and hence
> messages are dropped on this floor and lost.
>
> Guozhang
>
>
> On Mon, May 19, 2014 at 8:32 AM, Mingtao Zhang <vansriddles@gmail.com
> >wrote:
>
> > Hi,
> >
> > I got this from the log:
> >
> > [exec] 03:50:06.216 [ProducerSendThread-] [1;31mERROR [0;39m
> > [1;35mk.producer.async.ProducerSendThread [0;39m - Error in handling
> batch
> > of 100 events
> > [exec] kafka.common.FailedToSendMessageException: Failed to send
> > messages after 3 tries.
> > [exec] at kafka.producer.async.DefaultEventHandler.handle(Unknown
> > Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
> > [exec] at
> kafka.producer.async.ProducerSendThread.tryToHandle(Unknown
> > Source) [kafka_2.10-0.8.0.jar:0.8.0]
> > [exec] at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
> > Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
> > [exec] at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
> > Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
> > [exec] at
> scala.collection.immutable.Stream.foreach(Stream.scala:547)
> > ~[scala-library-2.10.1.jar:na]
> > [exec] at
> > kafka.producer.async.ProducerSendThread.processEvents(Unknown Source)
> > [kafka_2.10-0.8.0.jar:0.8.0]
> > [exec] at kafka.producer.async.ProducerSendThread.run(Unknown
> Source)
> > [kafka_2.10-0.8.0.jar:0.8.0]
> >
> > My question is:
> >
> > 1. What could cause this?
> > 2. Who should deal with the recovery? User or Kafka?
> >
> > Let me know if more log is needed. (It hangs the integration test, so I
> > have 3.5G of them ... from a maven build)
> >
> > --
> >
> > Best Regards,
> > Mingtao Zhang
> >
>
>
>
> --
> -- Guozhang
>
--
Best Regards,
Mingtao Zhang
Re: Cause and Recovery
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Mingtao,
1. Do you see any error/warn log before this entry?
2. The producer will re-try on sending when the previous trial is not
successful. In your case it has exhausted all the retries and hence
messages are dropped on this floor and lost.
Guozhang
On Mon, May 19, 2014 at 8:32 AM, Mingtao Zhang <va...@gmail.com>wrote:
> Hi,
>
> I got this from the log:
>
> [exec] 03:50:06.216 [ProducerSendThread-] [1;31mERROR [0;39m
> [1;35mk.producer.async.ProducerSendThread [0;39m - Error in handling batch
> of 100 events
> [exec] kafka.common.FailedToSendMessageException: Failed to send
> messages after 3 tries.
> [exec] at kafka.producer.async.DefaultEventHandler.handle(Unknown
> Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
> [exec] at kafka.producer.async.ProducerSendThread.tryToHandle(Unknown
> Source) [kafka_2.10-0.8.0.jar:0.8.0]
> [exec] at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
> Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
> [exec] at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
> Source) ~[kafka_2.10-0.8.0.jar:0.8.0]
> [exec] at scala.collection.immutable.Stream.foreach(Stream.scala:547)
> ~[scala-library-2.10.1.jar:na]
> [exec] at
> kafka.producer.async.ProducerSendThread.processEvents(Unknown Source)
> [kafka_2.10-0.8.0.jar:0.8.0]
> [exec] at kafka.producer.async.ProducerSendThread.run(Unknown Source)
> [kafka_2.10-0.8.0.jar:0.8.0]
>
> My question is:
>
> 1. What could cause this?
> 2. Who should deal with the recovery? User or Kafka?
>
> Let me know if more log is needed. (It hangs the integration test, so I
> have 3.5G of them ... from a maven build)
>
> --
>
> Best Regards,
> Mingtao Zhang
>
--
-- Guozhang