You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mo Firouz <Mo...@visualdna.com> on 2014/03/19 15:07:10 UTC
Migration Tool consumer rebalance error when attempting to migrate
from Kafka 0.7 to Kafka 0.8
Hello.
I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this
is to use the Migration Tool.
I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local
Vagrant (along with a zookeeper running locally). The Migration Tool
Producer will connect to these three nodes in order to replicate the data.
The Migration Tool Consumer(s) will connect to my Staging boxes to consume
Kafka 0.7 events.
However, upon running the Migration Tool, I'll get the following error:
*vagrant@precise64:~/Desktop$
> ~/Desktop/kafka_2.9.2-0.8.1/bin/kafka-run-class.sh
> kafka.tools.KafkaMigrationTool --kafka.07.jar kafka-0.7.19.jar
> --zkclient.01.jar zkclient-0.2.0.jar --num.producers 16
> --producer.config=/home/vagrant/Deskt/producerConfig.properties
> --consumer.config=/home/vagrant/Desktop/consumerConfig.properties
> --whitelist "prod_conversions"*
[2014-03-19 13:14:38,126] WARN Property mirror.topics.whitelist is not
> valid (kafka.utils.VerifiableProperties)
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Kafka migration tool failed due to:
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> Caused by: kafka.common.ConsumerRebalanceFailedException:
> mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
> rebalance after 4 retries
> at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> at
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> at
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> at
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> ... 5 more
> [2014-03-19 13:14:47,082] ERROR Kafka migration tool failed:
> (kafka.tools.KafkaMigrationTool)
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> Caused by: kafka.common.ConsumerRebalanceFailedException:
> mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
> rebalance after 4 retries
> at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> at
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> at
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> at
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> ... 5 more
My 3 *Kafka server* configurations are as follows:
*cat kafka-server1.properties *
> zookeeper.connect=localhost:2181
> zookeeper.connection.timeout.ms=1000000
> broker.id=1
> port=9092
> log.dir=/tmp/kafka-logs-1
>
> *cat kafka-server2.properties *
> zookeeper.connect=localhost:2181
> zookeeper.connection.timeout.ms=1000000
> broker.id=1
> port=9093
> log.dir=/tmp/kafka-logs-2
> *cat kafka-server3.properties *
> zookeeper.connect=localhost:2181
> zookeeper.connection.timeout.ms=1000000
> broker.id=1
> port=9094
> log.dir=/tmp/kafka-logs-3
My *consumer* configuration is:
*cat consumerConfig.properties *
> group.id = mfirouz-vagrant-kafkamigration
> zookeeper.connect = zk-01.s.visualdna.com:2181,zk-02.s.visualdna.com:2181,
> zk-03.s.visualdna.com:2181
> mirror.topics.whitelist = "prod_conversions"
And finally, my *producer* configuration is:
*cat producerConfig.properties*
> zk.connect = localhost:2181
> whitelist = "prod_conversions"
> producer.type = "sync"
> request.required.acks = 1
> metadata.broker.list = localhost:9092,localhost:9093,localhost:9094
I was wondering if anyone had any insight about this and how to fix it?
Thanks a lot in advance
Mo.
Re: Migration Tool consumer rebalance error when attempting to
migrate from Kafka 0.7 to Kafka 0.8
Posted by Jun Rao <ju...@gmail.com>.
The following is the issue. It seems that the 0.7 jar that you used in
MigrationTool is not the same as the one used in 0.7 broker. So, the
consumer can't parse the format of the broker registration in ZK.
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
exception during rebalance (kafka.consumer.ZookeeperConsumerConnector)
kafka.common.KafkaException: Failed to parse the broker info from
zookeeper: bl-queue-s03.visualdna.com-1394796157928:
bl-queue-s03.visualdna.com:9092
Thanks,
Jun
On Wed, Mar 19, 2014 at 10:44 AM, Mo Firouz <Mo...@visualdna.com> wrote:
> Hi Jun,
>
> Thanks for the reply.
>
> Please find below the DEBUG output of the log.
>
> *vagrant@precise64:~/Desktop$
> ~/Desktop/kafka_2.9.2-0.8.1/bin/kafka-run-class.sh
> kafka.tools.KafkaMigrationTool --kafka.07.jar kafka-0.7.19.jar
> --zkclient.01.jar zkclient-0.2.0.jar --num.producers 16
> --producer.config=/home/vagrant/Desktop/producerConfig.properties
> --consumer.config=/home/vagrant/Desktop/consumerConfig.properties
> --whitelist "prod_conversions"*
> [2014-03-19 17:35:29,856] INFO Verifying properties
> (kafka.utils.VerifiableProperties)
> [2014-03-19 17:35:29,925] INFO Property group.id is overridden to
> mfirouz-vagrant-kafkamigration (kafka.utils.VerifiableProperties)
> [2014-03-19 17:35:29,926] WARN Property mirror.topics.whitelist is not
> valid (kafka.utils.VerifiableProperties)
> [2014-03-19 17:35:29,927] INFO Property zookeeper.connect is overridden to
> zookeeper-01.stag.visualdna.com:2181,zookeeper-02.stag.visualdna.com:2181,
> zookeeper-03.stag.visualdna.com:2181 (kafka.utils.VerifiableProperties)
> [2014-03-19 17:35:29,952] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> Connecting to zookeeper instance at zookeeper-01.stag.visualdna.com:2181,
> zookeeper-02.stag.visualdna.com:2181,zookeeper-03.stag.visualdna.com:2181
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:29,969] DEBUG Creating new ZookKeeper instance to connect
> to zookeeper-01.stag.visualdna.com:2181,
> zookeeper-02.stag.visualdna.com:2181
> ,zookeeper-03.stag.visualdna.com:2181. (org.I0Itec.zkclient.ZkConnection)
> [2014-03-19 17:35:29,974] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2014-03-19 17:35:29,982] INFO Client
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,982] INFO Client environment:host.name=precise64
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,982] INFO Client environment:java.version=1.7.0_51
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,982] INFO Client environment:java.vendor=Oracle
> Corporation (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,983] INFO Client
> environment:java.home=/opt/java/jdk1.7.0_51/jre
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,983] INFO Client
>
> environment:java.class.path=:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../core/build/dependant-libs-2.8.0/*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../clients/build/libs//kafka-clients*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../examples/build/libs//kafka-examples*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/jopt-simple-3.2.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/kafka_2.9.2-0.8.1.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/log4j-1.2.15.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/metrics-annotation-2.2.0.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/metrics-core-2.2.0.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/scala-library-2.9.2.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/slf4j-api-1.7.2.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/snappy-java-1.0.5.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/zkclient-0.3.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/zookeeper-3.3.4.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../core/build/libs/kafka_2.8.0*.jar
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,983] INFO Client
>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,984] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,985] INFO Client environment:java.compiler=<NA>
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,985] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,985] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,986] INFO Client
> environment:os.version=3.2.0-23-generic (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,986] INFO Client environment:user.name=vagrant
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,987] INFO Client environment:user.home=/home/vagrant
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,987] INFO Client
> environment:user.dir=/home/vagrant/Desktop (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,990] INFO Initiating client connection, connectString=
> zookeeper-01.stag.visualdna.com:2181,zookeeper-02.stag.visualdna.com:2181,
> zookeeper-03.stag.visualdna.com:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@50e7a7f2
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-19 17:35:29,993] DEBUG zookeeper.disableAutoWatchReset is false
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,027] DEBUG Awaiting connection to Zookeeper server
> (org.I0Itec.zkclient.ZkClient)
> [2014-03-19 17:35:30,027] INFO Opening socket connection to server
>
> zookeeper-03.stag.visualdna.com/10.200.10.63:2181(org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,027] DEBUG Waiting for keeper state SyncConnected
> (org.I0Itec.zkclient.ZkClient)
> [2014-03-19 17:35:30,042] INFO Socket connection established to
> zookeeper-03.stag.visualdna.com/10.200.10.63:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,047] DEBUG Session establishment request sent on
>
> zookeeper-03.stag.visualdna.com/10.200.10.63:2181(org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,068] INFO Session establishment complete on server
> zookeeper-03.stag.visualdna.com/10.200.10.63:2181, sessionid =
> 0x3448da1f8f0644b, negotiated timeout = 6000
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,072] DEBUG Received event: WatchedEvent
> state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
> [2014-03-19 17:35:30,073] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2014-03-19 17:35:30,073] DEBUG Leaving process event
> (org.I0Itec.zkclient.ZkClient)
> [2014-03-19 17:35:30,073] DEBUG State is SyncConnected
> (org.I0Itec.zkclient.ZkClient)
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> [2014-03-19 17:35:30,111] DEBUG Initializing task scheduler.
> (kafka.utils.KafkaScheduler)
> [2014-03-19 17:35:30,114] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], starting
> auto committer every 60000 ms (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,118] DEBUG Scheduling task kafka-consumer-autocommit
> with initial delay 60000 ms and period 60000 ms.
> (kafka.utils.KafkaScheduler)
> [2014-03-19 17:35:30,210] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 1,8
> replyHeader:: 1,150361056246,0 request:: '/brokers/topics,F response::
>
> v{'profileupdates_gsantoro,'prod_interactions,'datasync,'audiencemembership,'prod_profileupdates,'audienceactivity,'dataexport,'prod_useractivity,'dataexport-gsantoro,'browsingactivity,'tracking_events,'profileactivity,'prod_conversions,'prod_pageviews,'prod_datasync,'prod_dataexport,'profileupdates,'pageviews,'useractivity,'conversions,'prod_profileactivity,'interactions,'quizactivity}
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,221] DEBUG profileupdates_gsantoro filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,222] DEBUG prod_interactions filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,222] DEBUG datasync filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,223] DEBUG audiencemembership filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,224] DEBUG prod_profileupdates filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,224] DEBUG audienceactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,225] DEBUG dataexport filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,225] DEBUG prod_useractivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,225] DEBUG dataexport-gsantoro filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,226] DEBUG browsingactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,226] DEBUG tracking_events filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,227] DEBUG profileactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,227] DEBUG prod_conversions allowed
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,229] DEBUG prod_pageviews filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,230] DEBUG prod_datasync filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,230] DEBUG prod_dataexport filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,231] DEBUG profileupdates filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,231] DEBUG pageviews filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,232] DEBUG useractivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,232] DEBUG conversions filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,233] DEBUG prod_profileactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,234] DEBUG interactions filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,234] DEBUG quizactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,252] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
> registering consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce in ZK
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,409] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 2,1
> replyHeader:: 2,150361056248,0 request::
>
> '/consumers/mfirouz-vagrant-kafkamigration/ids/mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b2270726f645f636f6e76657273696f6e73223a317d2c227061747465726e223a2277686974655f6c697374222c2274696d657374616d70223a2231333935323530353330323537227d,v{s{31,s{'world,'anyone}}},1
> response::
>
> '/consumers/mfirouz-vagrant-kafkamigration/ids/mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,410] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
> registering consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce in ZK
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,427] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], starting
> watcher executor thread for consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,433] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 3,8
> replyHeader:: 3,150361056250,0 request:: '/brokers/topics,F response::
>
> v{'profileupdates_gsantoro,'prod_interactions,'datasync,'audiencemembership,'prod_profileupdates,'audienceactivity,'dataexport,'prod_useractivity,'dataexport-gsantoro,'browsingactivity,'tracking_events,'profileactivity,'prod_conversions,'prod_pageviews,'prod_datasync,'prod_dataexport,'profileupdates,'pageviews,'useractivity,'conversions,'prod_profileactivity,'interactions,'quizactivity}
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,437] DEBUG profileupdates_gsantoro filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,437] DEBUG prod_interactions filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,437] DEBUG datasync filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,438] DEBUG audiencemembership filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,438] DEBUG prod_profileupdates filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,438] DEBUG audienceactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,438] DEBUG dataexport filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,439] DEBUG prod_useractivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,439] DEBUG dataexport-gsantoro filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,439] DEBUG browsingactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,439] DEBUG tracking_events filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,440] DEBUG profileactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,440] DEBUG prod_conversions allowed
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,440] DEBUG prod_pageviews filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,441] DEBUG prod_datasync filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,441] DEBUG prod_dataexport filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,441] DEBUG profileupdates filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,441] DEBUG pageviews filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,441] DEBUG useractivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,442] DEBUG conversions filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,442] DEBUG prod_profileactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,442] DEBUG interactions filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,442] DEBUG quizactivity filtered
> (kafka.consumer.Whitelist)
> [2014-03-19 17:35:30,491] DEBUG
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Adding
> topicThreadId
>
> (prod_conversions,mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce-0)
> and queue [] to topicThreadIdAndQueues data structure
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,499] DEBUG
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], adding
> topic prod_conversions and 1 streams to map.
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,517] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 4,3
> replyHeader:: 4,150361056251,0 request::
> '/consumers/mfirouz-vagrant-kafkamigration/ids,T response::
>
> s{150358573750,150358573750,1395165000820,1395165000820,0,35,0,0,0,1,150361056248}
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,525] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 5,8
> replyHeader:: 5,150361056251,0 request::
> '/consumers/mfirouz-vagrant-kafkamigration/ids,T response::
> v{'mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce}
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,536] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 6,3
> replyHeader:: 6,150361056251,0 request::
> '/brokers/topics/prod_conversions,T response::
>
> s{42963010923,42963010923,1363690601510,1363690601510,0,883,0,0,0,3,150357230474}
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,536] DEBUG Subscribed data changes for
> /brokers/topics/prod_conversions (org.I0Itec.zkclient.ZkClient)
> [2014-03-19 17:35:30,542] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
> rebalancing consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #0
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,550] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 7,8
> replyHeader:: 7,150361056251,0 request:: '/brokers/ids,F response::
> v{'3,'2,'1} (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,563] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 8,4
> replyHeader:: 8,150361056251,0 request:: '/brokers/ids/3,F response::
>
> #626c2d71756575652d7330332e76697375616c646e612e636f6d2d313339343739363135373932383a626c2d71756575652d7330332e76697375616c646e612e636f6d3a39303932,s{150346957985,150346957985,1394796158123,1394796158123,0,0,0,163411213060484448,72,0,150346957985}
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:30,649] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> exception during rebalance (kafka.consumer.ZookeeperConsumerConnector)
> kafka.common.KafkaException: Failed to parse the broker info from
> zookeeper: bl-queue-s03.visualdna.com-1394796157928:
> bl-queue-s03.visualdna.com:9092
> at kafka.cluster.Broker$.createBroker(Broker.scala:45)
> at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:506)
> at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:504)
> at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> at
>
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
> at
>
> scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
> at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:504)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:407)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:402)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> Caused by: kafka.common.BrokerNotAvailableException: Broker id 3 does not
> exist
> at kafka.cluster.Broker$.createBroker(Broker.scala:42)
> ... 20 more
> [2014-03-19 17:35:30,669] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
> rebalancing consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #0
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,670] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> Rebalancing attempt failed. Clearing the cache before the next rebalancing
> operation is triggered (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,674] INFO [ConsumerFetcherManager-1395250530084]
> Stopping leader finder thread (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:30,675] INFO [ConsumerFetcherManager-1395250530084]
> Stopping all fetchers (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:30,677] INFO [ConsumerFetcherManager-1395250530084] All
> connections stopped (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:30,679] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
> all relevant queues for this fetcher
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,681] DEBUG Clearing the current data chunk for this
> consumer iterator (kafka.consumer.ConsumerIterator)
> [2014-03-19 17:35:30,682] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
> the data chunks in all the consumer message iterators
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:30,683] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> Committing all offsets after clearing the fetcher queues
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:32,603] DEBUG Got ping response for sessionid:
> 0x3448da1f8f0644b after 41ms (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:32,687] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
> rebalancing consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #1
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:32,693] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 9,8
> replyHeader:: 9,150361056321,0 request:: '/brokers/ids,F response::
> v{'3,'2,'1} (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:32,712] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 10,4
> replyHeader:: 10,150361056321,0 request:: '/brokers/ids/3,F response::
>
> #626c2d71756575652d7330332e76697375616c646e612e636f6d2d313339343739363135373932383a626c2d71756575652d7330332e76697375616c646e612e636f6d3a39303932,s{150346957985,150346957985,1394796158123,1394796158123,0,0,0,163411213060484448,72,0,150346957985}
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:32,715] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> exception during rebalance (kafka.consumer.ZookeeperConsumerConnector)
> kafka.common.KafkaException: Failed to parse the broker info from
> zookeeper: bl-queue-s03.visualdna.com-1394796157928:
> bl-queue-s03.visualdna.com:9092
> at kafka.cluster.Broker$.createBroker(Broker.scala:45)
> at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:506)
> at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:504)
> at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> at
>
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
> at
>
> scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
> at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:504)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:407)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:402)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> Caused by: kafka.common.BrokerNotAvailableException: Broker id 3 does not
> exist
> at kafka.cluster.Broker$.createBroker(Broker.scala:42)
> ... 20 more
> [2014-03-19 17:35:32,720] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
> rebalancing consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #1
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:32,720] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> Rebalancing attempt failed. Clearing the cache before the next rebalancing
> operation is triggered (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:32,721] INFO [ConsumerFetcherManager-1395250530084]
> Stopping leader finder thread (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:32,721] INFO [ConsumerFetcherManager-1395250530084]
> Stopping all fetchers (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:32,721] INFO [ConsumerFetcherManager-1395250530084] All
> connections stopped (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:32,722] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
> all relevant queues for this fetcher
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:32,722] DEBUG Clearing the current data chunk for this
> consumer iterator (kafka.consumer.ConsumerIterator)
> [2014-03-19 17:35:32,722] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
> the data chunks in all the consumer message iterators
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:32,723] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> Committing all offsets after clearing the fetcher queues
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:34,705] DEBUG Got ping response for sessionid:
> 0x3448da1f8f0644b after 6ms (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:34,724] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
> rebalancing consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #2
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:34,732] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 11,8
> replyHeader:: 11,150361056365,0 request:: '/brokers/ids,F response::
> v{'3,'2,'1} (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:34,739] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 12,4
> replyHeader:: 12,150361056365,0 request:: '/brokers/ids/3,F response::
>
> #626c2d71756575652d7330332e76697375616c646e612e636f6d2d313339343739363135373932383a626c2d71756575652d7330332e76697375616c646e612e636f6d3a39303932,s{150346957985,150346957985,1394796158123,1394796158123,0,0,0,163411213060484448,72,0,150346957985}
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:34,741] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> exception during rebalance (kafka.consumer.ZookeeperConsumerConnector)
> kafka.common.KafkaException: Failed to parse the broker info from
> zookeeper: bl-queue-s03.visualdna.com-1394796157928:
> bl-queue-s03.visualdna.com:9092
> at kafka.cluster.Broker$.createBroker(Broker.scala:45)
> at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:506)
> at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:504)
> at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> at
>
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
> at
>
> scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
> at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:504)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:407)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:402)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> Caused by: kafka.common.BrokerNotAvailableException: Broker id 3 does not
> exist
> at kafka.cluster.Broker$.createBroker(Broker.scala:42)
> ... 20 more
> [2014-03-19 17:35:34,744] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
> rebalancing consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #2
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:34,744] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> Rebalancing attempt failed. Clearing the cache before the next rebalancing
> operation is triggered (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:34,745] INFO [ConsumerFetcherManager-1395250530084]
> Stopping leader finder thread (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:34,745] INFO [ConsumerFetcherManager-1395250530084]
> Stopping all fetchers (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:34,746] INFO [ConsumerFetcherManager-1395250530084] All
> connections stopped (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:34,746] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
> all relevant queues for this fetcher
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:34,746] DEBUG Clearing the current data chunk for this
> consumer iterator (kafka.consumer.ConsumerIterator)
> [2014-03-19 17:35:34,747] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
> the data chunks in all the consumer message iterators
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:34,747] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> Committing all offsets after clearing the fetcher queues
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:36,744] DEBUG Got ping response for sessionid:
> 0x3448da1f8f0644b after 6ms (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:36,749] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
> rebalancing consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #3
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:36,762] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 13,8
> replyHeader:: 13,150361056417,0 request:: '/brokers/ids,F response::
> v{'3,'2,'1} (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:36,782] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
> packet:: clientPath:null serverPath:null finished:false header:: 14,4
> replyHeader:: 14,150361056418,0 request:: '/brokers/ids/3,F response::
>
> #626c2d71756575652d7330332e76697375616c646e612e636f6d2d313339343739363135373932383a626c2d71756575652d7330332e76697375616c646e612e636f6d3a39303932,s{150346957985,150346957985,1394796158123,1394796158123,0,0,0,163411213060484448,72,0,150346957985}
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:36,785] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> exception during rebalance (kafka.consumer.ZookeeperConsumerConnector)
> kafka.common.KafkaException: Failed to parse the broker info from
> zookeeper: bl-queue-s03.visualdna.com-1394796157928:
> bl-queue-s03.visualdna.com:9092
> at kafka.cluster.Broker$.createBroker(Broker.scala:45)
> at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:506)
> at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:504)
> at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> at
>
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
> at
>
> scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
> at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:504)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:407)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:81)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:402)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> Caused by: kafka.common.BrokerNotAvailableException: Broker id 3 does not
> exist
> at kafka.cluster.Broker$.createBroker(Broker.scala:42)
> ... 20 more
> [2014-03-19 17:35:36,788] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
> rebalancing consumer
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #3
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:36,789] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> Rebalancing attempt failed. Clearing the cache before the next rebalancing
> operation is triggered (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:36,789] INFO [ConsumerFetcherManager-1395250530084]
> Stopping leader finder thread (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:36,789] INFO [ConsumerFetcherManager-1395250530084]
> Stopping all fetchers (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:36,789] INFO [ConsumerFetcherManager-1395250530084] All
> connections stopped (kafka.consumer.ConsumerFetcherManager)
> [2014-03-19 17:35:36,790] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
> all relevant queues for this fetcher
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:36,790] DEBUG Clearing the current data chunk for this
> consumer iterator (kafka.consumer.ConsumerIterator)
> [2014-03-19 17:35:36,790] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
> the data chunks in all the consumer message iterators
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:36,790] INFO
> [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
> Committing all offsets after clearing the fetcher queues
> (kafka.consumer.ZookeeperConsumerConnector)
> [2014-03-19 17:35:38,779] DEBUG Got ping response for sessionid:
> 0x3448da1f8f0644b after 5ms (org.apache.zookeeper.ClientCnxn)
> Kafka migration tool failed due to:
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> Caused by: kafka.common.ConsumerRebalanceFailedException:
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce can't
> rebalance after 4 retries
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> ... 5 more
>
> [2014-03-19 17:35:38,799] ERROR Kafka migration tool failed:
> (kafka.tools.KafkaMigrationTool)
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> Caused by: kafka.common.ConsumerRebalanceFailedException:
> mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce can't
> rebalance after 4 retries
> at
>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> at
>
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> at
>
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> at
>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> ... 5 more
> [2014-03-19 17:35:40,783] DEBUG Got ping response for sessionid:
> 0x3448da1f8f0644b after 7ms (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:42,786] DEBUG Got ping response for sessionid:
> 0x3448da1f8f0644b after 7ms (org.apache.zookeeper.ClientCnxn)
> [2014-03-19 17:35:44,787] DEBUG Got ping response for sessionid:
> 0x3448da1f8f0644b after 6ms (org.apache.zookeeper.ClientCnxn)
>
> Thanks,
> Mo.
>
> On 19 March 2014 14:07, Mo Firouz <Mo...@visualdna.com> wrote:
>
> > Hello.
> >
> > I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this
> > is to use the Migration Tool.
> >
> > I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local
> > Vagrant (along with a zookeeper running locally). The Migration Tool
> > Producer will connect to these three nodes in order to replicate the
> data.
> > The Migration Tool Consumer(s) will connect to my Staging boxes to
> consume
> > Kafka 0.7 events.
> >
> > However, upon running the Migration Tool, I'll get the following error:
> >
> > *vagrant@precise64:~/Desktop$
> >> ~/Desktop/kafka_2.9.2-0.8.1/bin/kafka-run-class.sh
> >> kafka.tools.KafkaMigrationTool --kafka.07.jar kafka-0.7.19.jar
> >> --zkclient.01.jar zkclient-0.2.0.jar --num.producers 16
> >> --producer.config=/home/vagrant/Deskt/producerConfig.properties
> >> --consumer.config=/home/vagrant/Desktop/consumerConfig.properties
> >> --whitelist "prod_conversions"*
> >
> > [2014-03-19 13:14:38,126] WARN Property mirror.topics.whitelist is not
> >> valid (kafka.utils.VerifiableProperties)
> >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >> SLF4J: Defaulting to no-operation (NOP) logger implementation
> >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> >> further details.
> >> Kafka migration tool failed due to:
> >> java.lang.reflect.InvocationTargetException
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> >> Caused by: kafka.common.ConsumerRebalanceFailedException:
> >> mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
> >> rebalance after 4 retries
> >> at
> >>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> >> at
> >>
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> >> at
> >>
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> >> at
> >>
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> >> at
> >>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> >> at
> >>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> >> ... 5 more
> >> [2014-03-19 13:14:47,082] ERROR Kafka migration tool failed:
> >> (kafka.tools.KafkaMigrationTool)
> >> java.lang.reflect.InvocationTargetException
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> >> Caused by: kafka.common.ConsumerRebalanceFailedException:
> >> mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
> >> rebalance after 4 retries
> >> at
> >>
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> >> at
> >>
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> >> at
> >>
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> >> at
> >>
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> >> at
> >>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> >> at
> >>
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> >> ... 5 more
> >
> >
> > My 3 *Kafka server* configurations are as follows:
> >
> > *cat kafka-server1.properties *
> >> zookeeper.connect=localhost:2181
> >> zookeeper.connection.timeout.ms=1000000
> >> broker.id=1
> >> port=9092
> >> log.dir=/tmp/kafka-logs-1
> >
> >
> >
> >>
> >> * cat kafka-server2.properties *
> >> zookeeper.connect=localhost:2181
> >> zookeeper.connection.timeout.ms=1000000
> >> broker.id=1
> >> port=9093
> >> log.dir=/tmp/kafka-logs-2
> >
> >
> >
> >> *cat kafka-server3.properties *
> >> zookeeper.connect=localhost:2181
> >> zookeeper.connection.timeout.ms=1000000
> >> broker.id=1
> >> port=9094
> >> log.dir=/tmp/kafka-logs-3
> >
> >
> > My *consumer* configuration is:
> >
> > *cat consumerConfig.properties *
> >> group.id = mfirouz-vagrant-kafkamigration
> >> zookeeper.connect = zk-01.s.visualdna.com:2181,
> zk-02.s.visualdna.com:2181
> >> ,zk-03.s.visualdna.com:2181
> >> mirror.topics.whitelist = "prod_conversions"
> >
> >
> > And finally, my *producer* configuration is:
> >
> > *cat producerConfig.properties*
> >> zk.connect = localhost:2181
> >> whitelist = "prod_conversions"
> >> producer.type = "sync"
> >> request.required.acks = 1
> >> metadata.broker.list = localhost:9092,localhost:9093,localhost:9094
> >
> >
> > I was wondering if anyone had any insight about this and how to fix it?
> >
> > Thanks a lot in advance
> > Mo.
> >
>
Re: Migration Tool consumer rebalance error when attempting to
migrate from Kafka 0.7 to Kafka 0.8
Posted by Mo Firouz <Mo...@visualdna.com>.
Hi Jun,
Thanks for the reply.
Please find below the DEBUG output of the log.
*vagrant@precise64:~/Desktop$
~/Desktop/kafka_2.9.2-0.8.1/bin/kafka-run-class.sh
kafka.tools.KafkaMigrationTool --kafka.07.jar kafka-0.7.19.jar
--zkclient.01.jar zkclient-0.2.0.jar --num.producers 16
--producer.config=/home/vagrant/Desktop/producerConfig.properties
--consumer.config=/home/vagrant/Desktop/consumerConfig.properties
--whitelist "prod_conversions"*
[2014-03-19 17:35:29,856] INFO Verifying properties
(kafka.utils.VerifiableProperties)
[2014-03-19 17:35:29,925] INFO Property group.id is overridden to
mfirouz-vagrant-kafkamigration (kafka.utils.VerifiableProperties)
[2014-03-19 17:35:29,926] WARN Property mirror.topics.whitelist is not
valid (kafka.utils.VerifiableProperties)
[2014-03-19 17:35:29,927] INFO Property zookeeper.connect is overridden to
zookeeper-01.stag.visualdna.com:2181,zookeeper-02.stag.visualdna.com:2181,
zookeeper-03.stag.visualdna.com:2181 (kafka.utils.VerifiableProperties)
[2014-03-19 17:35:29,952] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
Connecting to zookeeper instance at zookeeper-01.stag.visualdna.com:2181,
zookeeper-02.stag.visualdna.com:2181,zookeeper-03.stag.visualdna.com:2181(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:29,969] DEBUG Creating new ZookKeeper instance to connect
to zookeeper-01.stag.visualdna.com:2181,zookeeper-02.stag.visualdna.com:2181
,zookeeper-03.stag.visualdna.com:2181. (org.I0Itec.zkclient.ZkConnection)
[2014-03-19 17:35:29,974] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2014-03-19 17:35:29,982] INFO Client
environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,982] INFO Client environment:host.name=precise64
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,982] INFO Client environment:java.version=1.7.0_51
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,982] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,983] INFO Client
environment:java.home=/opt/java/jdk1.7.0_51/jre
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,983] INFO Client
environment:java.class.path=:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../core/build/dependant-libs-2.8.0/*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../clients/build/libs//kafka-clients*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../examples/build/libs//kafka-examples*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/jopt-simple-3.2.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/kafka_2.9.2-0.8.1.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/log4j-1.2.15.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/metrics-annotation-2.2.0.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/metrics-core-2.2.0.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/scala-library-2.9.2.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/slf4j-api-1.7.2.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/snappy-java-1.0.5.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/zkclient-0.3.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../libs/zookeeper-3.3.4.jar:/home/vagrant/Desktop/kafka_2.9.2-0.8.1/bin/../core/build/libs/kafka_2.8.0*.jar
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,983] INFO Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,984] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,985] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,985] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,985] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,986] INFO Client
environment:os.version=3.2.0-23-generic (org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,986] INFO Client environment:user.name=vagrant
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,987] INFO Client environment:user.home=/home/vagrant
(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,987] INFO Client
environment:user.dir=/home/vagrant/Desktop (org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,990] INFO Initiating client connection, connectString=
zookeeper-01.stag.visualdna.com:2181,zookeeper-02.stag.visualdna.com:2181,
zookeeper-03.stag.visualdna.com:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@50e7a7f2(org.apache.zookeeper.ZooKeeper)
[2014-03-19 17:35:29,993] DEBUG zookeeper.disableAutoWatchReset is false
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,027] DEBUG Awaiting connection to Zookeeper server
(org.I0Itec.zkclient.ZkClient)
[2014-03-19 17:35:30,027] INFO Opening socket connection to server
zookeeper-03.stag.visualdna.com/10.200.10.63:2181(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,027] DEBUG Waiting for keeper state SyncConnected
(org.I0Itec.zkclient.ZkClient)
[2014-03-19 17:35:30,042] INFO Socket connection established to
zookeeper-03.stag.visualdna.com/10.200.10.63:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,047] DEBUG Session establishment request sent on
zookeeper-03.stag.visualdna.com/10.200.10.63:2181(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,068] INFO Session establishment complete on server
zookeeper-03.stag.visualdna.com/10.200.10.63:2181, sessionid =
0x3448da1f8f0644b, negotiated timeout = 6000
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,072] DEBUG Received event: WatchedEvent
state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2014-03-19 17:35:30,073] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2014-03-19 17:35:30,073] DEBUG Leaving process event
(org.I0Itec.zkclient.ZkClient)
[2014-03-19 17:35:30,073] DEBUG State is SyncConnected
(org.I0Itec.zkclient.ZkClient)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
[2014-03-19 17:35:30,111] DEBUG Initializing task scheduler.
(kafka.utils.KafkaScheduler)
[2014-03-19 17:35:30,114] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], starting
auto committer every 60000 ms (kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,118] DEBUG Scheduling task kafka-consumer-autocommit
with initial delay 60000 ms and period 60000 ms.
(kafka.utils.KafkaScheduler)
[2014-03-19 17:35:30,210] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 1,8
replyHeader:: 1,150361056246,0 request:: '/brokers/topics,F response::
v{'profileupdates_gsantoro,'prod_interactions,'datasync,'audiencemembership,'prod_profileupdates,'audienceactivity,'dataexport,'prod_useractivity,'dataexport-gsantoro,'browsingactivity,'tracking_events,'profileactivity,'prod_conversions,'prod_pageviews,'prod_datasync,'prod_dataexport,'profileupdates,'pageviews,'useractivity,'conversions,'prod_profileactivity,'interactions,'quizactivity}
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,221] DEBUG profileupdates_gsantoro filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,222] DEBUG prod_interactions filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,222] DEBUG datasync filtered (kafka.consumer.Whitelist)
[2014-03-19 17:35:30,223] DEBUG audiencemembership filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,224] DEBUG prod_profileupdates filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,224] DEBUG audienceactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,225] DEBUG dataexport filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,225] DEBUG prod_useractivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,225] DEBUG dataexport-gsantoro filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,226] DEBUG browsingactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,226] DEBUG tracking_events filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,227] DEBUG profileactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,227] DEBUG prod_conversions allowed
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,229] DEBUG prod_pageviews filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,230] DEBUG prod_datasync filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,230] DEBUG prod_dataexport filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,231] DEBUG profileupdates filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,231] DEBUG pageviews filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,232] DEBUG useractivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,232] DEBUG conversions filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,233] DEBUG prod_profileactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,234] DEBUG interactions filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,234] DEBUG quizactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,252] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
registering consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce in ZK
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,409] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 2,1
replyHeader:: 2,150361056248,0 request::
'/consumers/mfirouz-vagrant-kafkamigration/ids/mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b2270726f645f636f6e76657273696f6e73223a317d2c227061747465726e223a2277686974655f6c697374222c2274696d657374616d70223a2231333935323530353330323537227d,v{s{31,s{'world,'anyone}}},1
response::
'/consumers/mfirouz-vagrant-kafkamigration/ids/mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,410] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
registering consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce in ZK
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,427] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], starting
watcher executor thread for consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,433] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 3,8
replyHeader:: 3,150361056250,0 request:: '/brokers/topics,F response::
v{'profileupdates_gsantoro,'prod_interactions,'datasync,'audiencemembership,'prod_profileupdates,'audienceactivity,'dataexport,'prod_useractivity,'dataexport-gsantoro,'browsingactivity,'tracking_events,'profileactivity,'prod_conversions,'prod_pageviews,'prod_datasync,'prod_dataexport,'profileupdates,'pageviews,'useractivity,'conversions,'prod_profileactivity,'interactions,'quizactivity}
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,437] DEBUG profileupdates_gsantoro filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,437] DEBUG prod_interactions filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,437] DEBUG datasync filtered (kafka.consumer.Whitelist)
[2014-03-19 17:35:30,438] DEBUG audiencemembership filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,438] DEBUG prod_profileupdates filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,438] DEBUG audienceactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,438] DEBUG dataexport filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,439] DEBUG prod_useractivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,439] DEBUG dataexport-gsantoro filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,439] DEBUG browsingactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,439] DEBUG tracking_events filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,440] DEBUG profileactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,440] DEBUG prod_conversions allowed
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,440] DEBUG prod_pageviews filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,441] DEBUG prod_datasync filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,441] DEBUG prod_dataexport filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,441] DEBUG profileupdates filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,441] DEBUG pageviews filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,441] DEBUG useractivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,442] DEBUG conversions filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,442] DEBUG prod_profileactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,442] DEBUG interactions filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,442] DEBUG quizactivity filtered
(kafka.consumer.Whitelist)
[2014-03-19 17:35:30,491] DEBUG
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Adding
topicThreadId
(prod_conversions,mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce-0)
and queue [] to topicThreadIdAndQueues data structure
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,499] DEBUG
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], adding
topic prod_conversions and 1 streams to map.
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,517] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 4,3
replyHeader:: 4,150361056251,0 request::
'/consumers/mfirouz-vagrant-kafkamigration/ids,T response::
s{150358573750,150358573750,1395165000820,1395165000820,0,35,0,0,0,1,150361056248}
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,525] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 5,8
replyHeader:: 5,150361056251,0 request::
'/consumers/mfirouz-vagrant-kafkamigration/ids,T response::
v{'mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce}
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,536] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 6,3
replyHeader:: 6,150361056251,0 request::
'/brokers/topics/prod_conversions,T response::
s{42963010923,42963010923,1363690601510,1363690601510,0,883,0,0,0,3,150357230474}
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,536] DEBUG Subscribed data changes for
/brokers/topics/prod_conversions (org.I0Itec.zkclient.ZkClient)
[2014-03-19 17:35:30,542] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
rebalancing consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #0
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,550] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 7,8
replyHeader:: 7,150361056251,0 request:: '/brokers/ids,F response::
v{'3,'2,'1} (org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,563] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 8,4
replyHeader:: 8,150361056251,0 request:: '/brokers/ids/3,F response::
#626c2d71756575652d7330332e76697375616c646e612e636f6d2d313339343739363135373932383a626c2d71756575652d7330332e76697375616c646e612e636f6d3a39303932,s{150346957985,150346957985,1394796158123,1394796158123,0,0,0,163411213060484448,72,0,150346957985}
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:30,649] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
exception during rebalance (kafka.consumer.ZookeeperConsumerConnector)
kafka.common.KafkaException: Failed to parse the broker info from
zookeeper: bl-queue-s03.visualdna.com-1394796157928:
bl-queue-s03.visualdna.com:9092
at kafka.cluster.Broker$.createBroker(Broker.scala:45)
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:506)
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:504)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at
scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
at
scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:504)
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:407)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:402)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
Caused by: kafka.common.BrokerNotAvailableException: Broker id 3 does not
exist
at kafka.cluster.Broker$.createBroker(Broker.scala:42)
... 20 more
[2014-03-19 17:35:30,669] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
rebalancing consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #0
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,670] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
Rebalancing attempt failed. Clearing the cache before the next rebalancing
operation is triggered (kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,674] INFO [ConsumerFetcherManager-1395250530084]
Stopping leader finder thread (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:30,675] INFO [ConsumerFetcherManager-1395250530084]
Stopping all fetchers (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:30,677] INFO [ConsumerFetcherManager-1395250530084] All
connections stopped (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:30,679] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
all relevant queues for this fetcher
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,681] DEBUG Clearing the current data chunk for this
consumer iterator (kafka.consumer.ConsumerIterator)
[2014-03-19 17:35:30,682] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
the data chunks in all the consumer message iterators
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:30,683] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
Committing all offsets after clearing the fetcher queues
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:32,603] DEBUG Got ping response for sessionid:
0x3448da1f8f0644b after 41ms (org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:32,687] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
rebalancing consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #1
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:32,693] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 9,8
replyHeader:: 9,150361056321,0 request:: '/brokers/ids,F response::
v{'3,'2,'1} (org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:32,712] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 10,4
replyHeader:: 10,150361056321,0 request:: '/brokers/ids/3,F response::
#626c2d71756575652d7330332e76697375616c646e612e636f6d2d313339343739363135373932383a626c2d71756575652d7330332e76697375616c646e612e636f6d3a39303932,s{150346957985,150346957985,1394796158123,1394796158123,0,0,0,163411213060484448,72,0,150346957985}
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:32,715] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
exception during rebalance (kafka.consumer.ZookeeperConsumerConnector)
kafka.common.KafkaException: Failed to parse the broker info from
zookeeper: bl-queue-s03.visualdna.com-1394796157928:
bl-queue-s03.visualdna.com:9092
at kafka.cluster.Broker$.createBroker(Broker.scala:45)
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:506)
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:504)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at
scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
at
scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:504)
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:407)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:402)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
Caused by: kafka.common.BrokerNotAvailableException: Broker id 3 does not
exist
at kafka.cluster.Broker$.createBroker(Broker.scala:42)
... 20 more
[2014-03-19 17:35:32,720] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
rebalancing consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #1
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:32,720] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
Rebalancing attempt failed. Clearing the cache before the next rebalancing
operation is triggered (kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:32,721] INFO [ConsumerFetcherManager-1395250530084]
Stopping leader finder thread (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:32,721] INFO [ConsumerFetcherManager-1395250530084]
Stopping all fetchers (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:32,721] INFO [ConsumerFetcherManager-1395250530084] All
connections stopped (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:32,722] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
all relevant queues for this fetcher
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:32,722] DEBUG Clearing the current data chunk for this
consumer iterator (kafka.consumer.ConsumerIterator)
[2014-03-19 17:35:32,722] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
the data chunks in all the consumer message iterators
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:32,723] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
Committing all offsets after clearing the fetcher queues
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:34,705] DEBUG Got ping response for sessionid:
0x3448da1f8f0644b after 6ms (org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:34,724] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
rebalancing consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #2
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:34,732] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 11,8
replyHeader:: 11,150361056365,0 request:: '/brokers/ids,F response::
v{'3,'2,'1} (org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:34,739] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 12,4
replyHeader:: 12,150361056365,0 request:: '/brokers/ids/3,F response::
#626c2d71756575652d7330332e76697375616c646e612e636f6d2d313339343739363135373932383a626c2d71756575652d7330332e76697375616c646e612e636f6d3a39303932,s{150346957985,150346957985,1394796158123,1394796158123,0,0,0,163411213060484448,72,0,150346957985}
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:34,741] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
exception during rebalance (kafka.consumer.ZookeeperConsumerConnector)
kafka.common.KafkaException: Failed to parse the broker info from
zookeeper: bl-queue-s03.visualdna.com-1394796157928:
bl-queue-s03.visualdna.com:9092
at kafka.cluster.Broker$.createBroker(Broker.scala:45)
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:506)
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:504)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at
scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
at
scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:504)
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:407)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:402)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
Caused by: kafka.common.BrokerNotAvailableException: Broker id 3 does not
exist
at kafka.cluster.Broker$.createBroker(Broker.scala:42)
... 20 more
[2014-03-19 17:35:34,744] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
rebalancing consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #2
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:34,744] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
Rebalancing attempt failed. Clearing the cache before the next rebalancing
operation is triggered (kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:34,745] INFO [ConsumerFetcherManager-1395250530084]
Stopping leader finder thread (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:34,745] INFO [ConsumerFetcherManager-1395250530084]
Stopping all fetchers (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:34,746] INFO [ConsumerFetcherManager-1395250530084] All
connections stopped (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:34,746] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
all relevant queues for this fetcher
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:34,746] DEBUG Clearing the current data chunk for this
consumer iterator (kafka.consumer.ConsumerIterator)
[2014-03-19 17:35:34,747] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
the data chunks in all the consumer message iterators
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:34,747] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
Committing all offsets after clearing the fetcher queues
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:36,744] DEBUG Got ping response for sessionid:
0x3448da1f8f0644b after 6ms (org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:36,749] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], begin
rebalancing consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #3
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:36,762] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 13,8
replyHeader:: 13,150361056417,0 request:: '/brokers/ids,F response::
v{'3,'2,'1} (org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:36,782] DEBUG Reading reply sessionid:0x3448da1f8f0644b,
packet:: clientPath:null serverPath:null finished:false header:: 14,4
replyHeader:: 14,150361056418,0 request:: '/brokers/ids/3,F response::
#626c2d71756575652d7330332e76697375616c646e612e636f6d2d313339343739363135373932383a626c2d71756575652d7330332e76697375616c646e612e636f6d3a39303932,s{150346957985,150346957985,1394796158123,1394796158123,0,0,0,163411213060484448,72,0,150346957985}
(org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:36,785] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
exception during rebalance (kafka.consumer.ZookeeperConsumerConnector)
kafka.common.KafkaException: Failed to parse the broker info from
zookeeper: bl-queue-s03.visualdna.com-1394796157928:
bl-queue-s03.visualdna.com:9092
at kafka.cluster.Broker$.createBroker(Broker.scala:45)
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:506)
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:504)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at
scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
at
scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:504)
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:407)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:81)
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:402)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
Caused by: kafka.common.BrokerNotAvailableException: Broker id 3 does not
exist
at kafka.cluster.Broker$.createBroker(Broker.scala:42)
... 20 more
[2014-03-19 17:35:36,788] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], end
rebalancing consumer
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce try #3
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:36,789] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
Rebalancing attempt failed. Clearing the cache before the next rebalancing
operation is triggered (kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:36,789] INFO [ConsumerFetcherManager-1395250530084]
Stopping leader finder thread (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:36,789] INFO [ConsumerFetcherManager-1395250530084]
Stopping all fetchers (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:36,789] INFO [ConsumerFetcherManager-1395250530084] All
connections stopped (kafka.consumer.ConsumerFetcherManager)
[2014-03-19 17:35:36,790] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
all relevant queues for this fetcher
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:36,790] DEBUG Clearing the current data chunk for this
consumer iterator (kafka.consumer.ConsumerIterator)
[2014-03-19 17:35:36,790] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], Cleared
the data chunks in all the consumer message iterators
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:36,790] INFO
[mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce],
Committing all offsets after clearing the fetcher queues
(kafka.consumer.ZookeeperConsumerConnector)
[2014-03-19 17:35:38,779] DEBUG Got ping response for sessionid:
0x3448da1f8f0644b after 5ms (org.apache.zookeeper.ClientCnxn)
Kafka migration tool failed due to:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
Caused by: kafka.common.ConsumerRebalanceFailedException:
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce can't
rebalance after 4 retries
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
... 5 more
[2014-03-19 17:35:38,799] ERROR Kafka migration tool failed:
(kafka.tools.KafkaMigrationTool)
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
Caused by: kafka.common.ConsumerRebalanceFailedException:
mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce can't
rebalance after 4 retries
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
... 5 more
[2014-03-19 17:35:40,783] DEBUG Got ping response for sessionid:
0x3448da1f8f0644b after 7ms (org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:42,786] DEBUG Got ping response for sessionid:
0x3448da1f8f0644b after 7ms (org.apache.zookeeper.ClientCnxn)
[2014-03-19 17:35:44,787] DEBUG Got ping response for sessionid:
0x3448da1f8f0644b after 6ms (org.apache.zookeeper.ClientCnxn)
Thanks,
Mo.
On 19 March 2014 14:07, Mo Firouz <Mo...@visualdna.com> wrote:
> Hello.
>
> I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this
> is to use the Migration Tool.
>
> I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local
> Vagrant (along with a zookeeper running locally). The Migration Tool
> Producer will connect to these three nodes in order to replicate the data.
> The Migration Tool Consumer(s) will connect to my Staging boxes to consume
> Kafka 0.7 events.
>
> However, upon running the Migration Tool, I'll get the following error:
>
> *vagrant@precise64:~/Desktop$
>> ~/Desktop/kafka_2.9.2-0.8.1/bin/kafka-run-class.sh
>> kafka.tools.KafkaMigrationTool --kafka.07.jar kafka-0.7.19.jar
>> --zkclient.01.jar zkclient-0.2.0.jar --num.producers 16
>> --producer.config=/home/vagrant/Deskt/producerConfig.properties
>> --consumer.config=/home/vagrant/Desktop/consumerConfig.properties
>> --whitelist "prod_conversions"*
>
> [2014-03-19 13:14:38,126] WARN Property mirror.topics.whitelist is not
>> valid (kafka.utils.VerifiableProperties)
>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further details.
>> Kafka migration tool failed due to:
>> java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
>> Caused by: kafka.common.ConsumerRebalanceFailedException:
>> mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
>> rebalance after 4 retries
>> at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
>> at
>> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
>> at
>> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
>> at
>> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
>> at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
>> at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
>> ... 5 more
>> [2014-03-19 13:14:47,082] ERROR Kafka migration tool failed:
>> (kafka.tools.KafkaMigrationTool)
>> java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
>> Caused by: kafka.common.ConsumerRebalanceFailedException:
>> mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
>> rebalance after 4 retries
>> at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
>> at
>> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
>> at
>> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
>> at
>> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
>> at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
>> at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
>> ... 5 more
>
>
> My 3 *Kafka server* configurations are as follows:
>
> *cat kafka-server1.properties *
>> zookeeper.connect=localhost:2181
>> zookeeper.connection.timeout.ms=1000000
>> broker.id=1
>> port=9092
>> log.dir=/tmp/kafka-logs-1
>
>
>
>>
>> * cat kafka-server2.properties *
>> zookeeper.connect=localhost:2181
>> zookeeper.connection.timeout.ms=1000000
>> broker.id=1
>> port=9093
>> log.dir=/tmp/kafka-logs-2
>
>
>
>> *cat kafka-server3.properties *
>> zookeeper.connect=localhost:2181
>> zookeeper.connection.timeout.ms=1000000
>> broker.id=1
>> port=9094
>> log.dir=/tmp/kafka-logs-3
>
>
> My *consumer* configuration is:
>
> *cat consumerConfig.properties *
>> group.id = mfirouz-vagrant-kafkamigration
>> zookeeper.connect = zk-01.s.visualdna.com:2181,zk-02.s.visualdna.com:2181
>> ,zk-03.s.visualdna.com:2181
>> mirror.topics.whitelist = "prod_conversions"
>
>
> And finally, my *producer* configuration is:
>
> *cat producerConfig.properties*
>> zk.connect = localhost:2181
>> whitelist = "prod_conversions"
>> producer.type = "sync"
>> request.required.acks = 1
>> metadata.broker.list = localhost:9092,localhost:9093,localhost:9094
>
>
> I was wondering if anyone had any insight about this and how to fix it?
>
> Thanks a lot in advance
> Mo.
>
Re: Migration Tool consumer rebalance error when attempting to
migrate from Kafka 0.7 to Kafka 0.8
Posted by Steven A Robenalt <sr...@stanford.edu>.
Hi Mo,
Is the broker id in your server properties really the same for all 3, or is
that a typo?
Steve
On Wed, Mar 19, 2014 at 8:10 AM, Jun Rao <ju...@gmail.com> wrote:
> Were there any logging before that? There should be some logging that tells
> you why the rebalance failed.
>
> Thanks,
>
> Jun
>
>
> On Wed, Mar 19, 2014 at 7:07 AM, Mo Firouz <Mo...@visualdna.com>
> wrote:
>
> > Hello.
> >
> > I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this
> > is to use the Migration Tool.
> >
> > I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local
> > Vagrant (along with a zookeeper running locally). The Migration Tool
> > Producer will connect to these three nodes in order to replicate the
> data.
> > The Migration Tool Consumer(s) will connect to my Staging boxes to
> consume
> > Kafka 0.7 events.
> >
> > However, upon running the Migration Tool, I'll get the following error:
> >
> > *vagrant@precise64:~/Desktop$
> > > ~/Desktop/kafka_2.9.2-0.8.1/bin/kafka-run-class.sh
> > > kafka.tools.KafkaMigrationTool --kafka.07.jar kafka-0.7.19.jar
> > > --zkclient.01.jar zkclient-0.2.0.jar --num.producers 16
> > > --producer.config=/home/vagrant/Deskt/producerConfig.properties
> > > --consumer.config=/home/vagrant/Desktop/consumerConfig.properties
> > > --whitelist "prod_conversions"*
> >
> > [2014-03-19 13:14:38,126] WARN Property mirror.topics.whitelist is not
> > > valid (kafka.utils.VerifiableProperties)
> > > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> > > SLF4J: Defaulting to no-operation (NOP) logger implementation
> > > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> > further
> > > details.
> > > Kafka migration tool failed due to:
> > > java.lang.reflect.InvocationTargetException
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> > > Caused by: kafka.common.ConsumerRebalanceFailedException:
> > > mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
> > > rebalance after 4 retries
> > > at
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> > > at
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> > > at
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> > > at
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> > > at
> > >
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> > > at
> > >
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> > > ... 5 more
> > > [2014-03-19 13:14:47,082] ERROR Kafka migration tool failed:
> > > (kafka.tools.KafkaMigrationTool)
> > > java.lang.reflect.InvocationTargetException
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> > > Caused by: kafka.common.ConsumerRebalanceFailedException:
> > > mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
> > > rebalance after 4 retries
> > > at
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> > > at
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> > > at
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> > > at
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> > > at
> > >
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> > > at
> > >
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> > > ... 5 more
> >
> >
> > My 3 *Kafka server* configurations are as follows:
> >
> > *cat kafka-server1.properties *
> > > zookeeper.connect=localhost:2181
> > > zookeeper.connection.timeout.ms=1000000
> > > broker.id=1
> > > port=9092
> > > log.dir=/tmp/kafka-logs-1
> >
> >
> >
> > >
> > > *cat kafka-server2.properties *
> > > zookeeper.connect=localhost:2181
> > > zookeeper.connection.timeout.ms=1000000
> > > broker.id=1
> > > port=9093
> > > log.dir=/tmp/kafka-logs-2
> >
> >
> >
> > > *cat kafka-server3.properties *
> > > zookeeper.connect=localhost:2181
> > > zookeeper.connection.timeout.ms=1000000
> > > broker.id=1
> > > port=9094
> > > log.dir=/tmp/kafka-logs-3
> >
> >
> > My *consumer* configuration is:
> >
> > *cat consumerConfig.properties *
> > > group.id = mfirouz-vagrant-kafkamigration
> > > zookeeper.connect = zk-01.s.visualdna.com:2181,
> > zk-02.s.visualdna.com:2181,
> > > zk-03.s.visualdna.com:2181
> > > mirror.topics.whitelist = "prod_conversions"
> >
> >
> > And finally, my *producer* configuration is:
> >
> > *cat producerConfig.properties*
> > > zk.connect = localhost:2181
> > > whitelist = "prod_conversions"
> > > producer.type = "sync"
> > > request.required.acks = 1
> > > metadata.broker.list = localhost:9092,localhost:9093,localhost:9094
> >
> >
> > I was wondering if anyone had any insight about this and how to fix it?
> >
> > Thanks a lot in advance
> > Mo.
> >
>
--
Steve Robenalt
Software Architect
HighWire | Stanford University
425 Broadway St, Redwood City, CA 94063
srobenal@stanford.edu
http://highwire.stanford.edu
Re: Migration Tool consumer rebalance error when attempting to
migrate from Kafka 0.7 to Kafka 0.8
Posted by Jun Rao <ju...@gmail.com>.
Were there any logging before that? There should be some logging that tells
you why the rebalance failed.
Thanks,
Jun
On Wed, Mar 19, 2014 at 7:07 AM, Mo Firouz <Mo...@visualdna.com> wrote:
> Hello.
>
> I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this
> is to use the Migration Tool.
>
> I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local
> Vagrant (along with a zookeeper running locally). The Migration Tool
> Producer will connect to these three nodes in order to replicate the data.
> The Migration Tool Consumer(s) will connect to my Staging boxes to consume
> Kafka 0.7 events.
>
> However, upon running the Migration Tool, I'll get the following error:
>
> *vagrant@precise64:~/Desktop$
> > ~/Desktop/kafka_2.9.2-0.8.1/bin/kafka-run-class.sh
> > kafka.tools.KafkaMigrationTool --kafka.07.jar kafka-0.7.19.jar
> > --zkclient.01.jar zkclient-0.2.0.jar --num.producers 16
> > --producer.config=/home/vagrant/Deskt/producerConfig.properties
> > --consumer.config=/home/vagrant/Desktop/consumerConfig.properties
> > --whitelist "prod_conversions"*
>
> [2014-03-19 13:14:38,126] WARN Property mirror.topics.whitelist is not
> > valid (kafka.utils.VerifiableProperties)
> > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> > SLF4J: Defaulting to no-operation (NOP) logger implementation
> > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> further
> > details.
> > Kafka migration tool failed due to:
> > java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> > Caused by: kafka.common.ConsumerRebalanceFailedException:
> > mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
> > rebalance after 4 retries
> > at
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> > at
> >
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> > at
> >
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> > at
> >
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> > at
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> > at
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> > ... 5 more
> > [2014-03-19 13:14:47,082] ERROR Kafka migration tool failed:
> > (kafka.tools.KafkaMigrationTool)
> > java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
> > Caused by: kafka.common.ConsumerRebalanceFailedException:
> > mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
> > rebalance after 4 retries
> > at
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> > at
> >
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> > at
> >
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
> > at
> >
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
> > at
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
> > at
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
> > ... 5 more
>
>
> My 3 *Kafka server* configurations are as follows:
>
> *cat kafka-server1.properties *
> > zookeeper.connect=localhost:2181
> > zookeeper.connection.timeout.ms=1000000
> > broker.id=1
> > port=9092
> > log.dir=/tmp/kafka-logs-1
>
>
>
> >
> > *cat kafka-server2.properties *
> > zookeeper.connect=localhost:2181
> > zookeeper.connection.timeout.ms=1000000
> > broker.id=1
> > port=9093
> > log.dir=/tmp/kafka-logs-2
>
>
>
> > *cat kafka-server3.properties *
> > zookeeper.connect=localhost:2181
> > zookeeper.connection.timeout.ms=1000000
> > broker.id=1
> > port=9094
> > log.dir=/tmp/kafka-logs-3
>
>
> My *consumer* configuration is:
>
> *cat consumerConfig.properties *
> > group.id = mfirouz-vagrant-kafkamigration
> > zookeeper.connect = zk-01.s.visualdna.com:2181,
> zk-02.s.visualdna.com:2181,
> > zk-03.s.visualdna.com:2181
> > mirror.topics.whitelist = "prod_conversions"
>
>
> And finally, my *producer* configuration is:
>
> *cat producerConfig.properties*
> > zk.connect = localhost:2181
> > whitelist = "prod_conversions"
> > producer.type = "sync"
> > request.required.acks = 1
> > metadata.broker.list = localhost:9092,localhost:9093,localhost:9094
>
>
> I was wondering if anyone had any insight about this and how to fix it?
>
> Thanks a lot in advance
> Mo.
>
Re: Migration Tool consumer rebalance error when attempting to
migrate from Kafka 0.7 to Kafka 0.8
Posted by Mo Firouz <Mo...@visualdna.com>.
Hi Jun,
Thanks for your response and help. Yes indeed BOTH the kafka and zookeeper
variations were different that was causing this error!
It's now working!
Thanks for you help and time!
Cheers,
Mo.
On 19 March 2014 14:07, Mo Firouz <Mo...@visualdna.com> wrote:
> Hello.
>
> I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this
> is to use the Migration Tool.
>
> I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local
> Vagrant (along with a zookeeper running locally). The Migration Tool
> Producer will connect to these three nodes in order to replicate the data.
> The Migration Tool Consumer(s) will connect to my Staging boxes to consume
> Kafka 0.7 events.
>
> However, upon running the Migration Tool, I'll get the following error:
>
> *vagrant@precise64:~/Desktop$
>> ~/Desktop/kafka_2.9.2-0.8.1/bin/kafka-run-class.sh
>> kafka.tools.KafkaMigrationTool --kafka.07.jar kafka-0.7.19.jar
>> --zkclient.01.jar zkclient-0.2.0.jar --num.producers 16
>> --producer.config=/home/vagrant/Deskt/producerConfig.properties
>> --consumer.config=/home/vagrant/Desktop/consumerConfig.properties
>> --whitelist "prod_conversions"*
>
> [2014-03-19 13:14:38,126] WARN Property mirror.topics.whitelist is not
>> valid (kafka.utils.VerifiableProperties)
>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further details.
>> Kafka migration tool failed due to:
>> java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
>> Caused by: kafka.common.ConsumerRebalanceFailedException:
>> mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
>> rebalance after 4 retries
>> at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
>> at
>> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
>> at
>> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
>> at
>> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
>> at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
>> at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
>> ... 5 more
>> [2014-03-19 13:14:47,082] ERROR Kafka migration tool failed:
>> (kafka.tools.KafkaMigrationTool)
>> java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217)
>> Caused by: kafka.common.ConsumerRebalanceFailedException:
>> mfirouz-vagrant-kafkamigration_precise64-1395234878153-3d6d9aaa can't
>> rebalance after 4 retries
>> at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
>> at
>> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
>> at
>> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:756)
>> at
>> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
>> at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:96)
>> at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100)
>> ... 5 more
>
>
> My 3 *Kafka server* configurations are as follows:
>
> *cat kafka-server1.properties *
>> zookeeper.connect=localhost:2181
>> zookeeper.connection.timeout.ms=1000000
>> broker.id=1
>> port=9092
>> log.dir=/tmp/kafka-logs-1
>
>
>
>>
>> * cat kafka-server2.properties *
>> zookeeper.connect=localhost:2181
>> zookeeper.connection.timeout.ms=1000000
>> broker.id=1
>> port=9093
>> log.dir=/tmp/kafka-logs-2
>
>
>
>> *cat kafka-server3.properties *
>> zookeeper.connect=localhost:2181
>> zookeeper.connection.timeout.ms=1000000
>> broker.id=1
>> port=9094
>> log.dir=/tmp/kafka-logs-3
>
>
> My *consumer* configuration is:
>
> *cat consumerConfig.properties *
>> group.id = mfirouz-vagrant-kafkamigration
>> zookeeper.connect = zk-01.s.visualdna.com:2181,zk-02.s.visualdna.com:2181
>> ,zk-03.s.visualdna.com:2181
>> mirror.topics.whitelist = "prod_conversions"
>
>
> And finally, my *producer* configuration is:
>
> *cat producerConfig.properties*
>> zk.connect = localhost:2181
>> whitelist = "prod_conversions"
>> producer.type = "sync"
>> request.required.acks = 1
>> metadata.broker.list = localhost:9092,localhost:9093,localhost:9094
>
>
> I was wondering if anyone had any insight about this and how to fix it?
>
> Thanks a lot in advance
> Mo.
>