Please clean your ZK and Kafka data if you check out 0.8 HEAD

[Swapnil Ghike <sg...@linkedin.com>]

Hi all,

Today we checked in a patch that standardizes Zookeeper data to use JSON format (KAFKA-755<https://issues.apache.org/jira/browse/KAFKA-755>). If you plan to rebase or clone to use the HEAD of 0.8 branch, your next build will have changes to the way Zk data is read and written. This data includes replica assignment, brokers' and controller's information, and consumer subscriptions.

For working off of 0.8 HEAD, you need to :
1. Stop ZK, Kafka brokers and all console consumers
2. Wipe out all ZK data and kafka log data
3. Upgrade kafka brokers and all console consumers to HEAD and restart them

Thanks,
Swapnil

Re: Please clean your ZK and Kafka data if you check out 0.8 HEAD

[Swapnil Ghike <sg...@linkedin.com>]

If you are using the console consumer from the 0.8/bin directory then it
should be automatically updated when you package the kafka jar. Is your
console consumer using a separate kafka jar that did not possibly get
updated when you did git pull?

Thanks,
Swapnil

On 2/28/13 12:04 AM, "ben fleis" <be...@monkey.org> wrote:

>Is it possible to upgrade one without upgrading the other?  I did a git
>pull, ./sbt update package.  Then for kicks I did sbt clean package, just
>to be sure.
>
>ben

Re: Please clean your ZK and Kafka data if you check out 0.8 HEAD

[ben fleis <be...@monkey.org>]

Is it possible to upgrade one without upgrading the other?  I did a git
pull, ./sbt update package.  Then for kicks I did sbt clean package, just
to be sure.

ben

Re: Please clean your ZK and Kafka data if you check out 0.8 HEAD

[Jun Rao <ju...@gmail.com>]

Did you upgrade your console consumer?

Thanks,

Jun

On Wed, Feb 27, 2013 at 3:40 AM, ben fleis <be...@monkey.org> wrote:

> Sorry, I should have clarified something -- using the normal
> ConsoleConsumer does *not* produce this error.  So something in my
> formatter, or in the logic of a custom formatter must change?
>
> ben
>

Re: Please clean your ZK and Kafka data if you check out 0.8 HEAD

[ben fleis <be...@monkey.org>]

Sorry, I should have clarified something -- using the normal
ConsoleConsumer does *not* produce this error.  So something in my
formatter, or in the logic of a custom formatter must change?

ben

Re: Please clean your ZK and Kafka data if you check out 0.8 HEAD

[ben fleis <be...@monkey.org>]

Hi Swapnil,

I have done exactly this (killed and wiped everything), and have found that
my custom formatter based ConsoleConsumer fails:

[2013-02-27 12:34:31,420] ERROR Error processing message, stopping
consumer:  (kafka.consumer.ConsoleConsumer$)
java.lang.NumberFormatException: For input string: ""192.168.2.132",
"jmx_port""
        at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
        at java.lang.Integer.parseInt(Integer.java:449)
        at java.lang.Integer.parseInt(Integer.java:499)
        at
scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
        at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
        at kafka.cluster.Broker$.createBroker(Broker.scala:34)
        at
kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:450)
        at
kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:448)
        at scala.collection.Iterator$class.foreach(Iterator.scala:631)
        at
scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474)
        at
scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
        at
scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:521)
        at kafka.utils.ZkUtils$.getCluster(ZkUtils.scala:448)
        at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:371)
        at
scala.collection.immutable.Range$ByOne$class.foreach$mVc$sp(Range.scala:282)
        at
scala.collection.immutable.Range$$anon$2.foreach$mVc$sp(Range.scala:265)
        at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:368)
        at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:697)
        at
kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:732)
        at
kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:141)
        at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:183)
        at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)

I haven't tracked down why the mismatch only occurs with my formatter, but
advice is appreciated.

ben