You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yifan Ying <na...@gmail.com> on 2016/07/29 07:04:44 UTC

Mirrormaker between 0.8.2.1 cluster and 0.10 cluster

Hi all,

I am trying to use the mirrormaker on the 0.10 cluster to mirror the
0.8.2.1 cluster into 0.10 cluster. Then I got a bunch of consumer errors as
follows:

 Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@f9533ee
(kafka.consumer.ConsumerFetcherThread)

java.nio.BufferUnderflowException

at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:145)

at java.nio.ByteBuffer.get(ByteBuffer.java:694)

at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:40)

at kafka.api.TopicData$.readFrom(FetchResponse.scala:96)

at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)

at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)

at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)

at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)

at scala.collection.immutable.Range.foreach(Range.scala:160)

at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)

at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)

at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)

at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)

at
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)

at
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)

at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)

at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)


Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@5d1339d8

java.lang.IllegalArgumentException

at java.nio.Buffer.limit(Buffer.java:267)

at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)

at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)

at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)

at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

at scala.collection.immutable.Range.foreach(Range.scala:141)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)

at scala.collection.AbstractTraversable.map(Traversable.scala:105)

at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)

at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)

at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)

at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)

at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)

at scala.collection.immutable.Range.foreach(Range.scala:141)

at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)

at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)

at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)

at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)

at
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)

at
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)

at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)

at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)

Is there any compatibility issue when using the new mirrormaker to mirror a
0.8.2.1 cluster into a 0.10 cluster?

Thanks.


-- 
Yifan

Re: Mirrormaker between 0.8.2.1 cluster and 0.10 cluster

Posted by Gwen Shapira <gw...@confluent.io>.
You need to use the old mirrormaker (0.8.2.1) to mirror 0.8.2.1 to 0.10.0.0.

This is true in general - always use MirrorMaker from the older release.
Because new Kafka can talk to old clients and not the other way around.

Gwen

On Fri, Jul 29, 2016 at 12:04 AM, Yifan Ying <na...@gmail.com> wrote:
> Hi all,
>
> I am trying to use the mirrormaker on the 0.10 cluster to mirror the
> 0.8.2.1 cluster into 0.10 cluster. Then I got a bunch of consumer errors as
> follows:
>
>  Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@f9533ee
> (kafka.consumer.ConsumerFetcherThread)
>
> java.nio.BufferUnderflowException
>
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:145)
>
> at java.nio.ByteBuffer.get(ByteBuffer.java:694)
>
> at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:40)
>
> at kafka.api.TopicData$.readFrom(FetchResponse.scala:96)
>
> at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
>
> at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
>
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>
> at scala.collection.immutable.Range.foreach(Range.scala:160)
>
> at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
>
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
>
> at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
>
> at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
>
> at
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
>
> at
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
>
> at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
>
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
>
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
>
>
> Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@5d1339d8
>
> java.lang.IllegalArgumentException
>
> at java.nio.Buffer.limit(Buffer.java:267)
>
> at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
>
> at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
>
> at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>
> at scala.collection.immutable.Range.foreach(Range.scala:141)
>
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>
> at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
>
> at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
>
> at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
>
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>
> at scala.collection.immutable.Range.foreach(Range.scala:141)
>
> at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
>
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
>
> at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
>
> at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
>
> at
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
>
> at
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
>
> at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
>
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
>
> Is there any compatibility issue when using the new mirrormaker to mirror a
> 0.8.2.1 cluster into a 0.10 cluster?
>
> Thanks.
>
>
> --
> Yifan