You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Samuel Zhou <zh...@gmail.com> on 2016/01/09 02:14:02 UTC
Memory records is not writable in MirrorMaker
Hi,
I found a problem that says Memory records is not writable in MirrorMaker
when producer sending out messages:
[2016-01-08 18:52:30,729] FATAL [mirrormaker-thread-32] Mirror maker thread
failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
java.lang.IllegalStateException: Memory records is not writable
at
org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
at
org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:439)
at
kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
The above message repeated several times and at last the MirrorMaker is
shutdown:
[2016-01-08 18:52:31,256] FATAL [mirrormaker-thread-59] Mirror maker thread
exited abnormally, stopping the whole mirror maker.
(kafka.tools.MirrorMaker$MirrorMakerThread)
MirrorMaker was shutdown around 30 minutes right after every time I started
it, it is using plain text protocol(No SSL), and the mirrormaker has
options of "--abort.on.send.failure false" and "--num.streams 100" . And it
is running with kafka 0.9 release. Does anybody know how to solve this
problem?
Thanks!
Samuel
Re: Memory records is not writable in MirrorMaker
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks.
Ismael
On Mon, Jan 25, 2016 at 8:28 PM, Meghana Narasimhan <
mnarasimhan@bandwidth.com> wrote:
> I have created a JIRA, https://issues.apache.org/jira/browse/KAFKA-3147
>
> Thanks,
> Meghana
>
> On Tue, Jan 19, 2016 at 5:07 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Can you please file an issue in JIRA for this?
> >
> > Ismael
> >
> > On Tue, Jan 12, 2016 at 2:40 PM, Meghana Narasimhan <
> > mnarasimhan@bandwidth.com> wrote:
> >
> > > Hi,
> > > Came across a similar issue. We are running a 3 node cluster (kafka
> > version
> > > 0.9) and Node 0 also has a few mirror makers running.
> > > When we do a rolling restart of the cluster, the mirror maker shuts
> down
> > > with the following errors.
> > >
> > > [2016-01-11 20:16:00,348] WARN Got error produce response with
> > correlation
> > > id 12491674 on topic-partition test-99, retrying (2147483646 attempts
> > > left). Error: NOT_LEADER_FOR_PARTITION
> > > (org.apache.kafka.clients.producer.internals.Sender)
> > > [2016-01-11 20:16:00,853] FATAL [mirrormaker-thread-0] Mirror maker
> > thread
> > > failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
> > > java.lang.IllegalStateException: Memory records is not writable
> > > at
> > >
> >
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:435)
> > > at
> > > kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
> > > at
> > >
> > >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > > at
> > >
> > >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > > at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > > at
> > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > > at
> > > kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:398)
> > > [2016-01-11 20:16:01,072] WARN Got error produce response with
> > correlation
> > > id 12491679 on topic-partition test-75, retrying (2147483646 attempts
> > > left). Error: NETWORK_EXCEPTION
> > > (org.apache.kafka.clients.producer.internals.Sender)
> > > [2016-01-11 20:16:01,073] WARN Got error produce response with
> > correlation
> > > id 12491679 on topic-partition test-93, retrying (2147483646 attempts
> > > left). Error: NETWORK_EXCEPTION
> > > (org.apache.kafka.clients.producer.internals.Sender)
> > > [2016-01-11 20:16:01,073] WARN Got error produce response with
> > correlation
> > > id 12491679 on topic-partition test-24, retrying (2147483646 attempts
> > > left). Error: NETWORK_EXCEPTION
> > > (org.apache.kafka.clients.producer.internals.Sender)
> > >
> > > [2016-01-11 20:16:20,479] FATAL [mirrormaker-thread-0] Mirror maker
> > thread
> > > exited abnormally, stopping the whole mirror maker.
> > > (kafka.tools.MirrorMaker$MirrorMakerThread)
> > >
> > >
> > > Curious if the NOT_LEADER_FOR_PARTITION is because of a potential bug
> > > hinted at in the thread ,
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201505.mbox/%3CCAJS3ho_u8s1Xou_kudNfjAMyPJtMrjLW10QVkNGn2YQkdan0+A@mail.gmail.com%3E
> > >
> > >
> > > And I think the mirror maker shuts down because of the
> > > "abort.on.send.failure" which is set to true in our case.
> > >
> > > Thanks,
> > > Meghana
> > >
> > > On Fri, Jan 8, 2016 at 8:14 PM, Samuel Zhou <zh...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I found a problem that says Memory records is not writable in
> > MirrorMaker
> > > > when producer sending out messages:
> > > >
> > > > [2016-01-08 18:52:30,729] FATAL [mirrormaker-thread-32] Mirror maker
> > > thread
> > > > failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
> > > > java.lang.IllegalStateException: Memory records is not writable
> > > > at
> > > >
> > >
> >
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:439)
> > > > at
> > > >
> kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > > > at
> scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > > > at
> > scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > > > at
> > > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > > at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > > >
> > > > The above message repeated several times and at last the MirrorMaker
> is
> > > > shutdown:
> > > >
> > > > [2016-01-08 18:52:31,256] FATAL [mirrormaker-thread-59] Mirror maker
> > > thread
> > > > exited abnormally, stopping the whole mirror maker.
> > > > (kafka.tools.MirrorMaker$MirrorMakerThread)
> > > >
> > > > MirrorMaker was shutdown around 30 minutes right after every time I
> > > started
> > > > it, it is using plain text protocol(No SSL), and the mirrormaker has
> > > > options of "--abort.on.send.failure false" and "--num.streams 100" .
> > And
> > > it
> > > > is running with kafka 0.9 release. Does anybody know how to solve
> this
> > > > problem?
> > > >
> > > > Thanks!
> > > >
> > > > Samuel
> > > >
> > >
> >
>
Re: Memory records is not writable in MirrorMaker
Posted by Meghana Narasimhan <mn...@bandwidth.com>.
I have created a JIRA, https://issues.apache.org/jira/browse/KAFKA-3147
Thanks,
Meghana
On Tue, Jan 19, 2016 at 5:07 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Can you please file an issue in JIRA for this?
>
> Ismael
>
> On Tue, Jan 12, 2016 at 2:40 PM, Meghana Narasimhan <
> mnarasimhan@bandwidth.com> wrote:
>
> > Hi,
> > Came across a similar issue. We are running a 3 node cluster (kafka
> version
> > 0.9) and Node 0 also has a few mirror makers running.
> > When we do a rolling restart of the cluster, the mirror maker shuts down
> > with the following errors.
> >
> > [2016-01-11 20:16:00,348] WARN Got error produce response with
> correlation
> > id 12491674 on topic-partition test-99, retrying (2147483646 attempts
> > left). Error: NOT_LEADER_FOR_PARTITION
> > (org.apache.kafka.clients.producer.internals.Sender)
> > [2016-01-11 20:16:00,853] FATAL [mirrormaker-thread-0] Mirror maker
> thread
> > failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
> > java.lang.IllegalStateException: Memory records is not writable
> > at
> >
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
> > at
> >
> >
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
> > at
> >
> >
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
> > at
> >
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:435)
> > at
> > kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
> > at
> >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > at
> >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > at
> > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > at
> > kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:398)
> > [2016-01-11 20:16:01,072] WARN Got error produce response with
> correlation
> > id 12491679 on topic-partition test-75, retrying (2147483646 attempts
> > left). Error: NETWORK_EXCEPTION
> > (org.apache.kafka.clients.producer.internals.Sender)
> > [2016-01-11 20:16:01,073] WARN Got error produce response with
> correlation
> > id 12491679 on topic-partition test-93, retrying (2147483646 attempts
> > left). Error: NETWORK_EXCEPTION
> > (org.apache.kafka.clients.producer.internals.Sender)
> > [2016-01-11 20:16:01,073] WARN Got error produce response with
> correlation
> > id 12491679 on topic-partition test-24, retrying (2147483646 attempts
> > left). Error: NETWORK_EXCEPTION
> > (org.apache.kafka.clients.producer.internals.Sender)
> >
> > [2016-01-11 20:16:20,479] FATAL [mirrormaker-thread-0] Mirror maker
> thread
> > exited abnormally, stopping the whole mirror maker.
> > (kafka.tools.MirrorMaker$MirrorMakerThread)
> >
> >
> > Curious if the NOT_LEADER_FOR_PARTITION is because of a potential bug
> > hinted at in the thread ,
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201505.mbox/%3CCAJS3ho_u8s1Xou_kudNfjAMyPJtMrjLW10QVkNGn2YQkdan0+A@mail.gmail.com%3E
> >
> >
> > And I think the mirror maker shuts down because of the
> > "abort.on.send.failure" which is set to true in our case.
> >
> > Thanks,
> > Meghana
> >
> > On Fri, Jan 8, 2016 at 8:14 PM, Samuel Zhou <zh...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I found a problem that says Memory records is not writable in
> MirrorMaker
> > > when producer sending out messages:
> > >
> > > [2016-01-08 18:52:30,729] FATAL [mirrormaker-thread-32] Mirror maker
> > thread
> > > failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
> > > java.lang.IllegalStateException: Memory records is not writable
> > > at
> > >
> >
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:439)
> > > at
> > > kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
> > > at
> > >
> > >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > > at
> > >
> > >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > > at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > > at
> > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > >
> > > The above message repeated several times and at last the MirrorMaker is
> > > shutdown:
> > >
> > > [2016-01-08 18:52:31,256] FATAL [mirrormaker-thread-59] Mirror maker
> > thread
> > > exited abnormally, stopping the whole mirror maker.
> > > (kafka.tools.MirrorMaker$MirrorMakerThread)
> > >
> > > MirrorMaker was shutdown around 30 minutes right after every time I
> > started
> > > it, it is using plain text protocol(No SSL), and the mirrormaker has
> > > options of "--abort.on.send.failure false" and "--num.streams 100" .
> And
> > it
> > > is running with kafka 0.9 release. Does anybody know how to solve this
> > > problem?
> > >
> > > Thanks!
> > >
> > > Samuel
> > >
> >
>
Re: Memory records is not writable in MirrorMaker
Posted by Ismael Juma <is...@juma.me.uk>.
Can you please file an issue in JIRA for this?
Ismael
On Tue, Jan 12, 2016 at 2:40 PM, Meghana Narasimhan <
mnarasimhan@bandwidth.com> wrote:
> Hi,
> Came across a similar issue. We are running a 3 node cluster (kafka version
> 0.9) and Node 0 also has a few mirror makers running.
> When we do a rolling restart of the cluster, the mirror maker shuts down
> with the following errors.
>
> [2016-01-11 20:16:00,348] WARN Got error produce response with correlation
> id 12491674 on topic-partition test-99, retrying (2147483646 attempts
> left). Error: NOT_LEADER_FOR_PARTITION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:00,853] FATAL [mirrormaker-thread-0] Mirror maker thread
> failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
> java.lang.IllegalStateException: Memory records is not writable
> at
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
> at
>
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
> at
>
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
> at
>
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:435)
> at
> kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
> at
>
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> at
>
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:398)
> [2016-01-11 20:16:01,072] WARN Got error produce response with correlation
> id 12491679 on topic-partition test-75, retrying (2147483646 attempts
> left). Error: NETWORK_EXCEPTION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:01,073] WARN Got error produce response with correlation
> id 12491679 on topic-partition test-93, retrying (2147483646 attempts
> left). Error: NETWORK_EXCEPTION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:01,073] WARN Got error produce response with correlation
> id 12491679 on topic-partition test-24, retrying (2147483646 attempts
> left). Error: NETWORK_EXCEPTION
> (org.apache.kafka.clients.producer.internals.Sender)
>
> [2016-01-11 20:16:20,479] FATAL [mirrormaker-thread-0] Mirror maker thread
> exited abnormally, stopping the whole mirror maker.
> (kafka.tools.MirrorMaker$MirrorMakerThread)
>
>
> Curious if the NOT_LEADER_FOR_PARTITION is because of a potential bug
> hinted at in the thread ,
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201505.mbox/%3CCAJS3ho_u8s1Xou_kudNfjAMyPJtMrjLW10QVkNGn2YQkdan0+A@mail.gmail.com%3E
>
>
> And I think the mirror maker shuts down because of the
> "abort.on.send.failure" which is set to true in our case.
>
> Thanks,
> Meghana
>
> On Fri, Jan 8, 2016 at 8:14 PM, Samuel Zhou <zh...@gmail.com> wrote:
>
> > Hi,
> >
> > I found a problem that says Memory records is not writable in MirrorMaker
> > when producer sending out messages:
> >
> > [2016-01-08 18:52:30,729] FATAL [mirrormaker-thread-32] Mirror maker
> thread
> > failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
> > java.lang.IllegalStateException: Memory records is not writable
> > at
> >
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
> > at
> >
> >
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
> > at
> >
> >
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
> > at
> >
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:439)
> > at
> > kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
> > at
> >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > at
> >
> >
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > at
> > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> >
> > The above message repeated several times and at last the MirrorMaker is
> > shutdown:
> >
> > [2016-01-08 18:52:31,256] FATAL [mirrormaker-thread-59] Mirror maker
> thread
> > exited abnormally, stopping the whole mirror maker.
> > (kafka.tools.MirrorMaker$MirrorMakerThread)
> >
> > MirrorMaker was shutdown around 30 minutes right after every time I
> started
> > it, it is using plain text protocol(No SSL), and the mirrormaker has
> > options of "--abort.on.send.failure false" and "--num.streams 100" . And
> it
> > is running with kafka 0.9 release. Does anybody know how to solve this
> > problem?
> >
> > Thanks!
> >
> > Samuel
> >
>
Re: Memory records is not writable in MirrorMaker
Posted by Meghana Narasimhan <mn...@bandwidth.com>.
Hi,
Came across a similar issue. We are running a 3 node cluster (kafka version
0.9) and Node 0 also has a few mirror makers running.
When we do a rolling restart of the cluster, the mirror maker shuts down
with the following errors.
[2016-01-11 20:16:00,348] WARN Got error produce response with correlation
id 12491674 on topic-partition test-99, retrying (2147483646 attempts
left). Error: NOT_LEADER_FOR_PARTITION
(org.apache.kafka.clients.producer.internals.Sender)
[2016-01-11 20:16:00,853] FATAL [mirrormaker-thread-0] Mirror maker thread
failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
java.lang.IllegalStateException: Memory records is not writable
at
org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
at
org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:435)
at
kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:398)
[2016-01-11 20:16:01,072] WARN Got error produce response with correlation
id 12491679 on topic-partition test-75, retrying (2147483646 attempts
left). Error: NETWORK_EXCEPTION
(org.apache.kafka.clients.producer.internals.Sender)
[2016-01-11 20:16:01,073] WARN Got error produce response with correlation
id 12491679 on topic-partition test-93, retrying (2147483646 attempts
left). Error: NETWORK_EXCEPTION
(org.apache.kafka.clients.producer.internals.Sender)
[2016-01-11 20:16:01,073] WARN Got error produce response with correlation
id 12491679 on topic-partition test-24, retrying (2147483646 attempts
left). Error: NETWORK_EXCEPTION
(org.apache.kafka.clients.producer.internals.Sender)
[2016-01-11 20:16:20,479] FATAL [mirrormaker-thread-0] Mirror maker thread
exited abnormally, stopping the whole mirror maker.
(kafka.tools.MirrorMaker$MirrorMakerThread)
Curious if the NOT_LEADER_FOR_PARTITION is because of a potential bug
hinted at in the thread ,
http://mail-archives.apache.org/mod_mbox/kafka-users/201505.mbox/%3CCAJS3ho_u8s1Xou_kudNfjAMyPJtMrjLW10QVkNGn2YQkdan0+A@mail.gmail.com%3E
And I think the mirror maker shuts down because of the
"abort.on.send.failure" which is set to true in our case.
Thanks,
Meghana
On Fri, Jan 8, 2016 at 8:14 PM, Samuel Zhou <zh...@gmail.com> wrote:
> Hi,
>
> I found a problem that says Memory records is not writable in MirrorMaker
> when producer sending out messages:
>
> [2016-01-08 18:52:30,729] FATAL [mirrormaker-thread-32] Mirror maker thread
> failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
> java.lang.IllegalStateException: Memory records is not writable
> at
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
> at
>
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
> at
>
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
> at
>
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:439)
> at
> kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
> at
>
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> at
>
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>
> The above message repeated several times and at last the MirrorMaker is
> shutdown:
>
> [2016-01-08 18:52:31,256] FATAL [mirrormaker-thread-59] Mirror maker thread
> exited abnormally, stopping the whole mirror maker.
> (kafka.tools.MirrorMaker$MirrorMakerThread)
>
> MirrorMaker was shutdown around 30 minutes right after every time I started
> it, it is using plain text protocol(No SSL), and the mirrormaker has
> options of "--abort.on.send.failure false" and "--num.streams 100" . And it
> is running with kafka 0.9 release. Does anybody know how to solve this
> problem?
>
> Thanks!
>
> Samuel
>