You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sunil Parmar <su...@gmail.com> on 2018/01/12 00:40:48 UTC

mirror maker producer thread dies with error

We see multiple instance of this error

2017-12-23 05:30:53,722 WARN
org.apache.kafka.clients.producer.internals.Sender: Got error produce
response with correlation id 41987642 on topic-partition events-14,
retrying (2147482899 attempts left). Error: NOT_LEADER_FOR_PARTITION

Followed by this error (repeating multilpe times)

2017-12-23 05:31:09,165 INFO
org.apache.kafka.clients.producer.KafkaProducer: Proceeding to force close
the producer since pending requests could not be completed within timeout 0
ms.
2017-12-23 05:31:09,165 ERROR
org.apache.kafka.clients.producer.internals.ErrorLoggingCallback: Error
when sending message to topic events with key: 36 bytes, value: 8794 bytes
with error: Batch Expired

Once in a while I also notice following error happens only once or two
times between restarts of mm.
2017-09-10 20:02:47,723 FATAL kafka.tools.MirrorMaker$MirrorMakerThread:
[mirrormaker-thread-14] Mirror maker thread failure due to
java.lang.IllegalStateException: Memory records is not writable
    at
org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
    at
org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
    at
org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:166)
    at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:438)
    at
kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:628)
    at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$6.apply(MirrorMaker.scala:423)
    at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$6.apply(MirrorMaker.scala:423)
    at scala.collection.Iterator$class.foreach(Iterator.scala:742)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:423)
2017-09-10 20:02:47,728 INFO kafka.tools.MirrorMaker$MirrorMakerThread:
[mirrormaker-thread-14] Flushing producer.
2017-09-10 20:02:47,780 WARN
org.apache.kafka.clients.producer.internals.Sender: Got error produce
response with correlation id 20892 on topic-partition
thirdpartycalls_ingestion_am1-2, retrying (2147483645 attempts left).
Error: NOT_LEADER_FOR_PARTITION


followed by

Clean shutdown of mirror maker

We're using CDH, Kafka 0.9

Can someone explains the reason/meaning of the errors we see ? This will
help us debug this issue further. My understanding is that the producer
threads are failing because for some reason partition leader is changing at
run time. We also noticed that this happens on the perticular day of the
week but I coudln't locate any weekly scheduler in any configuration.

Any pointers will be helpful

Thanks,
Sunil Parmar

Re: mirror maker producer thread dies with error

Posted by Manikumar <ma...@gmail.com>.
"Memory records is not writable" error was fixed in 0.10.0.0 release
https://issues.apache.org/jira/browse/KAFKA-3594

On Fri, Jan 12, 2018 at 6:10 AM, Sunil Parmar <su...@gmail.com> wrote:

> We see multiple instance of this error
>
> 2017-12-23 05:30:53,722 WARN
> org.apache.kafka.clients.producer.internals.Sender: Got error produce
> response with correlation id 41987642 on topic-partition events-14,
> retrying (2147482899 attempts left). Error: NOT_LEADER_FOR_PARTITION
>
> Followed by this error (repeating multilpe times)
>
> 2017-12-23 05:31:09,165 INFO
> org.apache.kafka.clients.producer.KafkaProducer: Proceeding to force close
> the producer since pending requests could not be completed within timeout 0
> ms.
> 2017-12-23 05:31:09,165 ERROR
> org.apache.kafka.clients.producer.internals.ErrorLoggingCallback: Error
> when sending message to topic events with key: 36 bytes, value: 8794 bytes
> with error: Batch Expired
>
> Once in a while I also notice following error happens only once or two
> times between restarts of mm.
> 2017-09-10 20:02:47,723 FATAL kafka.tools.MirrorMaker$MirrorMakerThread:
> [mirrormaker-thread-14] Mirror maker thread failure due to
> java.lang.IllegalStateException: Memory records is not writable
>     at
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
>     at
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(
> RecordBatch.java:69)
>     at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(
> RecordAccumulator.java:166)
>     at
> org.apache.kafka.clients.producer.KafkaProducer.send(
> KafkaProducer.java:438)
>     at
> kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:628)
>     at
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$
> run$6.apply(MirrorMaker.scala:423)
>     at
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$
> run$6.apply(MirrorMaker.scala:423)
>     at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>     at kafka.tools.MirrorMaker$MirrorMakerThread.run(
> MirrorMaker.scala:423)
> 2017-09-10 20:02:47,728 INFO kafka.tools.MirrorMaker$MirrorMakerThread:
> [mirrormaker-thread-14] Flushing producer.
> 2017-09-10 20:02:47,780 WARN
> org.apache.kafka.clients.producer.internals.Sender: Got error produce
> response with correlation id 20892 on topic-partition
> thirdpartycalls_ingestion_am1-2, retrying (2147483645 attempts left).
> Error: NOT_LEADER_FOR_PARTITION
>
>
> followed by
>
> Clean shutdown of mirror maker
>
> We're using CDH, Kafka 0.9
>
> Can someone explains the reason/meaning of the errors we see ? This will
> help us debug this issue further. My understanding is that the producer
> threads are failing because for some reason partition leader is changing at
> run time. We also noticed that this happens on the perticular day of the
> week but I coudln't locate any weekly scheduler in any configuration.
>
> Any pointers will be helpful
>
> Thanks,
> Sunil Parmar
>