You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Karl Kirch <kk...@wdtinc.com> on 2013/04/24 00:35:13 UTC
Occasional batch send errors
I occasionally am getting some batch send errors from the stock async producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
Is there anyway to check what happens when those batch errors occur?
Or bump up the retry count? (looks like it only did a single retry).
I need the speed of the async producer, but it needs to be reliable (I see a handful of these a day but in a weather alerting system it only takes missing one let alone 25 or 100/1000).
Here's a stack trace of one of the errors that I'm seeing.
22:23:39.405 [ProducerSendThread-1824508747] WARN k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24]
at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24]
at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24]
at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na]
at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na]
at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na]
at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) [apns-consumer-1.0.jar:na]
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) [apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) [apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) [apns-consumer-1.0.jar:na]
at scala.collection.immutable.Stream.foreach(Stream.scala:260) [apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) [apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) [apns-consumer-1.0.jar:na]
22:23:39.406 [ProducerSendThread-1824508747] ERROR k.p.a.ProducerSendThread - Error in handling batch of 27 events
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24]
at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24]
at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24]
at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na]
at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na]
at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na]
at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) ~[apns-consumer-1.0.jar:na]
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) ~[apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) [apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) [apns-consumer-1.0.jar:na]
at scala.collection.immutable.Stream.foreach(Stream.scala:260) [apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) [apns-consumer-1.0.jar:na]
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) [apns-consumer-1.0.jar:na]
Re: Occasional batch send errors
Posted by Neha Narkhede <ne...@gmail.com>.
It is highly recommended that Kafka and Zookeeper be deployed on different
boxes. Also make sure they get dedicated disks, separate from log4j and the
OS.
Thanks,
Neha
On Wednesday, April 24, 2013, Karl Kirch wrote:
> So switched to sync producer to see what would happen.
> I still get the connection reset by peer error randomly (I say randomly,
> but seems to be connected to some zookeeper CancelledKeyExceptions), but
> unfortunately it throws an error on the message after the one that didn't
> get sent.
>
> Is that the way its supposed to work?
>
> Karl
>
> On Apr 23, 2013, at 7:18 PM, Xavier Stevens <xavier@gaikai.com<javascript:;>
> >
> wrote:
>
> > Usually when these types of errors are because you're not connecting to
> the
> > proper host:port. Double check your configs, make sure everything is
> > running and listening on the host:port you think they are.
> >
> > Have you tried using the sync producer to work out your bugs? My guess is
> > the sync producer would fail on the first message rather than failing
> when
> > the batch is submitted.
> >
> >
> > On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch <kk...@wdtinc.com> wrote:
> >
> >> Hmmm… that didn't seem to help.
> >> Anyone else see this sort of errors?
> >>
> >> Karl
> >>
> >>
> >> On Apr 23, 2013, at 5:58 PM, Karl Kirch <kk...@wdtinc.com>
> >> wrote:
> >>
> >>> I'm going to try bumping up the "numRetries" key in my producer config.
> >>> Is this a good option in this case?
> >>> I am using the zookeeper connect option so I'm aware that I may get
> >> stuck retrying to a failed node, but if it's just a temporary network
> >> glitch I'll at least get a bit more of a chance to recover.
> >>>
> >>> Thanks,
> >>> Karl
> >>>
> >>> On Apr 23, 2013, at 5:35 PM, Karl Kirch <kk...@wdtinc.com>
> >>> wrote:
> >>>
> >>>> I occasionally am getting some batch send errors from the stock async
> >> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
> >>>> Is there anyway to check what happens when those batch errors occur?
> >>>> Or bump up the retry count? (looks like it only did a single retry).
> >>>>
> >>>> I need the speed of the async producer, but it needs to be reliable (I
> >> see a handful of these a day but in a weather alerting system it only
> takes
> >> missing one let alone 25 or 100/1000).
> >>>>
> >>>> Here's a stack trace of one of the errors that I'm seeing.
> >>>>
> >>>> 22:23:39.405 [ProducerSendThread-1824508747] WARN
> >> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
> >>>> java.io.IOException: Connection reset by peer
> >>>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> >>>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
> >> ~[na:1.6.0_24]
> >>>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> >>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
> >> ~[na:1.6.0_24]
> >>>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> >> ~[na:1.6.0_24]
> >>>> at
> >>
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
> >> ~[apns-consumer-1.0.jar:na]
> >>>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
> >> ~[apns-consumer-1.0.jar:na]
> >>>> at
> >>
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
> >> ~[apns-consumer-1.0.jar:na]
> >>>> at
> >> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
> >> ~[apns-consumer-1.0.jar:na]
> >>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
> >> ~[apns-consumer-1.0.jar:na]
> >>>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
> >> ~[apns-consumer-1.0.jar:na]
> >>>> at
> >>
> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
> >> [apns-consumer-1.0.jar:na]
> >>>> at
> >>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
> >> [apns-consumer-1.0.jar:na]
> >>>> at
> >>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
> >> [apns-consumer-1.0.jar:na]
> >>>> at
> >>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Prod
Re: Occasional batch send errors
Posted by Karl Kirch <kk...@wdtinc.com>.
So switched to sync producer to see what would happen.
I still get the connection reset by peer error randomly (I say randomly, but seems to be connected to some zookeeper CancelledKeyExceptions), but unfortunately it throws an error on the message after the one that didn't get sent.
Is that the way its supposed to work?
Karl
On Apr 23, 2013, at 7:18 PM, Xavier Stevens <xa...@gaikai.com>
wrote:
> Usually when these types of errors are because you're not connecting to the
> proper host:port. Double check your configs, make sure everything is
> running and listening on the host:port you think they are.
>
> Have you tried using the sync producer to work out your bugs? My guess is
> the sync producer would fail on the first message rather than failing when
> the batch is submitted.
>
>
> On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch <kk...@wdtinc.com> wrote:
>
>> Hmmm… that didn't seem to help.
>> Anyone else see this sort of errors?
>>
>> Karl
>>
>>
>> On Apr 23, 2013, at 5:58 PM, Karl Kirch <kk...@wdtinc.com>
>> wrote:
>>
>>> I'm going to try bumping up the "numRetries" key in my producer config.
>>> Is this a good option in this case?
>>> I am using the zookeeper connect option so I'm aware that I may get
>> stuck retrying to a failed node, but if it's just a temporary network
>> glitch I'll at least get a bit more of a chance to recover.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Apr 23, 2013, at 5:35 PM, Karl Kirch <kk...@wdtinc.com>
>>> wrote:
>>>
>>>> I occasionally am getting some batch send errors from the stock async
>> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
>>>> Is there anyway to check what happens when those batch errors occur?
>>>> Or bump up the retry count? (looks like it only did a single retry).
>>>>
>>>> I need the speed of the async producer, but it needs to be reliable (I
>> see a handful of these a day but in a weather alerting system it only takes
>> missing one let alone 25 or 100/1000).
>>>>
>>>> Here's a stack trace of one of the errors that I'm seeing.
>>>>
>>>> 22:23:39.405 [ProducerSendThread-1824508747] WARN
>> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>> ~[na:1.6.0_24]
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>> ~[na:1.6.0_24]
>>>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>> ~[na:1.6.0_24]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>> [apns-consumer-1.0.jar:na]
>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>> [apns-consumer-1.0.jar:na]
>>>> 22:23:39.406 [ProducerSendThread-1824508747] ERROR
>> k.p.a.ProducerSendThread - Error in handling batch of 27 events
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>> ~[na:1.6.0_24]
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>> ~[na:1.6.0_24]
>>>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>> ~[na:1.6.0_24]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>> [apns-consumer-1.0.jar:na]
>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>> [apns-consumer-1.0.jar:na]
>>>>
>>>>
>>>
>>
>>
Re: Occasional batch send errors
Posted by Karl Kirch <kk...@wdtinc.com>.
I really need the speed of the async producer (unless the sync producer is able to get up in the 100k/sec range...) so the sync producer is going to be a tough sell.
I've also double checked my config settings and they're good.
I did notice some slow fsync warnings in the Kafka broker logs though. Our operations guys have us running on some virtual machines and it's quite possible that we're hitting some disk issues.
Karl
On Apr 23, 2013, at 7:18 PM, "Xavier Stevens" <xa...@gaikai.com> wrote:
> Usually when these types of errors are because you're not connecting to the
> proper host:port. Double check your configs, make sure everything is
> running and listening on the host:port you think they are.
>
> Have you tried using the sync producer to work out your bugs? My guess is
> the sync producer would fail on the first message rather than failing when
> the batch is submitted.
>
>
> On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch <kk...@wdtinc.com> wrote:
>
>> Hmmm… that didn't seem to help.
>> Anyone else see this sort of errors?
>>
>> Karl
>>
>>
>> On Apr 23, 2013, at 5:58 PM, Karl Kirch <kk...@wdtinc.com>
>> wrote:
>>
>>> I'm going to try bumping up the "numRetries" key in my producer config.
>>> Is this a good option in this case?
>>> I am using the zookeeper connect option so I'm aware that I may get
>> stuck retrying to a failed node, but if it's just a temporary network
>> glitch I'll at least get a bit more of a chance to recover.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Apr 23, 2013, at 5:35 PM, Karl Kirch <kk...@wdtinc.com>
>>> wrote:
>>>
>>>> I occasionally am getting some batch send errors from the stock async
>> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
>>>> Is there anyway to check what happens when those batch errors occur?
>>>> Or bump up the retry count? (looks like it only did a single retry).
>>>>
>>>> I need the speed of the async producer, but it needs to be reliable (I
>> see a handful of these a day but in a weather alerting system it only takes
>> missing one let alone 25 or 100/1000).
>>>>
>>>> Here's a stack trace of one of the errors that I'm seeing.
>>>>
>>>> 22:23:39.405 [ProducerSendThread-1824508747] WARN
>> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>> ~[na:1.6.0_24]
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>> ~[na:1.6.0_24]
>>>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>> ~[na:1.6.0_24]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>> [apns-consumer-1.0.jar:na]
>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>> [apns-consumer-1.0.jar:na]
>>>> 22:23:39.406 [ProducerSendThread-1824508747] ERROR
>> k.p.a.ProducerSendThread - Error in handling batch of 27 events
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>> ~[na:1.6.0_24]
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>> ~[na:1.6.0_24]
>>>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>> ~[na:1.6.0_24]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>> [apns-consumer-1.0.jar:na]
>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>> [apns-consumer-1.0.jar:na]
>>
>>
Re: Occasional batch send errors
Posted by Xavier Stevens <xa...@gaikai.com>.
Usually when these types of errors are because you're not connecting to the
proper host:port. Double check your configs, make sure everything is
running and listening on the host:port you think they are.
Have you tried using the sync producer to work out your bugs? My guess is
the sync producer would fail on the first message rather than failing when
the batch is submitted.
On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch <kk...@wdtinc.com> wrote:
> Hmmm… that didn't seem to help.
> Anyone else see this sort of errors?
>
> Karl
>
>
> On Apr 23, 2013, at 5:58 PM, Karl Kirch <kk...@wdtinc.com>
> wrote:
>
> > I'm going to try bumping up the "numRetries" key in my producer config.
> > Is this a good option in this case?
> > I am using the zookeeper connect option so I'm aware that I may get
> stuck retrying to a failed node, but if it's just a temporary network
> glitch I'll at least get a bit more of a chance to recover.
> >
> > Thanks,
> > Karl
> >
> > On Apr 23, 2013, at 5:35 PM, Karl Kirch <kk...@wdtinc.com>
> > wrote:
> >
> >> I occasionally am getting some batch send errors from the stock async
> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
> >> Is there anyway to check what happens when those batch errors occur?
> >> Or bump up the retry count? (looks like it only did a single retry).
> >>
> >> I need the speed of the async producer, but it needs to be reliable (I
> see a handful of these a day but in a weather alerting system it only takes
> missing one let alone 25 or 100/1000).
> >>
> >> Here's a stack trace of one of the errors that I'm seeing.
> >>
> >> 22:23:39.405 [ProducerSendThread-1824508747] WARN
> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
> >> java.io.IOException: Connection reset by peer
> >> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> >> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
> ~[na:1.6.0_24]
> >> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
> ~[na:1.6.0_24]
> >> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> ~[na:1.6.0_24]
> >> at
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
> [apns-consumer-1.0.jar:na]
> >> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
> [apns-consumer-1.0.jar:na]
> >> 22:23:39.406 [ProducerSendThread-1824508747] ERROR
> k.p.a.ProducerSendThread - Error in handling batch of 27 events
> >> java.io.IOException: Connection reset by peer
> >> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> >> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
> ~[na:1.6.0_24]
> >> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
> ~[na:1.6.0_24]
> >> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> ~[na:1.6.0_24]
> >> at
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
> [apns-consumer-1.0.jar:na]
> >> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
> [apns-consumer-1.0.jar:na]
> >>
> >>
> >
>
>
Re: Occasional batch send errors
Posted by Karl Kirch <kk...@wdtinc.com>.
Thanks Andrew,
I'm not seeing the event queue exception but, I'm running my cluster on a set of virtual machines which share the same physical hardware (I know, exactly what I'm not supposed to do) and I'm getting some slow fsync zookeeper warnings in my logs. I imagine that my broker writes are slow as well. It could be that I'm getting write contention because of the shared metal/disk.
I know kafka and zookeeper are susceptible to latency spikes...
Karl
On Apr 23, 2013, at 7:16 PM, Andrew Neilson <ar...@gmail.com>
wrote:
> Hey Karl, I have a very similar setup (3 kafka 0.7.2 brokers, 3 ZK 3.4.3
> nodes) that I'm running right now and am getting the same error on the
> producers. Haven't resolved it yet:
>
> ERROR ProducerSendThread--1585663279
> kafka.producer.async.ProducerSendThread - Error in handling batch of 200
> events
> java.io.IOException: Connection reset by peer
> ...
>
> For me these errors appear to be coupled with other errors like this:
>
> kafka.producer.async.AsyncProducer - Event queue is full of unsent
> messages, could not send event:
>
> As I understand it, this happens when you are producing faster than the
> brokers can persist the messages. It's possible these are two different
> issues...
>
> Anyway I've been doing a lot of work on this this afternoon so I may have
> more information later. Someone else probably knows more though.
>
>
>
> On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch <kk...@wdtinc.com> wrote:
>
>> Hmmm… that didn't seem to help.
>> Anyone else see this sort of errors?
>>
>> Karl
>>
>>
>> On Apr 23, 2013, at 5:58 PM, Karl Kirch <kk...@wdtinc.com>
>> wrote:
>>
>>> I'm going to try bumping up the "numRetries" key in my producer config.
>>> Is this a good option in this case?
>>> I am using the zookeeper connect option so I'm aware that I may get
>> stuck retrying to a failed node, but if it's just a temporary network
>> glitch I'll at least get a bit more of a chance to recover.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Apr 23, 2013, at 5:35 PM, Karl Kirch <kk...@wdtinc.com>
>>> wrote:
>>>
>>>> I occasionally am getting some batch send errors from the stock async
>> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
>>>> Is there anyway to check what happens when those batch errors occur?
>>>> Or bump up the retry count? (looks like it only did a single retry).
>>>>
>>>> I need the speed of the async producer, but it needs to be reliable (I
>> see a handful of these a day but in a weather alerting system it only takes
>> missing one let alone 25 or 100/1000).
>>>>
>>>> Here's a stack trace of one of the errors that I'm seeing.
>>>>
>>>> 22:23:39.405 [ProducerSendThread-1824508747] WARN
>> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>> ~[na:1.6.0_24]
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>> ~[na:1.6.0_24]
>>>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>> ~[na:1.6.0_24]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>> [apns-consumer-1.0.jar:na]
>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>> [apns-consumer-1.0.jar:na]
>>>> 22:23:39.406 [ProducerSendThread-1824508747] ERROR
>> k.p.a.ProducerSendThread - Error in handling batch of 27 events
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>> ~[na:1.6.0_24]
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>> ~[na:1.6.0_24]
>>>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>> ~[na:1.6.0_24]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>> ~[apns-consumer-1.0.jar:na]
>>>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>> ~[apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>> [apns-consumer-1.0.jar:na]
>>>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>> [apns-consumer-1.0.jar:na]
>>>> at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>> [apns-consumer-1.0.jar:na]
>>>>
>>>>
>>>
>>
>>
Re: Occasional batch send errors
Posted by Andrew Neilson <ar...@gmail.com>.
Hey Karl, I have a very similar setup (3 kafka 0.7.2 brokers, 3 ZK 3.4.3
nodes) that I'm running right now and am getting the same error on the
producers. Haven't resolved it yet:
ERROR ProducerSendThread--1585663279
kafka.producer.async.ProducerSendThread - Error in handling batch of 200
events
java.io.IOException: Connection reset by peer
...
For me these errors appear to be coupled with other errors like this:
kafka.producer.async.AsyncProducer - Event queue is full of unsent
messages, could not send event:
As I understand it, this happens when you are producing faster than the
brokers can persist the messages. It's possible these are two different
issues...
Anyway I've been doing a lot of work on this this afternoon so I may have
more information later. Someone else probably knows more though.
On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch <kk...@wdtinc.com> wrote:
> Hmmm… that didn't seem to help.
> Anyone else see this sort of errors?
>
> Karl
>
>
> On Apr 23, 2013, at 5:58 PM, Karl Kirch <kk...@wdtinc.com>
> wrote:
>
> > I'm going to try bumping up the "numRetries" key in my producer config.
> > Is this a good option in this case?
> > I am using the zookeeper connect option so I'm aware that I may get
> stuck retrying to a failed node, but if it's just a temporary network
> glitch I'll at least get a bit more of a chance to recover.
> >
> > Thanks,
> > Karl
> >
> > On Apr 23, 2013, at 5:35 PM, Karl Kirch <kk...@wdtinc.com>
> > wrote:
> >
> >> I occasionally am getting some batch send errors from the stock async
> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
> >> Is there anyway to check what happens when those batch errors occur?
> >> Or bump up the retry count? (looks like it only did a single retry).
> >>
> >> I need the speed of the async producer, but it needs to be reliable (I
> see a handful of these a day but in a weather alerting system it only takes
> missing one let alone 25 or 100/1000).
> >>
> >> Here's a stack trace of one of the errors that I'm seeing.
> >>
> >> 22:23:39.405 [ProducerSendThread-1824508747] WARN
> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
> >> java.io.IOException: Connection reset by peer
> >> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> >> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
> ~[na:1.6.0_24]
> >> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
> ~[na:1.6.0_24]
> >> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> ~[na:1.6.0_24]
> >> at
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
> [apns-consumer-1.0.jar:na]
> >> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
> [apns-consumer-1.0.jar:na]
> >> 22:23:39.406 [ProducerSendThread-1824508747] ERROR
> k.p.a.ProducerSendThread - Error in handling batch of 27 events
> >> java.io.IOException: Connection reset by peer
> >> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> >> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
> ~[na:1.6.0_24]
> >> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
> ~[na:1.6.0_24]
> >> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> ~[na:1.6.0_24]
> >> at
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
> ~[apns-consumer-1.0.jar:na]
> >> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
> ~[apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
> [apns-consumer-1.0.jar:na]
> >> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
> [apns-consumer-1.0.jar:na]
> >> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
> [apns-consumer-1.0.jar:na]
> >>
> >>
> >
>
>
Re: Occasional batch send errors
Posted by Karl Kirch <kk...@wdtinc.com>.
Hmmm… that didn't seem to help.
Anyone else see this sort of errors?
Karl
On Apr 23, 2013, at 5:58 PM, Karl Kirch <kk...@wdtinc.com>
wrote:
> I'm going to try bumping up the "numRetries" key in my producer config.
> Is this a good option in this case?
> I am using the zookeeper connect option so I'm aware that I may get stuck retrying to a failed node, but if it's just a temporary network glitch I'll at least get a bit more of a chance to recover.
>
> Thanks,
> Karl
>
> On Apr 23, 2013, at 5:35 PM, Karl Kirch <kk...@wdtinc.com>
> wrote:
>
>> I occasionally am getting some batch send errors from the stock async producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
>> Is there anyway to check what happens when those batch errors occur?
>> Or bump up the retry count? (looks like it only did a single retry).
>>
>> I need the speed of the async producer, but it needs to be reliable (I see a handful of these a day but in a weather alerting system it only takes missing one let alone 25 or 100/1000).
>>
>> Here's a stack trace of one of the errors that I'm seeing.
>>
>> 22:23:39.405 [ProducerSendThread-1824508747] WARN k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24]
>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24]
>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24]
>> at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na]
>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na]
>> at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) [apns-consumer-1.0.jar:na]
>> at scala.collection.immutable.Stream.foreach(Stream.scala:260) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) [apns-consumer-1.0.jar:na]
>> 22:23:39.406 [ProducerSendThread-1824508747] ERROR k.p.a.ProducerSendThread - Error in handling batch of 27 events
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24]
>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24]
>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24]
>> at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na]
>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na]
>> at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) [apns-consumer-1.0.jar:na]
>> at scala.collection.immutable.Stream.foreach(Stream.scala:260) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) [apns-consumer-1.0.jar:na]
>> at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) [apns-consumer-1.0.jar:na]
>>
>>
>
Re: Occasional batch send errors
Posted by Karl Kirch <kk...@wdtinc.com>.
I'm going to try bumping up the "numRetries" key in my producer config.
Is this a good option in this case?
I am using the zookeeper connect option so I'm aware that I may get stuck retrying to a failed node, but if it's just a temporary network glitch I'll at least get a bit more of a chance to recover.
Thanks,
Karl
On Apr 23, 2013, at 5:35 PM, Karl Kirch <kk...@wdtinc.com>
wrote:
> I occasionally am getting some batch send errors from the stock async producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
> Is there anyway to check what happens when those batch errors occur?
> Or bump up the retry count? (looks like it only did a single retry).
>
> I need the speed of the async producer, but it needs to be reliable (I see a handful of these a day but in a weather alerting system it only takes missing one let alone 25 or 100/1000).
>
> Here's a stack trace of one of the errors that I'm seeing.
>
> 22:23:39.405 [ProducerSendThread-1824508747] WARN k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24]
> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24]
> at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24]
> at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na]
> at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na]
> at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) [apns-consumer-1.0.jar:na]
> at scala.collection.immutable.Stream.foreach(Stream.scala:260) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) [apns-consumer-1.0.jar:na]
> 22:23:39.406 [ProducerSendThread-1824508747] ERROR k.p.a.ProducerSendThread - Error in handling batch of 27 events
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33) ~[na:1.6.0_24]
> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367) ~[na:1.6.0_24]
> at java.nio.channels.SocketChannel.write(SocketChannel.java:360) ~[na:1.6.0_24]
> at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49) ~[apns-consumer-1.0.jar:na]
> at kafka.network.Send$class.writeCompletely(Transmission.scala:73) ~[apns-consumer-1.0.jar:na]
> at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.send(SyncProducer.scala:94) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44) ~[apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71) [apns-consumer-1.0.jar:na]
> at scala.collection.immutable.Stream.foreach(Stream.scala:260) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70) [apns-consumer-1.0.jar:na]
> at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41) [apns-consumer-1.0.jar:na]
>
>
Re: Occasional batch send errors
Posted by Karl Kirch <kk...@wdtinc.com>.
So I'm seeing CancelledKeyExceptions cropping up about the time that the connections get reset.
Is this a zookeeper error that I'm hitting?
Karl
On Apr 24, 2013, at 9:55 AM, Karl Kirch <kk...@wdtinc.com>
wrote:
> Just got logging cranked up. Will let you know when I see it again.
>
> Thanks,
> Karl
>
> On Apr 23, 2013, at 8:11 PM, Jun Rao <ju...@gmail.com>
> wrote:
>
>> This means that the broker closed the socket connection for some reason.
>> The broker log around the same time should show the reason. Could you dig
>> that out?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Tue, Apr 23, 2013 at 3:35 PM, Karl Kirch <kk...@wdtinc.com> wrote:
>>
>>> I occasionally am getting some batch send errors from the stock async
>>> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
>>> Is there anyway to check what happens when those batch errors occur?
>>> Or bump up the retry count? (looks like it only did a single retry).
>>>
>>> I need the speed of the async producer, but it needs to be reliable (I see
>>> a handful of these a day but in a weather alerting system it only takes
>>> missing one let alone 25 or 100/1000).
>>>
>>> Here's a stack trace of one of the errors that I'm seeing.
>>>
>>> 22:23:39.405 [ProducerSendThread-1824508747] WARN
>>> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
>>> java.io.IOException: Connection reset by peer
>>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>>> ~[na:1.6.0_24]
>>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>>> ~[na:1.6.0_24]
>>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>>> ~[na:1.6.0_24]
>>> at
>>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>>> ~[apns-consumer-1.0.jar:na]
>>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>>> ~[apns-consumer-1.0.jar:na]
>>> at
>>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>>> ~[apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>>> ~[apns-consumer-1.0.jar:na]
>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>>> ~[apns-consumer-1.0.jar:na]
>>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>>> ~[apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>>> [apns-consumer-1.0.jar:na]
>>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>>> [apns-consumer-1.0.jar:na]
>>> 22:23:39.406 [ProducerSendThread-1824508747] ERROR
>>> k.p.a.ProducerSendThread - Error in handling batch of 27 events
>>> java.io.IOException: Connection reset by peer
>>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>>> ~[na:1.6.0_24]
>>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>>> ~[na:1.6.0_24]
>>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>>> ~[na:1.6.0_24]
>>> at
>>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>>> ~[apns-consumer-1.0.jar:na]
>>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>>> ~[apns-consumer-1.0.jar:na]
>>> at
>>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>>> ~[apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>>> ~[apns-consumer-1.0.jar:na]
>>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>>> ~[apns-consumer-1.0.jar:na]
>>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>>> ~[apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>>> ~[apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>>> ~[apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>>> [apns-consumer-1.0.jar:na]
>>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>>> [apns-consumer-1.0.jar:na]
>>> at
>>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>>> [apns-consumer-1.0.jar:na]
>>>
>>>
>>>
>
Re: Occasional batch send errors
Posted by Karl Kirch <kk...@wdtinc.com>.
Just got logging cranked up. Will let you know when I see it again.
Thanks,
Karl
On Apr 23, 2013, at 8:11 PM, Jun Rao <ju...@gmail.com>
wrote:
> This means that the broker closed the socket connection for some reason.
> The broker log around the same time should show the reason. Could you dig
> that out?
>
> Thanks,
>
> Jun
>
>
> On Tue, Apr 23, 2013 at 3:35 PM, Karl Kirch <kk...@wdtinc.com> wrote:
>
>> I occasionally am getting some batch send errors from the stock async
>> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
>> Is there anyway to check what happens when those batch errors occur?
>> Or bump up the retry count? (looks like it only did a single retry).
>>
>> I need the speed of the async producer, but it needs to be reliable (I see
>> a handful of these a day but in a weather alerting system it only takes
>> missing one let alone 25 or 100/1000).
>>
>> Here's a stack trace of one of the errors that I'm seeing.
>>
>> 22:23:39.405 [ProducerSendThread-1824508747] WARN
>> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>> ~[na:1.6.0_24]
>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>> ~[na:1.6.0_24]
>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>> ~[na:1.6.0_24]
>> at
>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>> ~[apns-consumer-1.0.jar:na]
>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>> ~[apns-consumer-1.0.jar:na]
>> at
>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>> ~[apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>> ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>> ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>> ~[apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>> [apns-consumer-1.0.jar:na]
>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>> [apns-consumer-1.0.jar:na]
>> 22:23:39.406 [ProducerSendThread-1824508747] ERROR
>> k.p.a.ProducerSendThread - Error in handling batch of 27 events
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
>> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
>> ~[na:1.6.0_24]
>> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
>> ~[na:1.6.0_24]
>> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
>> ~[na:1.6.0_24]
>> at
>> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
>> ~[apns-consumer-1.0.jar:na]
>> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
>> ~[apns-consumer-1.0.jar:na]
>> at
>> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
>> ~[apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
>> ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
>> ~[apns-consumer-1.0.jar:na]
>> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
>> ~[apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
>> ~[apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
>> ~[apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
>> [apns-consumer-1.0.jar:na]
>> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
>> [apns-consumer-1.0.jar:na]
>> at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
>> [apns-consumer-1.0.jar:na]
>>
>>
>>
Re: Occasional batch send errors
Posted by Jun Rao <ju...@gmail.com>.
This means that the broker closed the socket connection for some reason.
The broker log around the same time should show the reason. Could you dig
that out?
Thanks,
Jun
On Tue, Apr 23, 2013 at 3:35 PM, Karl Kirch <kk...@wdtinc.com> wrote:
> I occasionally am getting some batch send errors from the stock async
> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
> Is there anyway to check what happens when those batch errors occur?
> Or bump up the retry count? (looks like it only did a single retry).
>
> I need the speed of the async producer, but it needs to be reliable (I see
> a handful of these a day but in a weather alerting system it only takes
> missing one let alone 25 or 100/1000).
>
> Here's a stack trace of one of the errors that I'm seeing.
>
> 22:23:39.405 [ProducerSendThread-1824508747] WARN
> k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
> ~[na:1.6.0_24]
> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
> ~[na:1.6.0_24]
> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> ~[na:1.6.0_24]
> at
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
> ~[apns-consumer-1.0.jar:na]
> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
> ~[apns-consumer-1.0.jar:na]
> at
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
> ~[apns-consumer-1.0.jar:na]
> at
> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
> ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
> ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
> ~[apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
> [apns-consumer-1.0.jar:na]
> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
> [apns-consumer-1.0.jar:na]
> 22:23:39.406 [ProducerSendThread-1824508747] ERROR
> k.p.a.ProducerSendThread - Error in handling batch of 27 events
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
> ~[na:1.6.0_24]
> at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
> ~[na:1.6.0_24]
> at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> ~[na:1.6.0_24]
> at
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
> ~[apns-consumer-1.0.jar:na]
> at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
> ~[apns-consumer-1.0.jar:na]
> at
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
> ~[apns-consumer-1.0.jar:na]
> at
> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
> ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
> ~[apns-consumer-1.0.jar:na]
> at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
> ~[apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
> ~[apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
> ~[apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
> [apns-consumer-1.0.jar:na]
> at scala.collection.immutable.Stream.foreach(Stream.scala:260)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
> [apns-consumer-1.0.jar:na]
> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
> [apns-consumer-1.0.jar:na]
>
>
>