You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (Commented) (JIRA)" <ji...@apache.org> on 2012/04/12 17:01:18 UTC
[jira] [Commented] (KAFKA-331) recurrent produce errors
[ https://issues.apache.org/jira/browse/KAFKA-331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252476#comment-13252476 ]
Jun Rao commented on KAFKA-331:
-------------------------------
The producer is thread safe. Are you using the sync or the async mode in the producer? Do you have a simple test that can reproduce this?
> recurrent produce errors
> ------------------------
>
> Key: KAFKA-331
> URL: https://issues.apache.org/jira/browse/KAFKA-331
> Project: Kafka
> Issue Type: Bug
> Reporter: Pierre-Yves Ritschard
>
> I am using trunk and regularily see such errors popping up:
> 32477890 [kafka-processor-7] ERROR kafka.server.KafkaRequestHandlers - Error processing ProduceRequest on pref^@^@^@:0
> java.io.FileNotFoundException: /mnt/kafka/logs/pref^@^@^@-0/00000000000000000000.kafka (Is a directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at kafka.utils.Utils$.openChannel(Utils.scala:324)
> at kafka.message.FileMessageSet.<init>(FileMessageSet.scala:75)
> at kafka.log.Log.loadSegments(Log.scala:144)
> at kafka.log.Log.<init>(Log.scala:116)
> at kafka.log.LogManager.createLog(LogManager.scala:149)
> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:204)
> at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
> at kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
> at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
> at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
> at kafka.network.Processor.handle(SocketServer.scala:296)
> at kafka.network.Processor.read(SocketServer.scala:319)
> at kafka.network.Processor.run(SocketServer.scala:214)
> at java.lang.Thread.run(Thread.java:679)
> 32477890 [kafka-processor-7] ERROR kafka.network.Processor - Closing socket for /xx.xx.xx.xx because of error
> java.io.FileNotFoundException: /mnt/kafka/logs/pref^@^@^@-0/00000000000000000000.kafka (Is a directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at kafka.utils.Utils$.openChannel(Utils.scala:324)
> at kafka.message.FileMessageSet.<init>(FileMessageSet.scala:75)
> at kafka.log.Log.loadSegments(Log.scala:144)
> at kafka.log.Log.<init>(Log.scala:116)
> at kafka.log.LogManager.createLog(LogManager.scala:149)
> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:204)
> at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
> at kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
> at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
> at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
> at kafka.network.Processor.handle(SocketServer.scala:296)
> at kafka.network.Processor.read(SocketServer.scala:319)
> at kafka.network.Processor.run(SocketServer.scala:214)
> at java.lang.Thread.run(Thread.java:679)
> This results in a "pref" directory created inside the log dir. The original topic should be prefix, somehow a NUL gets inserted there.
> The producing was done with a kafka.javaapi.producer.Producer instance, on which send was called with a kafka.javaapi.producer.ProducerData instance.
> There are no log entries created inside that dir and no impact on the overall operation of the broker operations and consumers.
> Is the producer thread-safe ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (KAFKA-331) recurrent produce errors
Posted by Neha Narkhede <ne...@gmail.com>.
Pierre,
Thanks for filing this bug ! I see this bug intermittently on the 0.8
branch unit tests recently. I'll take a look at reproducing it.
Thanks,
Neha
On Thu, Apr 12, 2012 at 9:48 AM, Jun Rao <ju...@gmail.com> wrote:
> How often does it occur? Are you using sync or async producer? If async,
> are you using the default eventhandler? If so, could you enable trace
> logging there and see if the topic name for each event is correct?
>
> Thanks,
>
> Jun
>
>
> On Thu, Apr 12, 2012 at 8:08 AM, Pierre-Yves Ritschard <py...@spootnik.org>wrote:
>
>> unfortunately i cannot reproduce, this happens every now and then
>>
>> On Thu, Apr 12, 2012 at 5:01 PM, Jun Rao (Commented) (JIRA)
>> <ji...@apache.org> wrote:
>> >
>> > [
>> https://issues.apache.org/jira/browse/KAFKA-331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252476#comment-13252476]
>> >
>> > Jun Rao commented on KAFKA-331:
>> > -------------------------------
>> >
>> > The producer is thread safe. Are you using the sync or the async mode in
>> the producer? Do you have a simple test that can reproduce this?
>> >
>> >> recurrent produce errors
>> >> ------------------------
>> >>
>> >> Key: KAFKA-331
>> >> URL: https://issues.apache.org/jira/browse/KAFKA-331
>> >> Project: Kafka
>> >> Issue Type: Bug
>> >> Reporter: Pierre-Yves Ritschard
>> >>
>> >> I am using trunk and regularily see such errors popping up:
>> >> 32477890 [kafka-processor-7] ERROR kafka.server.KafkaRequestHandlers -
>> Error processing ProduceRequest on pref^@^@^@:0
>> >> java.io.FileNotFoundException:
>> /mnt/kafka/logs/pref^@^@^@-0/00000000000000000000.kafka (Is a directory)
>> >> at java.io.RandomAccessFile.open(Native Method)
>> >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>> >> at kafka.utils.Utils$.openChannel(Utils.scala:324)
>> >> at kafka.message.FileMessageSet.<init>(FileMessageSet.scala:75)
>> >> at kafka.log.Log.loadSegments(Log.scala:144)
>> >> at kafka.log.Log.<init>(Log.scala:116)
>> >> at kafka.log.LogManager.createLog(LogManager.scala:149)
>> >> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:204)
>> >> at
>> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
>> >> at
>> kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
>> >> at
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
>> >> at
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
>> >> at kafka.network.Processor.handle(SocketServer.scala:296)
>> >> at kafka.network.Processor.read(SocketServer.scala:319)
>> >> at kafka.network.Processor.run(SocketServer.scala:214)
>> >> at java.lang.Thread.run(Thread.java:679)
>> >> 32477890 [kafka-processor-7] ERROR kafka.network.Processor - Closing
>> socket for /xx.xx.xx.xx because of error
>> >> java.io.FileNotFoundException:
>> /mnt/kafka/logs/pref^@^@^@-0/00000000000000000000.kafka (Is a directory)
>> >> at java.io.RandomAccessFile.open(Native Method)
>> >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>> >> at kafka.utils.Utils$.openChannel(Utils.scala:324)
>> >> at kafka.message.FileMessageSet.<init>(FileMessageSet.scala:75)
>> >> at kafka.log.Log.loadSegments(Log.scala:144)
>> >> at kafka.log.Log.<init>(Log.scala:116)
>> >> at kafka.log.LogManager.createLog(LogManager.scala:149)
>> >> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:204)
>> >> at
>> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
>> >> at
>> kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
>> >> at
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
>> >> at
>> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
>> >> at kafka.network.Processor.handle(SocketServer.scala:296)
>> >> at kafka.network.Processor.read(SocketServer.scala:319)
>> >> at kafka.network.Processor.run(SocketServer.scala:214)
>> >> at java.lang.Thread.run(Thread.java:679)
>> >> This results in a "pref" directory created inside the log dir. The
>> original topic should be prefix, somehow a NUL gets inserted there.
>> >> The producing was done with a kafka.javaapi.producer.Producer instance,
>> on which send was called with a kafka.javaapi.producer.ProducerData
>> instance.
>> >> There are no log entries created inside that dir and no impact on the
>> overall operation of the broker operations and consumers.
>> >> Is the producer thread-safe ?
>> >
>> > --
>> > This message is automatically generated by JIRA.
>> > If you think it was sent incorrectly, please contact your JIRA
>> administrators:
>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>> > For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>> >
>> >
>>
Re: [jira] [Commented] (KAFKA-331) recurrent produce errors
Posted by Jun Rao <ju...@gmail.com>.
How often does it occur? Are you using sync or async producer? If async,
are you using the default eventhandler? If so, could you enable trace
logging there and see if the topic name for each event is correct?
Thanks,
Jun
On Thu, Apr 12, 2012 at 8:08 AM, Pierre-Yves Ritschard <py...@spootnik.org>wrote:
> unfortunately i cannot reproduce, this happens every now and then
>
> On Thu, Apr 12, 2012 at 5:01 PM, Jun Rao (Commented) (JIRA)
> <ji...@apache.org> wrote:
> >
> > [
> https://issues.apache.org/jira/browse/KAFKA-331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252476#comment-13252476]
> >
> > Jun Rao commented on KAFKA-331:
> > -------------------------------
> >
> > The producer is thread safe. Are you using the sync or the async mode in
> the producer? Do you have a simple test that can reproduce this?
> >
> >> recurrent produce errors
> >> ------------------------
> >>
> >> Key: KAFKA-331
> >> URL: https://issues.apache.org/jira/browse/KAFKA-331
> >> Project: Kafka
> >> Issue Type: Bug
> >> Reporter: Pierre-Yves Ritschard
> >>
> >> I am using trunk and regularily see such errors popping up:
> >> 32477890 [kafka-processor-7] ERROR kafka.server.KafkaRequestHandlers -
> Error processing ProduceRequest on pref^@^@^@:0
> >> java.io.FileNotFoundException:
> /mnt/kafka/logs/pref^@^@^@-0/00000000000000000000.kafka (Is a directory)
> >> at java.io.RandomAccessFile.open(Native Method)
> >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> >> at kafka.utils.Utils$.openChannel(Utils.scala:324)
> >> at kafka.message.FileMessageSet.<init>(FileMessageSet.scala:75)
> >> at kafka.log.Log.loadSegments(Log.scala:144)
> >> at kafka.log.Log.<init>(Log.scala:116)
> >> at kafka.log.LogManager.createLog(LogManager.scala:149)
> >> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:204)
> >> at
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
> >> at
> kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
> >> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
> >> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
> >> at kafka.network.Processor.handle(SocketServer.scala:296)
> >> at kafka.network.Processor.read(SocketServer.scala:319)
> >> at kafka.network.Processor.run(SocketServer.scala:214)
> >> at java.lang.Thread.run(Thread.java:679)
> >> 32477890 [kafka-processor-7] ERROR kafka.network.Processor - Closing
> socket for /xx.xx.xx.xx because of error
> >> java.io.FileNotFoundException:
> /mnt/kafka/logs/pref^@^@^@-0/00000000000000000000.kafka (Is a directory)
> >> at java.io.RandomAccessFile.open(Native Method)
> >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> >> at kafka.utils.Utils$.openChannel(Utils.scala:324)
> >> at kafka.message.FileMessageSet.<init>(FileMessageSet.scala:75)
> >> at kafka.log.Log.loadSegments(Log.scala:144)
> >> at kafka.log.Log.<init>(Log.scala:116)
> >> at kafka.log.LogManager.createLog(LogManager.scala:149)
> >> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:204)
> >> at
> kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
> >> at
> kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
> >> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
> >> at
> kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
> >> at kafka.network.Processor.handle(SocketServer.scala:296)
> >> at kafka.network.Processor.read(SocketServer.scala:319)
> >> at kafka.network.Processor.run(SocketServer.scala:214)
> >> at java.lang.Thread.run(Thread.java:679)
> >> This results in a "pref" directory created inside the log dir. The
> original topic should be prefix, somehow a NUL gets inserted there.
> >> The producing was done with a kafka.javaapi.producer.Producer instance,
> on which send was called with a kafka.javaapi.producer.ProducerData
> instance.
> >> There are no log entries created inside that dir and no impact on the
> overall operation of the broker operations and consumers.
> >> Is the producer thread-safe ?
> >
> > --
> > This message is automatically generated by JIRA.
> > If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
> >
>
Re: [jira] [Commented] (KAFKA-331) recurrent produce errors
Posted by Pierre-Yves Ritschard <py...@spootnik.org>.
unfortunately i cannot reproduce, this happens every now and then
On Thu, Apr 12, 2012 at 5:01 PM, Jun Rao (Commented) (JIRA)
<ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/KAFKA-331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252476#comment-13252476 ]
>
> Jun Rao commented on KAFKA-331:
> -------------------------------
>
> The producer is thread safe. Are you using the sync or the async mode in the producer? Do you have a simple test that can reproduce this?
>
>> recurrent produce errors
>> ------------------------
>>
>> Key: KAFKA-331
>> URL: https://issues.apache.org/jira/browse/KAFKA-331
>> Project: Kafka
>> Issue Type: Bug
>> Reporter: Pierre-Yves Ritschard
>>
>> I am using trunk and regularily see such errors popping up:
>> 32477890 [kafka-processor-7] ERROR kafka.server.KafkaRequestHandlers - Error processing ProduceRequest on pref^@^@^@:0
>> java.io.FileNotFoundException: /mnt/kafka/logs/pref^@^@^@-0/00000000000000000000.kafka (Is a directory)
>> at java.io.RandomAccessFile.open(Native Method)
>> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>> at kafka.utils.Utils$.openChannel(Utils.scala:324)
>> at kafka.message.FileMessageSet.<init>(FileMessageSet.scala:75)
>> at kafka.log.Log.loadSegments(Log.scala:144)
>> at kafka.log.Log.<init>(Log.scala:116)
>> at kafka.log.LogManager.createLog(LogManager.scala:149)
>> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:204)
>> at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
>> at kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
>> at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
>> at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
>> at kafka.network.Processor.handle(SocketServer.scala:296)
>> at kafka.network.Processor.read(SocketServer.scala:319)
>> at kafka.network.Processor.run(SocketServer.scala:214)
>> at java.lang.Thread.run(Thread.java:679)
>> 32477890 [kafka-processor-7] ERROR kafka.network.Processor - Closing socket for /xx.xx.xx.xx because of error
>> java.io.FileNotFoundException: /mnt/kafka/logs/pref^@^@^@-0/00000000000000000000.kafka (Is a directory)
>> at java.io.RandomAccessFile.open(Native Method)
>> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>> at kafka.utils.Utils$.openChannel(Utils.scala:324)
>> at kafka.message.FileMessageSet.<init>(FileMessageSet.scala:75)
>> at kafka.log.Log.loadSegments(Log.scala:144)
>> at kafka.log.Log.<init>(Log.scala:116)
>> at kafka.log.LogManager.createLog(LogManager.scala:149)
>> at kafka.log.LogManager.getOrCreateLog(LogManager.scala:204)
>> at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
>> at kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
>> at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
>> at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
>> at kafka.network.Processor.handle(SocketServer.scala:296)
>> at kafka.network.Processor.read(SocketServer.scala:319)
>> at kafka.network.Processor.run(SocketServer.scala:214)
>> at java.lang.Thread.run(Thread.java:679)
>> This results in a "pref" directory created inside the log dir. The original topic should be prefix, somehow a NUL gets inserted there.
>> The producing was done with a kafka.javaapi.producer.Producer instance, on which send was called with a kafka.javaapi.producer.ProducerData instance.
>> There are no log entries created inside that dir and no impact on the overall operation of the broker operations and consumers.
>> Is the producer thread-safe ?
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>