You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Holger Buss <HB...@gmx.de> on 2012/10/01 13:12:30 UTC
Kafka Start failure because empty folders
Hi,
we had the problem, that the start of kafka brokes down, because the written Log directories are empty, have strange names and doesn't exist in Zookeeper as a topic.
Here are a ls from the kafka logs folder:
/var/lib/kafka/logs/
drwxr-xr-x 2 kafka users 4096 24. Aug 13:22 r
drwxr-xr-x 2 kafka users 4096 24. Aug 00:54 re
drwxr-xr-x 2 kafka users 4096 29. Aug 01:34 rec
drwxr-xr-x 2 kafka users 4096 12. Sep 05:30 reci
drwxr-xr-x 2 kafka users 4096 22. Aug 01:08 recip
drwxr-xr-x 2 kafka users 4096 11. Aug 06:22 recipi
drwxr-xr-x 2 kafka users 4096 23. Aug 05:12 recipie
drwxr-xr-x 2 kafka users 4096 5. Sep 01:08 recipien
drwxr-xr-x 2 kafka users 4096 22. Aug 00:50 recipient
drwxr-xr-x 2 kafka users 4096 27. Aug 01:13 recipientE
drwxr-xr-x 2 kafka users 4096 4. Sep 01:02 recipientEv
drwxr-xr-x 2 kafka users 4096 13. Aug 05:53 recipientEve
drwxr-xr-x 2 kafka users 4096 23. Aug 01:06 recipientEven
drwxr-xr-x 2 kafka users 4096 4. Sep 01:07 recipientEvent
drwxr-xr-x 2 kafka users 4096 20. Sep 10:32 recipientEvents-0
drwxr-xr-x 2 kafka users 4096 20. Sep 10:38 recipientEvents-1
drwxr-xr-x 2 kafka users 4096 20. Sep 09:41 recipientEvents-10
drwxr-xr-x 2 kafka users 4096 20. Sep 09:26 recipientEvents-11
drwxr-xr-x 2 kafka users 4096 20. Sep 10:57 recipientEvents-12
drwxr-xr-x 2 kafka users 4096 20. Sep 11:24 recipientEvents-13
drwxr-xr-x 2 kafka users 4096 20. Sep 09:41 recipientEvents-14
drwxr-xr-x 2 kafka users 4096 20. Sep 10:51 recipientEvents-15
drwxr-xr-x 2 kafka users 4096 20. Sep 11:41 recipientEvents-16
drwxr-xr-x 2 kafka users 4096 20. Sep 10:06 recipientEvents-17
drwxr-xr-x 2 kafka users 4096 20. Sep 09:26 recipientEvents-18
drwxr-xr-x 2 kafka users 4096 20. Sep 12:02 recipientEvents-19
drwxr-xr-x 2 kafka users 4096 20. Sep 10:28 recipientEvents-2
drwxr-xr-x 2 kafka users 4096 20. Sep 10:45 recipientEvents-3
drwxr-xr-x 2 kafka users 4096 20. Sep 09:31 recipientEvents-4
drwxr-xr-x 2 kafka users 4096 20. Sep 09:03 recipientEvents-5
The error Meesage at kafka startup:
Picked up _JAVA_OPTIONS: -XX:DefaultInitialRAMFraction=128 -XX:DefaultMaxRAMFraction=16
[2012-09-20 12:03:32,509] INFO Starting Kafka server... (kafka.server.KafkaServer)
[2012-09-20 12:03:32,588] INFO Loading log 'systemEvents-1' (kafka.log.LogManager)
[2012-09-20 12:03:32,646] INFO Loading the last segment /var/kafka_1_02/logs/systemEvents-1/00000000000000000000.kafka in mutable mode, recovery false (kafka.log.Log)
[2012-09-20 12:03:32,675] INFO Loading log 'recipientEvents-3' (kafka.log.LogManager)
[2012-09-20 12:03:32,680] INFO Loading the last segment /var/kafka_1_02/logs/recipientEvents-3/00000000011691660453.kafka in mutable mode, recovery false (kafka.log.Log)
[2012-09-20 12:03:32,681] INFO Loading log 'recipi' (kafka.log.LogManager)
[2012-09-20 12:03:32,697] FATAL Fatal error during KafkaServerStable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1937)
at kafka.utils.Utils$.getTopicPartition(Utils.scala:558)
at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:71)
at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:65)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
at kafka.log.LogManager.<init>(LogManager.scala:65)
at kafka.server.KafkaServer.startup(KafkaServer.scala:58)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:50)
at kafka.Kafka.main(Kafka.scala)
[2012-09-20 12:03:32,700] INFO Shutting down Kafka server (kafka.server.KafkaServer)
[2012-09-20 12:03:32,700] INFO shutdown scheduler kafka-logcleaner- (kafka.utils.KafkaScheduler)
[2012-09-20 12:03:32,701] INFO Kafka server shut down completed (kafka.server.KafkaServer)
r
re
rec
reci
recip
recipi
recipie
recipien
recipient
recipientE
recipientEv
recipientEve
recipientEven
recipientEvent
Who create this empty folders? Possible broken Messages?
Best regards
Holger Buss
Re: Kafka Start failure because empty folders
Posted by Jun Rao <ju...@gmail.com>.
Interesting. A valid directory name is of the form topic-partitionId.
What's under those invalid directories (e.g, r, re, etc)? We recently fixed
a topic name issue (https://issues.apache.org/jira/browse/KAFKA-495). Not
sure if it's related. Also, are you using a non-java producer?
Thanks,
Jun
On Mon, Oct 1, 2012 at 4:12 AM, Holger Buss <HB...@gmx.de> wrote:
> Hi,
>
> we had the problem, that the start of kafka brokes down, because the
> written Log directories are empty, have strange names and doesn't exist in
> Zookeeper as a topic.
>
> Here are a ls from the kafka logs folder:
>
> /var/lib/kafka/logs/
> drwxr-xr-x 2 kafka users 4096 24. Aug 13:22 r
> drwxr-xr-x 2 kafka users 4096 24. Aug 00:54 re
> drwxr-xr-x 2 kafka users 4096 29. Aug 01:34 rec
> drwxr-xr-x 2 kafka users 4096 12. Sep 05:30 reci
> drwxr-xr-x 2 kafka users 4096 22. Aug 01:08 recip
> drwxr-xr-x 2 kafka users 4096 11. Aug 06:22 recipi
> drwxr-xr-x 2 kafka users 4096 23. Aug 05:12 recipie
> drwxr-xr-x 2 kafka users 4096 5. Sep 01:08 recipien
> drwxr-xr-x 2 kafka users 4096 22. Aug 00:50 recipient
> drwxr-xr-x 2 kafka users 4096 27. Aug 01:13 recipientE
> drwxr-xr-x 2 kafka users 4096 4. Sep 01:02 recipientEv
> drwxr-xr-x 2 kafka users 4096 13. Aug 05:53 recipientEve
> drwxr-xr-x 2 kafka users 4096 23. Aug 01:06 recipientEven
> drwxr-xr-x 2 kafka users 4096 4. Sep 01:07 recipientEvent
> drwxr-xr-x 2 kafka users 4096 20. Sep 10:32 recipientEvents-0
> drwxr-xr-x 2 kafka users 4096 20. Sep 10:38 recipientEvents-1
> drwxr-xr-x 2 kafka users 4096 20. Sep 09:41 recipientEvents-10
> drwxr-xr-x 2 kafka users 4096 20. Sep 09:26 recipientEvents-11
> drwxr-xr-x 2 kafka users 4096 20. Sep 10:57 recipientEvents-12
> drwxr-xr-x 2 kafka users 4096 20. Sep 11:24 recipientEvents-13
> drwxr-xr-x 2 kafka users 4096 20. Sep 09:41 recipientEvents-14
> drwxr-xr-x 2 kafka users 4096 20. Sep 10:51 recipientEvents-15
> drwxr-xr-x 2 kafka users 4096 20. Sep 11:41 recipientEvents-16
> drwxr-xr-x 2 kafka users 4096 20. Sep 10:06 recipientEvents-17
> drwxr-xr-x 2 kafka users 4096 20. Sep 09:26 recipientEvents-18
> drwxr-xr-x 2 kafka users 4096 20. Sep 12:02 recipientEvents-19
> drwxr-xr-x 2 kafka users 4096 20. Sep 10:28 recipientEvents-2
> drwxr-xr-x 2 kafka users 4096 20. Sep 10:45 recipientEvents-3
> drwxr-xr-x 2 kafka users 4096 20. Sep 09:31 recipientEvents-4
> drwxr-xr-x 2 kafka users 4096 20. Sep 09:03 recipientEvents-5
>
>
> The error Meesage at kafka startup:
>
> Picked up _JAVA_OPTIONS: -XX:DefaultInitialRAMFraction=128
> -XX:DefaultMaxRAMFraction=16
> [2012-09-20 12:03:32,509] INFO Starting Kafka server...
> (kafka.server.KafkaServer)
> [2012-09-20 12:03:32,588] INFO Loading log 'systemEvents-1'
> (kafka.log.LogManager)
> [2012-09-20 12:03:32,646] INFO Loading the last segment
> /var/kafka_1_02/logs/systemEvents-1/00000000000000000000.kafka in mutable
> mode, recovery false (kafka.log.Log)
> [2012-09-20 12:03:32,675] INFO Loading log 'recipientEvents-3'
> (kafka.log.LogManager)
> [2012-09-20 12:03:32,680] INFO Loading the last segment
> /var/kafka_1_02/logs/recipientEvents-3/00000000011691660453.kafka in
> mutable mode, recovery false (kafka.log.Log)
> [2012-09-20 12:03:32,681] INFO Loading log 'recipi' (kafka.log.LogManager)
> [2012-09-20 12:03:32,697] FATAL Fatal error during KafkaServerStable
> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1937)
> at kafka.utils.Utils$.getTopicPartition(Utils.scala:558)
> at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:71)
> at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:65)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
> at kafka.log.LogManager.<init>(LogManager.scala:65)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:58)
> at
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> at kafka.Kafka$.main(Kafka.scala:50)
> at kafka.Kafka.main(Kafka.scala)
> [2012-09-20 12:03:32,700] INFO Shutting down Kafka server
> (kafka.server.KafkaServer)
> [2012-09-20 12:03:32,700] INFO shutdown scheduler kafka-logcleaner-
> (kafka.utils.KafkaScheduler)
> [2012-09-20 12:03:32,701] INFO Kafka server shut down completed
> (kafka.server.KafkaServer)
>
> r
> re
> rec
> reci
> recip
> recipi
> recipie
> recipien
> recipient
> recipientE
> recipientEv
> recipientEve
> recipientEven
> recipientEvent
>
> Who create this empty folders? Possible broken Messages?
>
> Best regards
> Holger Buss
>