You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vjeran Marcinko <vj...@email.t-com.hr> on 2014/01/25 09:43:50 UTC
Using Kafka on Windows - file path problems
Hi,
I have a problem going through start guide on Windows. Basically what I
found are file path problems it seems:
If I try to set Kafka log dir path in server.properties using Windows file
separator, such as
D:\tmp\kafka-logs, or even
\tmp\kafka-logs
, then kafka cannot be started, since it throws
[2014-01-25 08:04:00,495] FATAL Fatal error during KafkaServerStable
startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.io.IOException: The filename, directory name, or volume label syntax is
incorrect
at java.io.WinNTFileSystem.canonicalize0(Native Method)
at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396)
at java.io.File.getCanonicalPath(File.java:559)
at
kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc
ala:70)
at
kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc
ala:70)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
206)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala
:34)
at
scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.mutable.WrappedArray.map(WrappedArray.scala:32)
at
kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:70)
at kafka.log.LogManager.<init>(LogManager.scala:60)
at kafka.server.KafkaServer.startup(KafkaServer.scala:63)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
If I use Unix file separator in settings, such as default /tmp/kafka-logs,
(and for those who don't use Windows, java on Windows accepts paths that
have unix path separator such as D:/tmp/some.file), then Kafka can be
started, but during topic creation I get following error which I assume is
due to trying to find the key using windows file separator as given in
exception log:
[2014-01-25 08:05:53,681] ERROR Error on broker 0 while processing
LeaderAndIsr request correlationId 6 received from controller 0 epoch 1 for
partition (mytopic,0) (state.change.logger)
java.util.NoSuchElementException: key not found: \tmp\kafka-logs
at scala.collection.MapLike$class.default(MapLike.scala:223)
at scala.collection.immutable.Map$Map1.default(Map.scala:93)
at scala.collection.MapLike$class.apply(MapLike.scala:134)
at scala.collection.immutable.Map$Map1.apply(Map.scala:93)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:88)
at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154)
at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
206)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:6
1)
at scala.collection.immutable.List.foreach(List.scala:45)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.List.map(List.scala:45)
at kafka.cluster.Partition.makeLeader(Partition.scala:154)
at
kafka.server.ReplicaManager.kafka$server$ReplicaManager$$makeLeader(ReplicaM
anager.scala:257)
at
kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(ReplicaM
anager.scala:221)
at
kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(ReplicaM
anager.scala:213)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:105)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:213)
at
kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:86)
at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
at
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:662)
Regards,
Vjeran
RE: Using Kafka on Windows - file path problems
Posted by Vjeran Marcinko <vj...@email.t-com.hr>.
Ugh, stupid me, I could see what was the problem from mentioned logged
lines.
One should escape the path if using \ char, since it interprets the \tmp as
tab followed by "mp" so using:
\\tmp\\kafka-logs work, and I guess that its normal behavior for
java.util.Properties.load() method.
-Vjeran
-----Original Message-----
From: Vjeran Marcinko [mailto:vjeran.marcinko@email.t-com.hr]
Sent: Saturday, January 25, 2014 6:10 PM
To: users@kafka.apache.org
Subject: RE: Using Kafka on Windows - file path problems
Even if in that one case the thing would work, it would be best if any
correct windows file path would work.
I checked again, and it behaves as I've written.
In case when one specifies /tmp/kafka-logs, it works, and during startup one
log line says:
[2014-01-25 18:04:08,335] INFO Property log.dirs is overridden to
/tmp/kafka-logs (kafka.utils.VerifiableProperties)
, but as I said during topic creation it fails.
But when I specify the path as \tmp\kafka-logs, then the startup fails, and
the same logged line says:
[2014-01-25 18:07:51,395] INFO Property log.dirs is overridden to
mpkafka-logs (kafka.utils.VerifiableProperties)
And it fails with mentioned:
java.io.IOException: The filename, directory name, or volume label syntax is
incorrect
at java.io.WinNTFileSystem.canonicalize0(Native Method)
at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396)
at java.io.File.getCanonicalPath(File.java:559)
at
kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc
ala:70)
at
kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc
ala:70)
The same startup failure occurs, when I specify full windows file path as
D:\tmp\kafka-logs, and then the logged line is:
[2014-01-25 18:08:44,377] INFO Property log.dirs is overridden to D:
mpkafka-logs (kafka.utils.VerifiableProperties)
I hope this helps somewhat.
-Vjeran
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Saturday, January 25, 2014 5:25 PM
To: users@kafka.apache.org
Subject: Re: Using Kafka on Windows - file path problems
It sounds like \tmp\kafka-logs should work since that's what
File.getParent() returns. Not sure why you can't use that to create the
file.
Thanks,
Jun
On Sat, Jan 25, 2014 at 12:43 AM, Vjeran Marcinko <
vjeran.marcinko@email.t-com.hr> wrote:
> Hi,
>
> I have a problem going through start guide on Windows. Basically what
> I found are file path problems it seems:
>
> If I try to set Kafka log dir path in server.properties using Windows
> file separator, such as D:\tmp\kafka-logs, or even \tmp\kafka-logs ,
> then kafka cannot be started, since it throws
>
> [2014-01-25 08:04:00,495] FATAL Fatal error during KafkaServerStable
> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> java.io.IOException: The filename, directory name, or volume label
> syntax is incorrect
> at java.io.WinNTFileSystem.canonicalize0(Native Method)
> at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396)
> at java.io.File.getCanonicalPath(File.java:559)
> at
>
> kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogMana
> ger.sc
> ala:70)
> at
>
> kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogMana
> ger.sc
> ala:70)
> at
>
>
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
>
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized
> .scala
> :34)
> at
> scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at
scala.collection.mutable.WrappedArray.map(WrappedArray.scala:32)
> at
> kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:70)
> at kafka.log.LogManager.<init>(LogManager.scala:60)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:63)
> at
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> at kafka.Kafka$.main(Kafka.scala:46)
>
> If I use Unix file separator in settings, such as default
> /tmp/kafka-logs, (and for those who don't use Windows, java on Windows
> accepts paths that have unix path separator such as D:/tmp/some.file),
> then Kafka can be started, but during topic creation I get following
> error which I assume is due to trying to find the key using windows
> file separator as given in exception log:
>
> [2014-01-25 08:05:53,681] ERROR Error on broker 0 while processing
> LeaderAndIsr request correlationId 6 received from controller 0 epoch
> 1 for partition (mytopic,0) (state.change.logger)
> java.util.NoSuchElementException: key not found: \tmp\kafka-logs
> at scala.collection.MapLike$class.default(MapLike.scala:223)
> at scala.collection.immutable.Map$Map1.default(Map.scala:93)
> at scala.collection.MapLike$class.apply(MapLike.scala:134)
> at scala.collection.immutable.Map$Map1.apply(Map.scala:93)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:88)
> at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154)
> at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154)
> at
>
>
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
>
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.s
> cala:6
> 1)
> at scala.collection.immutable.List.foreach(List.scala:45)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.List.map(List.scala:45)
> at kafka.cluster.Partition.makeLeader(Partition.scala:154)
> at
>
> kafka.server.ReplicaManager.kafka$server$ReplicaManager$$makeLeader(Re
> plicaM
> anager.scala:257)
> at
>
> kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(Re
> plicaM
> anager.scala:221)
> at
>
> kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(Re
> plicaM
> anager.scala:213)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:105)
> at
>
>
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:213)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:86)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
> at
> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
> at java.lang.Thread.run(Thread.java:662)
>
> Regards,
> Vjeran
>
>
>
RE: Using Kafka on Windows - file path problems
Posted by Vjeran Marcinko <vj...@email.t-com.hr>.
Even if in that one case the thing would work, it would be best if any
correct windows file path would work.
I checked again, and it behaves as I've written.
In case when one specifies /tmp/kafka-logs, it works, and during startup one
log line says:
[2014-01-25 18:04:08,335] INFO Property log.dirs is overridden to
/tmp/kafka-logs (kafka.utils.VerifiableProperties)
, but as I said during topic creation it fails.
But when I specify the path as \tmp\kafka-logs, then the startup fails, and
the same logged line says:
[2014-01-25 18:07:51,395] INFO Property log.dirs is overridden to
mpkafka-logs (kafka.utils.VerifiableProperties)
And it fails with mentioned:
java.io.IOException: The filename, directory name, or volume label syntax is
incorrect
at java.io.WinNTFileSystem.canonicalize0(Native Method)
at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396)
at java.io.File.getCanonicalPath(File.java:559)
at
kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc
ala:70)
at
kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc
ala:70)
The same startup failure occurs, when I specify full windows file path as
D:\tmp\kafka-logs, and then the logged line is:
[2014-01-25 18:08:44,377] INFO Property log.dirs is overridden to D:
mpkafka-logs (kafka.utils.VerifiableProperties)
I hope this helps somewhat.
-Vjeran
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Saturday, January 25, 2014 5:25 PM
To: users@kafka.apache.org
Subject: Re: Using Kafka on Windows - file path problems
It sounds like \tmp\kafka-logs should work since that's what
File.getParent() returns. Not sure why you can't use that to create the
file.
Thanks,
Jun
On Sat, Jan 25, 2014 at 12:43 AM, Vjeran Marcinko <
vjeran.marcinko@email.t-com.hr> wrote:
> Hi,
>
> I have a problem going through start guide on Windows. Basically what
> I found are file path problems it seems:
>
> If I try to set Kafka log dir path in server.properties using Windows
> file separator, such as D:\tmp\kafka-logs, or even \tmp\kafka-logs ,
> then kafka cannot be started, since it throws
>
> [2014-01-25 08:04:00,495] FATAL Fatal error during KafkaServerStable
> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> java.io.IOException: The filename, directory name, or volume label
> syntax is incorrect
> at java.io.WinNTFileSystem.canonicalize0(Native Method)
> at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396)
> at java.io.File.getCanonicalPath(File.java:559)
> at
>
> kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogMana
> ger.sc
> ala:70)
> at
>
> kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogMana
> ger.sc
> ala:70)
> at
>
>
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
>
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized
> .scala
> :34)
> at
> scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at
scala.collection.mutable.WrappedArray.map(WrappedArray.scala:32)
> at
> kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:70)
> at kafka.log.LogManager.<init>(LogManager.scala:60)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:63)
> at
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> at kafka.Kafka$.main(Kafka.scala:46)
>
> If I use Unix file separator in settings, such as default
> /tmp/kafka-logs, (and for those who don't use Windows, java on Windows
> accepts paths that have unix path separator such as D:/tmp/some.file),
> then Kafka can be started, but during topic creation I get following
> error which I assume is due to trying to find the key using windows
> file separator as given in exception log:
>
> [2014-01-25 08:05:53,681] ERROR Error on broker 0 while processing
> LeaderAndIsr request correlationId 6 received from controller 0 epoch
> 1 for partition (mytopic,0) (state.change.logger)
> java.util.NoSuchElementException: key not found: \tmp\kafka-logs
> at scala.collection.MapLike$class.default(MapLike.scala:223)
> at scala.collection.immutable.Map$Map1.default(Map.scala:93)
> at scala.collection.MapLike$class.apply(MapLike.scala:134)
> at scala.collection.immutable.Map$Map1.apply(Map.scala:93)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:88)
> at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154)
> at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154)
> at
>
>
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
>
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.s
> cala:6
> 1)
> at scala.collection.immutable.List.foreach(List.scala:45)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.List.map(List.scala:45)
> at kafka.cluster.Partition.makeLeader(Partition.scala:154)
> at
>
> kafka.server.ReplicaManager.kafka$server$ReplicaManager$$makeLeader(Re
> plicaM
> anager.scala:257)
> at
>
> kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(Re
> plicaM
> anager.scala:221)
> at
>
> kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(Re
> plicaM
> anager.scala:213)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:105)
> at
>
>
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:213)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:86)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
> at
> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
> at java.lang.Thread.run(Thread.java:662)
>
> Regards,
> Vjeran
>
>
>
Re: Using Kafka on Windows - file path problems
Posted by Jun Rao <ju...@gmail.com>.
It sounds like \tmp\kafka-logs should work since that's what
File.getParent() returns. Not sure why you can't use that to create the
file.
Thanks,
Jun
On Sat, Jan 25, 2014 at 12:43 AM, Vjeran Marcinko <
vjeran.marcinko@email.t-com.hr> wrote:
> Hi,
>
> I have a problem going through start guide on Windows. Basically what I
> found are file path problems it seems:
>
> If I try to set Kafka log dir path in server.properties using Windows file
> separator, such as
> D:\tmp\kafka-logs, or even
> \tmp\kafka-logs
> , then kafka cannot be started, since it throws
>
> [2014-01-25 08:04:00,495] FATAL Fatal error during KafkaServerStable
> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> java.io.IOException: The filename, directory name, or volume label syntax
> is
> incorrect
> at java.io.WinNTFileSystem.canonicalize0(Native Method)
> at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396)
> at java.io.File.getCanonicalPath(File.java:559)
> at
>
> kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc
> ala:70)
> at
>
> kafka.log.LogManager$$anonfun$createAndValidateLogDirs$1.apply(LogManager.sc
> ala:70)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala
> :34)
> at
> scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.mutable.WrappedArray.map(WrappedArray.scala:32)
> at
> kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:70)
> at kafka.log.LogManager.<init>(LogManager.scala:60)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:63)
> at
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> at kafka.Kafka$.main(Kafka.scala:46)
>
> If I use Unix file separator in settings, such as default /tmp/kafka-logs,
> (and for those who don't use Windows, java on Windows accepts paths that
> have unix path separator such as D:/tmp/some.file), then Kafka can be
> started, but during topic creation I get following error which I assume is
> due to trying to find the key using windows file separator as given in
> exception log:
>
> [2014-01-25 08:05:53,681] ERROR Error on broker 0 while processing
> LeaderAndIsr request correlationId 6 received from controller 0 epoch 1 for
> partition (mytopic,0) (state.change.logger)
> java.util.NoSuchElementException: key not found: \tmp\kafka-logs
> at scala.collection.MapLike$class.default(MapLike.scala:223)
> at scala.collection.immutable.Map$Map1.default(Map.scala:93)
> at scala.collection.MapLike$class.apply(MapLike.scala:134)
> at scala.collection.immutable.Map$Map1.apply(Map.scala:93)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:88)
> at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154)
> at kafka.cluster.Partition$$anonfun$1.apply(Partition.scala:154)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:
> 206)
> at
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:6
> 1)
> at scala.collection.immutable.List.foreach(List.scala:45)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.List.map(List.scala:45)
> at kafka.cluster.Partition.makeLeader(Partition.scala:154)
> at
>
> kafka.server.ReplicaManager.kafka$server$ReplicaManager$$makeLeader(ReplicaM
> anager.scala:257)
> at
>
> kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(ReplicaM
> anager.scala:221)
> at
>
> kafka.server.ReplicaManager$$anonfun$becomeLeaderOrFollower$3.apply(ReplicaM
> anager.scala:213)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:105)
> at
>
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:213)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:86)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
> at
> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
> at java.lang.Thread.run(Thread.java:662)
>
> Regards,
> Vjeran
>
>
>