You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Young, Ben" <Be...@fisglobal.com> on 2018/03/20 09:37:16 UTC
Issue deleting topic on Windows
Hi,
We have an issue in our dev environments when running on Windows. To provide a clean environment for testing we tear down the topics and recreate them, but deleting the topics is causing the server to crash. We then need to delete the logs directory for it to be able to restart, it doesn't seem to be able to recover.
The backtrace is below. I wonder if something still has a handle open on the folder before the rename is attempted?
Thanks,
Ben
[2018-03-20 09:31:24,969] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions TradeFeed-3 (kafka.server.ReplicaFetcherManager)
[2018-03-20 09:31:25,012] ERROR Error while deleting TradeFeed-3 in dir f:\tmp\kafka-logs. (kafka.server.LogDirFailureChannel)
java.io.IOException: Failed to rename log directory from f:\tmp\kafka-logs\TradeFeed-3 to f:\tmp\kafka-logs\TradeFeed-3.23a377d8b9b042a9853adc5deec53a2f-delete
at kafka.log.LogManager.asyncDelete(LogManager.scala:671)
at kafka.cluster.Partition.$anonfun$delete$1(Partition.scala:178)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:225)
at kafka.cluster.Partition.delete(Partition.scala:173)
at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:341)
at kafka.server.ReplicaManager.$anonfun$stopReplicas$2(ReplicaManager.scala:373)
at scala.collection.Iterator.foreach(Iterator.scala:929)
at scala.collection.Iterator.foreach$(Iterator.scala:929)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
at scala.collection.IterableLike.foreach(IterableLike.scala:71)
at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:371)
at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:190)
at kafka.server.KafkaApis.handle(KafkaApis.scala:104)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:65)
at java.lang.Thread.run(Unknown Source)
[2018-03-20 09:31:25,013] INFO [ReplicaManager broker=0] Stopping serving replicas in dir f:\tmp\kafka-logs (kafka.server.ReplicaManager)
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
RE: Issue deleting topic on Windows
Posted by "Young, Ben" <Be...@fisglobal.com>.
Hi Lawrence,
Thanks for that! I'll take a look at your patches. Hopefully they can be merged into a future release
Ben
-----Original Message-----
From: Lawrence Craft [mailto:lawrence.craft@gmail.com]
Sent: 22 March 2018 19:27
To: users@kafka.apache.org
Subject: Re: Issue deleting topic on Windows
Hi Ben,
This sounds very similar to KAFKA-1194, an issue we encountered on our Windows servers at work. Basically, Kafka is unable to rename log files as they're locked by the OS. The bug is here and is still open:
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-1194&data=02%7C01%7CBen.Young%40fisglobal.com%7Cc91c72cdc5374631599708d5902af694%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C636573436557774999&sdata=bKbx6bYMSuIA5mNtPHVkWRwin1W7Uea3R2OzjPkBRIQ%3D&reserved=0. I am not aware of any straightforward workaround.
To get around this, we've had to modify the code and run a custom build.
You can try the changes from here: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F4431&data=02%7C01%7CBen.Young%40fisglobal.com%7Cc91c72cdc5374631599708d5902af694%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C636573436557774999&sdata=V0dXffpRMAvBU8vyQH6ZW71WpmuhFGB4uo5vQdMs1cM%3D&reserved=0
Hope this helps!
Lawrence
On Thu, Mar 22, 2018 at 9:34 AM, Young, Ben <Be...@fisglobal.com> wrote:
> Hi,
>
> Any thoughts on the below?
>
> Thanks!
>
> -----Original Message-----
>
> Sorry, I forgot to mention we're using version: kafka_2.12-1.0.0
>
> -----Original Message-----
>
> Hi,
>
> We have an issue in our dev environments when running on Windows. To
> provide a clean environment for testing we tear down the topics and
> recreate them, but deleting the topics is causing the server to crash.
> We then need to delete the logs directory for it to be able to
> restart, it doesn't seem to be able to recover.
>
> The backtrace is below. I wonder if something still has a handle open
> on the folder before the rename is attempted?
>
> Thanks,
> Ben
>
> [2018-03-20 09:31:24,969] INFO [ReplicaFetcherManager on broker 0]
> Removed fetcher for partitions TradeFeed-3
> (kafka.server.ReplicaFetcherManager)
> [2018-03-20 09:31:25,012] ERROR Error while deleting TradeFeed-3 in
> dir f:\tmp\kafka-logs. (kafka.server.LogDirFailureChannel)
> java.io.IOException: Failed to rename log directory from
> f:\tmp\kafka-logs\TradeFeed-3 to f:\tmp\kafka-logs\TradeFeed-3.
> 23a377d8b9b042a9853adc5deec53a2f-delete
> at kafka.log.LogManager.asyncDelete(LogManager.scala:671)
> at kafka.cluster.Partition.$anonfun$delete$1(Partition.scala:178)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:225)
> at kafka.cluster.Partition.delete(Partition.scala:173)
> at kafka.server.ReplicaManager.stopReplica(ReplicaManager.
> scala:341)
> at kafka.server.ReplicaManager.$anonfun$stopReplicas$2(
> ReplicaManager.scala:373)
> at scala.collection.Iterator.foreach(Iterator.scala:929)
> at scala.collection.Iterator.foreach$(Iterator.scala:929)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
> at scala.collection.IterableLike.foreach(IterableLike.scala:71)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.
> scala:371)
> at kafka.server.KafkaApis.handleStopReplicaRequest(
> KafkaApis.scala:190)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:104)
> at kafka.server.KafkaRequestHandler.run(
> KafkaRequestHandler.scala:65)
> at java.lang.Thread.run(Unknown Source)
> [2018-03-20 09:31:25,013] INFO [ReplicaManager broker=0] Stopping
> serving replicas in dir f:\tmp\kafka-logs
> (kafka.server.ReplicaManager)
>
>
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i)
> delete the message and all copies; (ii) do not disclose, distribute or
> use the message in any manner; and (iii) notify the sender
> immediately. In addition, please be aware that any message addressed
> to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i)
> delete the message and all copies; (ii) do not disclose, distribute or
> use the message in any manner; and (iii) notify the sender
> immediately. In addition, please be aware that any message addressed
> to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
>
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Re: Issue deleting topic on Windows
Posted by Lawrence Craft <la...@gmail.com>.
Hi Ben,
This sounds very similar to KAFKA-1194, an issue we encountered on our
Windows servers at work. Basically, Kafka is unable to rename log files as
they're locked by the OS. The bug is here and is still open:
https://issues.apache.org/jira/browse/KAFKA-1194. I am not aware of any
straightforward workaround.
To get around this, we've had to modify the code and run a custom build.
You can try the changes from here: https://github.com/apache/kafka/pull/4431
Hope this helps!
Lawrence
On Thu, Mar 22, 2018 at 9:34 AM, Young, Ben <Be...@fisglobal.com> wrote:
> Hi,
>
> Any thoughts on the below?
>
> Thanks!
>
> -----Original Message-----
>
> Sorry, I forgot to mention we're using version: kafka_2.12-1.0.0
>
> -----Original Message-----
>
> Hi,
>
> We have an issue in our dev environments when running on Windows. To
> provide a clean environment for testing we tear down the topics and
> recreate them, but deleting the topics is causing the server to crash. We
> then need to delete the logs directory for it to be able to restart, it
> doesn't seem to be able to recover.
>
> The backtrace is below. I wonder if something still has a handle open on
> the folder before the rename is attempted?
>
> Thanks,
> Ben
>
> [2018-03-20 09:31:24,969] INFO [ReplicaFetcherManager on broker 0] Removed
> fetcher for partitions TradeFeed-3 (kafka.server.ReplicaFetcherManager)
> [2018-03-20 09:31:25,012] ERROR Error while deleting TradeFeed-3 in dir
> f:\tmp\kafka-logs. (kafka.server.LogDirFailureChannel)
> java.io.IOException: Failed to rename log directory from
> f:\tmp\kafka-logs\TradeFeed-3 to f:\tmp\kafka-logs\TradeFeed-3.
> 23a377d8b9b042a9853adc5deec53a2f-delete
> at kafka.log.LogManager.asyncDelete(LogManager.scala:671)
> at kafka.cluster.Partition.$anonfun$delete$1(Partition.scala:178)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:225)
> at kafka.cluster.Partition.delete(Partition.scala:173)
> at kafka.server.ReplicaManager.stopReplica(ReplicaManager.
> scala:341)
> at kafka.server.ReplicaManager.$anonfun$stopReplicas$2(
> ReplicaManager.scala:373)
> at scala.collection.Iterator.foreach(Iterator.scala:929)
> at scala.collection.Iterator.foreach$(Iterator.scala:929)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
> at scala.collection.IterableLike.foreach(IterableLike.scala:71)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.
> scala:371)
> at kafka.server.KafkaApis.handleStopReplicaRequest(
> KafkaApis.scala:190)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:104)
> at kafka.server.KafkaRequestHandler.run(
> KafkaRequestHandler.scala:65)
> at java.lang.Thread.run(Unknown Source)
> [2018-03-20 09:31:25,013] INFO [ReplicaManager broker=0] Stopping serving
> replicas in dir f:\tmp\kafka-logs (kafka.server.ReplicaManager)
>
>
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>
RE: Issue deleting topic on Windows
Posted by "Young, Ben" <Be...@fisglobal.com>.
Hi,
Any thoughts on the below?
Thanks!
-----Original Message-----
Sorry, I forgot to mention we're using version: kafka_2.12-1.0.0
-----Original Message-----
Hi,
We have an issue in our dev environments when running on Windows. To provide a clean environment for testing we tear down the topics and recreate them, but deleting the topics is causing the server to crash. We then need to delete the logs directory for it to be able to restart, it doesn't seem to be able to recover.
The backtrace is below. I wonder if something still has a handle open on the folder before the rename is attempted?
Thanks,
Ben
[2018-03-20 09:31:24,969] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions TradeFeed-3 (kafka.server.ReplicaFetcherManager)
[2018-03-20 09:31:25,012] ERROR Error while deleting TradeFeed-3 in dir f:\tmp\kafka-logs. (kafka.server.LogDirFailureChannel)
java.io.IOException: Failed to rename log directory from f:\tmp\kafka-logs\TradeFeed-3 to f:\tmp\kafka-logs\TradeFeed-3.23a377d8b9b042a9853adc5deec53a2f-delete
at kafka.log.LogManager.asyncDelete(LogManager.scala:671)
at kafka.cluster.Partition.$anonfun$delete$1(Partition.scala:178)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:225)
at kafka.cluster.Partition.delete(Partition.scala:173)
at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:341)
at kafka.server.ReplicaManager.$anonfun$stopReplicas$2(ReplicaManager.scala:373)
at scala.collection.Iterator.foreach(Iterator.scala:929)
at scala.collection.Iterator.foreach$(Iterator.scala:929)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
at scala.collection.IterableLike.foreach(IterableLike.scala:71)
at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:371)
at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:190)
at kafka.server.KafkaApis.handle(KafkaApis.scala:104)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:65)
at java.lang.Thread.run(Unknown Source)
[2018-03-20 09:31:25,013] INFO [ReplicaManager broker=0] Stopping serving replicas in dir f:\tmp\kafka-logs (kafka.server.ReplicaManager)
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
RE: Issue deleting topic on Windows
Posted by "Young, Ben" <Be...@fisglobal.com>.
Sorry, I forgot to mention we're using version: kafka_2.12-1.0.0
-----Original Message-----
From: Young, Ben [mailto:Ben.Young@fisglobal.com]
Sent: 20 March 2018 09:37
To: users@kafka.apache.org
Subject: Issue deleting topic on Windows
Hi,
We have an issue in our dev environments when running on Windows. To provide a clean environment for testing we tear down the topics and recreate them, but deleting the topics is causing the server to crash. We then need to delete the logs directory for it to be able to restart, it doesn't seem to be able to recover.
The backtrace is below. I wonder if something still has a handle open on the folder before the rename is attempted?
Thanks,
Ben
[2018-03-20 09:31:24,969] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions TradeFeed-3 (kafka.server.ReplicaFetcherManager)
[2018-03-20 09:31:25,012] ERROR Error while deleting TradeFeed-3 in dir f:\tmp\kafka-logs. (kafka.server.LogDirFailureChannel)
java.io.IOException: Failed to rename log directory from f:\tmp\kafka-logs\TradeFeed-3 to f:\tmp\kafka-logs\TradeFeed-3.23a377d8b9b042a9853adc5deec53a2f-delete
at kafka.log.LogManager.asyncDelete(LogManager.scala:671)
at kafka.cluster.Partition.$anonfun$delete$1(Partition.scala:178)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:225)
at kafka.cluster.Partition.delete(Partition.scala:173)
at kafka.server.ReplicaManager.stopReplica(ReplicaManager.scala:341)
at kafka.server.ReplicaManager.$anonfun$stopReplicas$2(ReplicaManager.scala:373)
at scala.collection.Iterator.foreach(Iterator.scala:929)
at scala.collection.Iterator.foreach$(Iterator.scala:929)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
at scala.collection.IterableLike.foreach(IterableLike.scala:71)
at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.server.ReplicaManager.stopReplicas(ReplicaManager.scala:371)
at kafka.server.KafkaApis.handleStopReplicaRequest(KafkaApis.scala:190)
at kafka.server.KafkaApis.handle(KafkaApis.scala:104)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:65)
at java.lang.Thread.run(Unknown Source)
[2018-03-20 09:31:25,013] INFO [ReplicaManager broker=0] Stopping serving replicas in dir f:\tmp\kafka-logs (kafka.server.ReplicaManager)
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.