You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Neeraj Gulia <ne...@opsworld.in> on 2021/05/27 10:57:43 UTC
Kafka getting down every week due to log file deletion.
Hi team,
Our Kafka is getting down almost once or twice a month due to log file
deletion failure.
There is single node kafka broker is running in our system and gets down
every time it tires to delete the log files as cleanup and fails.
Sharing the Error Logs, we need a robust solution for this so that our
kafka broker doesn't gets down like this every time.
Regards,
Neeraj Gulia
Caused by: java.io.FileNotFoundException:
/tmp/kafka-logs/dokutopic-0/00000000000000000000.index (No such file or
directory)
at java.base/java.io.RandomAccessFile.open0(Native Method)
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:183)
at kafka.log.AbstractIndex.resize(AbstractIndex.scala:176)
at
kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:242)
at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:242)
at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:508)
at kafka.log.Log.$anonfun$roll$8(Log.scala:1954)
at kafka.log.Log.$anonfun$roll$2(Log.scala:1954)
at kafka.log.Log.roll(Log.scala:2387)
at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1749)
at kafka.log.Log.deleteSegments(Log.scala:2387)
at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1737)
at kafka.log.Log.deleteOldSegments(Log.scala:1806)
at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:1074)
at
kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:1071)
at scala.collection.immutable.List.foreach(List.scala:431)
at kafka.log.LogManager.cleanupLogs(LogManager.scala:1071)
at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:409)
at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[2021-05-27 09:34:07,972] WARN [ReplicaManager broker=0] Broker 0 stopped
fetcher for partitions
__consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,fliptopic-0,__consumer_offsets-25,webhook-events-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,dokutopic-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,post_payment_topic-0,__consumer_offsets-18,__consumer_offsets-37,topic-0,events-0,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,disbursementtopic-0,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,faspaytopic-0
and stopped moving logs for partitions because they are in the failed log
directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
[2021-05-27 09:34:07,974] WARN Stopping serving logs in dir /tmp/kafka-logs
(kafka.log.LogManager)
[2021-05-27 09:34:07,983] ERROR Shutdown broker because all log dirs in
/tmp/kafka-logs have failed (kafka.log.LogManager)
Re: Kafka getting down every week due to log file deletion.
Posted by Ran Lupovich <ra...@gmail.com>.
The main purpose of the /*tmp* directory is to temporarily store *files* when
installing an OS or software. If any *files* in the /*tmp* directory have
not been accessed for a while, they will be automatically *deleted* from
the system
בתאריך יום ה׳, 27 במאי 2021, 19:04, מאת Ran Lupovich <ranlupovich@gmail.com
>:
> Seems you log dir is sending your data to tmp folder, if I am bot mistken
> this dir automatically removing files from itself, causing the log deletuon
> procedure of the kafka internal to fail and shutdown broker on file not
> found
>
> בתאריך יום ה׳, 27 במאי 2021, 17:52, מאת Neeraj Gulia <
> neeraj.gulia@opsworld.in>:
>
>> Hi team,
>>
>> Our Kafka is getting down almost once or twice a month due to log file
>> deletion failure.
>>
>>
>> There is single node kafka broker is running in our system and gets down
>> every time it tires to delete the log files as cleanup and fails.
>>
>> Sharing the Error Logs, we need a robust solution for this so that our
>> kafka broker doesn't gets down like this every time.
>>
>> Regards,
>> Neeraj Gulia
>>
>> Caused by: java.io.FileNotFoundException:
>> /tmp/kafka-logs/dokutopic-0/00000000000000000000.index (No such file or
>> directory)
>> at java.base/java.io.RandomAccessFile.open0(Native Method)
>> at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
>> at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
>> at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
>> at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:183)
>> at kafka.log.AbstractIndex.resize(AbstractIndex.scala:176)
>> at
>>
>> kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:242)
>> at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:242)
>> at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:508)
>> at kafka.log.Log.$anonfun$roll$8(Log.scala:1954)
>> at kafka.log.Log.$anonfun$roll$2(Log.scala:1954)
>> at kafka.log.Log.roll(Log.scala:2387)
>> at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1749)
>> at kafka.log.Log.deleteSegments(Log.scala:2387)
>> at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1737)
>> at kafka.log.Log.deleteOldSegments(Log.scala:1806)
>> at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:1074)
>> at
>> kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:1071)
>> at scala.collection.immutable.List.foreach(List.scala:431)
>> at kafka.log.LogManager.cleanupLogs(LogManager.scala:1071)
>> at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:409)
>> at
>> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
>> at
>>
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> at
>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> at
>>
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> at
>>
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>>
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> [2021-05-27 09:34:07,972] WARN [ReplicaManager broker=0] Broker 0 stopped
>> fetcher for partitions
>>
>> __consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,fliptopic-0,__consumer_offsets-25,webhook-events-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,dokutopic-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,post_payment_topic-0,__consumer_offsets-18,__consumer_offsets-37,topic-0,events-0,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,disbursementtopic-0,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,faspaytopic-0
>> and stopped moving logs for partitions because they are in the failed log
>> directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
>> [2021-05-27 09:34:07,974] WARN Stopping serving logs in dir
>> /tmp/kafka-logs
>> (kafka.log.LogManager)
>> [2021-05-27 09:34:07,983] ERROR Shutdown broker because all log dirs in
>> /tmp/kafka-logs have failed (kafka.log.LogManager)
>>
>
Re: Kafka getting down every week due to log file deletion.
Posted by Ran Lupovich <ra...@gmail.com>.
The main purpose of the /*tmp* directory is to temporarily store *files* when
installing an OS or software. If any *files* in the /*tmp* directory have
not been accessed for a while, they will be automatically *deleted* from
the system
בתאריך יום ה׳, 27 במאי 2021, 19:04, מאת Ran Lupovich <ranlupovich@gmail.com
>:
> Seems you log dir is sending your data to tmp folder, if I am bot mistken
> this dir automatically removing files from itself, causing the log deletuon
> procedure of the kafka internal to fail and shutdown broker on file not
> found
>
> בתאריך יום ה׳, 27 במאי 2021, 17:52, מאת Neeraj Gulia <
> neeraj.gulia@opsworld.in>:
>
>> Hi team,
>>
>> Our Kafka is getting down almost once or twice a month due to log file
>> deletion failure.
>>
>>
>> There is single node kafka broker is running in our system and gets down
>> every time it tires to delete the log files as cleanup and fails.
>>
>> Sharing the Error Logs, we need a robust solution for this so that our
>> kafka broker doesn't gets down like this every time.
>>
>> Regards,
>> Neeraj Gulia
>>
>> Caused by: java.io.FileNotFoundException:
>> /tmp/kafka-logs/dokutopic-0/00000000000000000000.index (No such file or
>> directory)
>> at java.base/java.io.RandomAccessFile.open0(Native Method)
>> at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
>> at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
>> at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
>> at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:183)
>> at kafka.log.AbstractIndex.resize(AbstractIndex.scala:176)
>> at
>>
>> kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:242)
>> at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:242)
>> at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:508)
>> at kafka.log.Log.$anonfun$roll$8(Log.scala:1954)
>> at kafka.log.Log.$anonfun$roll$2(Log.scala:1954)
>> at kafka.log.Log.roll(Log.scala:2387)
>> at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1749)
>> at kafka.log.Log.deleteSegments(Log.scala:2387)
>> at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1737)
>> at kafka.log.Log.deleteOldSegments(Log.scala:1806)
>> at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:1074)
>> at
>> kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:1071)
>> at scala.collection.immutable.List.foreach(List.scala:431)
>> at kafka.log.LogManager.cleanupLogs(LogManager.scala:1071)
>> at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:409)
>> at
>> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
>> at
>>
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> at
>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> at
>>
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> at
>>
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>>
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> [2021-05-27 09:34:07,972] WARN [ReplicaManager broker=0] Broker 0 stopped
>> fetcher for partitions
>>
>> __consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,fliptopic-0,__consumer_offsets-25,webhook-events-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,dokutopic-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,post_payment_topic-0,__consumer_offsets-18,__consumer_offsets-37,topic-0,events-0,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,disbursementtopic-0,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,faspaytopic-0
>> and stopped moving logs for partitions because they are in the failed log
>> directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
>> [2021-05-27 09:34:07,974] WARN Stopping serving logs in dir
>> /tmp/kafka-logs
>> (kafka.log.LogManager)
>> [2021-05-27 09:34:07,983] ERROR Shutdown broker because all log dirs in
>> /tmp/kafka-logs have failed (kafka.log.LogManager)
>>
>
Re: Kafka getting down every week due to log file deletion.
Posted by Ran Lupovich <ra...@gmail.com>.
Seems you log dir is sending your data to tmp folder, if I am bot mistken
this dir automatically removing files from itself, causing the log deletuon
procedure of the kafka internal to fail and shutdown broker on file not
found
בתאריך יום ה׳, 27 במאי 2021, 17:52, מאת Neeraj Gulia <
neeraj.gulia@opsworld.in>:
> Hi team,
>
> Our Kafka is getting down almost once or twice a month due to log file
> deletion failure.
>
>
> There is single node kafka broker is running in our system and gets down
> every time it tires to delete the log files as cleanup and fails.
>
> Sharing the Error Logs, we need a robust solution for this so that our
> kafka broker doesn't gets down like this every time.
>
> Regards,
> Neeraj Gulia
>
> Caused by: java.io.FileNotFoundException:
> /tmp/kafka-logs/dokutopic-0/00000000000000000000.index (No such file or
> directory)
> at java.base/java.io.RandomAccessFile.open0(Native Method)
> at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
> at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
> at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
> at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:183)
> at kafka.log.AbstractIndex.resize(AbstractIndex.scala:176)
> at
> kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:242)
> at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:242)
> at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:508)
> at kafka.log.Log.$anonfun$roll$8(Log.scala:1954)
> at kafka.log.Log.$anonfun$roll$2(Log.scala:1954)
> at kafka.log.Log.roll(Log.scala:2387)
> at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1749)
> at kafka.log.Log.deleteSegments(Log.scala:2387)
> at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1737)
> at kafka.log.Log.deleteOldSegments(Log.scala:1806)
> at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:1074)
> at
> kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:1071)
> at scala.collection.immutable.List.foreach(List.scala:431)
> at kafka.log.LogManager.cleanupLogs(LogManager.scala:1071)
> at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:409)
> at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
>
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> [2021-05-27 09:34:07,972] WARN [ReplicaManager broker=0] Broker 0 stopped
> fetcher for partitions
>
> __consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,fliptopic-0,__consumer_offsets-25,webhook-events-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,dokutopic-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,post_payment_topic-0,__consumer_offsets-18,__consumer_offsets-37,topic-0,events-0,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,disbursementtopic-0,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,faspaytopic-0
> and stopped moving logs for partitions because they are in the failed log
> directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
> [2021-05-27 09:34:07,974] WARN Stopping serving logs in dir /tmp/kafka-logs
> (kafka.log.LogManager)
> [2021-05-27 09:34:07,983] ERROR Shutdown broker because all log dirs in
> /tmp/kafka-logs have failed (kafka.log.LogManager)
>
Re: Kafka getting down every week due to log file deletion.
Posted by Ran Lupovich <ra...@gmail.com>.
Seems you log dir is sending your data to tmp folder, if I am bot mistken
this dir automatically removing files from itself, causing the log deletuon
procedure of the kafka internal to fail and shutdown broker on file not
found
בתאריך יום ה׳, 27 במאי 2021, 17:52, מאת Neeraj Gulia <
neeraj.gulia@opsworld.in>:
> Hi team,
>
> Our Kafka is getting down almost once or twice a month due to log file
> deletion failure.
>
>
> There is single node kafka broker is running in our system and gets down
> every time it tires to delete the log files as cleanup and fails.
>
> Sharing the Error Logs, we need a robust solution for this so that our
> kafka broker doesn't gets down like this every time.
>
> Regards,
> Neeraj Gulia
>
> Caused by: java.io.FileNotFoundException:
> /tmp/kafka-logs/dokutopic-0/00000000000000000000.index (No such file or
> directory)
> at java.base/java.io.RandomAccessFile.open0(Native Method)
> at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
> at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
> at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
> at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:183)
> at kafka.log.AbstractIndex.resize(AbstractIndex.scala:176)
> at
> kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:242)
> at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:242)
> at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:508)
> at kafka.log.Log.$anonfun$roll$8(Log.scala:1954)
> at kafka.log.Log.$anonfun$roll$2(Log.scala:1954)
> at kafka.log.Log.roll(Log.scala:2387)
> at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1749)
> at kafka.log.Log.deleteSegments(Log.scala:2387)
> at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1737)
> at kafka.log.Log.deleteOldSegments(Log.scala:1806)
> at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:1074)
> at
> kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:1071)
> at scala.collection.immutable.List.foreach(List.scala:431)
> at kafka.log.LogManager.cleanupLogs(LogManager.scala:1071)
> at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:409)
> at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
>
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> [2021-05-27 09:34:07,972] WARN [ReplicaManager broker=0] Broker 0 stopped
> fetcher for partitions
>
> __consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,fliptopic-0,__consumer_offsets-25,webhook-events-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,dokutopic-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,post_payment_topic-0,__consumer_offsets-18,__consumer_offsets-37,topic-0,events-0,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,disbursementtopic-0,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,faspaytopic-0
> and stopped moving logs for partitions because they are in the failed log
> directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
> [2021-05-27 09:34:07,974] WARN Stopping serving logs in dir /tmp/kafka-logs
> (kafka.log.LogManager)
> [2021-05-27 09:34:07,983] ERROR Shutdown broker because all log dirs in
> /tmp/kafka-logs have failed (kafka.log.LogManager)
>