Log directory offline on AWS EBS

[Audrius Petrosius <a....@gmail.com>]

Hello,

We are encountering such issues on AWS EBS based system, nothing in AWS
logs.

Is it memory or IO issue, as it states in one line

in dir /srv/kafka/disk1 due to IOException
(kafka.server.LogDirFailureChannel)
java.io.IOException: Map failed

and in second

Caused by: java.lang.OutOfMemoryError: Map failed


> [2022-02-07 21:35:43,046] WARN [ReplicaManager broker=77] Unable to
> describe replica dirs for /srv/kafka/disk1 (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.KafkaStorageException: Log directory
> /srv/kafka/disk1 is offline


[2022-02-07 21:34:54,355] ERROR Failed to clean up log for
> invent_points_of_interest-39 in dir /srv/kafka/disk1 due to IOException
> (kafka.server.LogDirFailureChannel) java.io.IOException: Map failed at
> sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938) at kafka.log.
> AbstractIndex.<init>(AbstractIndex.scala:124) at kafka.log.OffsetIndex.<
> init>(OffsetIndex.scala:54) at kafka.log.LazyIndex$.$anonfun$forOffset$1(
> LazyIndex.scala:106) at kafka.log.LazyIndex.$anonfun$get$1(LazyIndex
> .scala:63) at kafka.log.LazyIndex.get(LazyIndex.scala:60) at kafka.log.
> LogSegment.offsetIndex(LogSegment.scala:64) at kafka.log.Cleaner
> .buildOffsetMapForSegment(LogCleaner.scala:932) at kafka.log.Cleaner.
> $anonfun$buildOffsetMap$5(LogCleaner.scala:908) at kafka.log.Cleaner.
> $anonfun$buildOffsetMap$5$adapted(LogCleaner.scala:904) at
> scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563) at
> scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:919) at
> scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889) at
> kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:904) at kafka.log.
> Cleaner.doClean(LogCleaner.scala:523) at kafka.log.Cleaner.clean(
> LogCleaner.scala:511) at kafka.log.LogCleaner$CleanerThread.cleanLog(
> LogCleaner.scala:380) at kafka.log.LogCleaner$CleanerThread
> .cleanFilthiestLog(LogCleaner.scala:352) at kafka.log.LogCleaner
> $CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:332) at kafka.log.
> LogCleaner$CleanerThread.doWork(LogCleaner.scala:321) at kafka.utils.
> ShutdownableThread.run(ShutdownableThread.scala:96) Caused by: java.lang.
> OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native
> Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935) ... 21
> more

[2022-02-07 21:34:54,356] WARN [ReplicaManager broker=77] Stopping serving
replicas in dir /srv/kafka/disk1 (kafka.server.ReplicaManager) -02-07 21:34:
54,588] WARN [ReplicaManager broker=77] While recording the replica LEO,
the partition multibooks-250 hasn't been created. (kafka.server.
ReplicaManager) [2022-02-07 21:34:54,696] WARN Stopping serving logs in dir
/srv/kafka/disk1 (kafka.log.LogManager)



>

Re: Log directory offline on AWS EBS

[Oleksandr Shulgin <ol...@zalando.de>]

On Thu, Feb 10, 2022 at 8:45 AM Audrius Petrosius <a....@gmail.com>
wrote:

> Hello,
>
> We are encountering such issues on AWS EBS based system, nothing in AWS
> logs.
>
> Is it memory or IO issue, as it states in one line
>
> in dir /srv/kafka/disk1 due to IOException
> (kafka.server.LogDirFailureChannel)
> java.io.IOException: Map failed

and in second
>
> Caused by: java.lang.OutOfMemoryError: Map failed
>

From the error it looks like a case of a missing tuning for the
`vm.max_map_count` kernel parameter:
https://kafka.apache.org/documentation/#os


Regards,
--
Alex