You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Sander van Loo (JIRA)" <ji...@apache.org> on 2018/11/16 14:14:00 UTC
[jira] [Created] (KAFKA-7637) ERROR Error while writing to
checkpoint file due to too many open files
Sander van Loo created KAFKA-7637:
-------------------------------------
Summary: ERROR Error while writing to checkpoint file due to too many open files
Key: KAFKA-7637
URL: https://issues.apache.org/jira/browse/KAFKA-7637
Project: Kafka
Issue Type: Bug
Affects Versions: 1.1.1
Environment: Red Hat Enterprise Linux Server release 7.4 (Maipo)
Reporter: Sander van Loo
We are running a 3 node Kafka cluster on version 1.1.1 on Red Hat Linux 7.
Max open files is set to 65000.
After running for a few days the nodes have the following open file counts:
* node01d: 2712
* node01e: 2770
* node01f: 4102
After a few weeks of runtime cluster crashes with the following error:
{noformat}
[2018-11-12 07:05:16,790] ERROR Error while writing to checkpoint file /var/lib/kafka/topics/replication-offset-checkpoint (kafka.server.LogDirFailureChannel)
java.io.FileNotFoundException: /var/lib/kafka/topics/replication-offset-checkpoint.tmp (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at kafka.server.checkpoints.CheckpointFile.liftedTree1$1(CheckpointFile.scala:52)
at kafka.server.checkpoints.CheckpointFile.write(CheckpointFile.scala:50)
at kafka.server.checkpoints.OffsetCheckpointFile.write(OffsetCheckpointFile.scala:59)
at kafka.server.ReplicaManager.$anonfun$checkpointHighWatermarks$9(ReplicaManager.scala:1384)
at kafka.server.ReplicaManager.$anonfun$checkpointHighWatermarks$9$adapted(ReplicaManager.scala:1384)
at scala.Option.foreach(Option.scala:257)
at kafka.server.ReplicaManager.$anonfun$checkpointHighWatermarks$7(ReplicaManager.scala:1384)
at kafka.server.ReplicaManager.$anonfun$checkpointHighWatermarks$7$adapted(ReplicaManager.scala:1381)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:120)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
at kafka.server.ReplicaManager.checkpointHighWatermarks(ReplicaManager.scala:1381)
at kafka.server.ReplicaManager.$anonfun$startHighWaterMarksCheckPointThread$1(ReplicaManager.scala:242)
at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
followed by this one:
{noformat}
[2018-11-12 07:05:16,792] ERROR [ReplicaManager broker=3] Error while writing to highwatermark file in directory /var/lib/kafka/topics (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.KafkaStorageException: Error while writing to checkpoint file /var/lib/kafka/topics/replication-offset-checkpoint
Caused by: java.io.FileNotFoundException: /var/lib/kafka/topics/replication-offset-checkpoint.tmp (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at kafka.server.checkpoints.CheckpointFile.liftedTree1$1(CheckpointFile.scala:52)
at kafka.server.checkpoints.CheckpointFile.write(CheckpointFile.scala:50)
at kafka.server.checkpoints.OffsetCheckpointFile.write(OffsetCheckpointFile.scala:59)
at kafka.server.ReplicaManager.$anonfun$checkpointHighWatermarks$9(ReplicaManager.scala:1384)
at kafka.server.ReplicaManager.$anonfun$checkpointHighWatermarks$9$adapted(ReplicaManager.scala:1384)
at scala.Option.foreach(Option.scala:257)
at kafka.server.ReplicaManager.$anonfun$checkpointHighWatermarks$7(ReplicaManager.scala:1384)
at kafka.server.ReplicaManager.$anonfun$checkpointHighWatermarks$7$adapted(ReplicaManager.scala:1381)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:120)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
at kafka.server.ReplicaManager.checkpointHighWatermarks(ReplicaManager.scala:1381)
at kafka.server.ReplicaManager.$anonfun$startHighWaterMarksCheckPointThread$1(ReplicaManager.scala:242)
at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
and the cluster never recovers from this.
The number of open files appears to creap up slowly as time progresses and during normal operations we see many errors like below:
{noformat}
[2018-11-16 00:46:35,082] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error for partition check_node01d.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 00:49:34,947] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error for partition check_node01d.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 01:11:01,754] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Error for partition check_node01f.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 01:28:34,982] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error for partition check_node01d.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 01:38:48,423] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Error for partition check_node01e.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 01:40:35,052] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Error for partition check_node01d.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 02:11:01,734] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Error for partition check_node01f.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 04:01:35,036] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error for partition check_node01d.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 04:19:35,013] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error for partition check_node01d.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 04:59:48,750] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Error for partition check_node01e.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 05:08:01,681] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error for partition check_node01f.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 05:56:01,536] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error for partition check_node01f.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 06:17:48,516] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error for partition check_node01e.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
[2018-11-16 06:20:01,709] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error for partition check_node01f.local-0 at offset 1 (kafka.server.ReplicaFetcherThread)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)