You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Michael Bingham (Jira)" <ji...@apache.org> on 2020/10/01 18:48:00 UTC
[jira] [Created] (KAFKA-10564) Continuous logging about deleting
obsolete state directories
Michael Bingham created KAFKA-10564:
---------------------------------------
Summary: Continuous logging about deleting obsolete state directories
Key: KAFKA-10564
URL: https://issues.apache.org/jira/browse/KAFKA-10564
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.6.0
Reporter: Michael Bingham
The internal process which automatically cleans up obsolete task state directories was modified in https://issues.apache.org/jira/browse/KAFKA-6647. The current logic in 2.6 is to remove all files from the task directory except the {{.lock}} file:
[https://github.com/apache/kafka/blob/2.6/streams/src/main/java/org/apache/kafka/streams/processor/internals/StateDirectory.java#L335]
However, the directory is only removed in its entirely for a manual cleanup:
[https://github.com/apache/kafka/blob/2.6/streams/src/main/java/org/apache/kafka/streams/processor/internals/StateDirectory.java#L349-L353]
The result of this is that Streams will continue revisiting this directory and trying to clean it up, since it determines what to clean based on last-modification time of the task directory (which is now no longer deleted during the automatic cleanup). So a user will see log messages like:
stream-thread [app-c2d773e6-8ac3-4435-9777-378e0ec0ab82-CleanupThread] Deleting obsolete state directory 0_8 for task 0_8 as 60600001ms has elapsed (cleanup delay is 600000ms)
repeated again and again.
This issue doesn't seem to break anything - it's more about avoiding unnecessary logging and cleaning up empty task directories.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)