You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/02/02 13:01:51 UTC
[jira] [Commented] (KAFKA-4724) Clean up of state directories can
possibly remove stores that are about to be used by another thread
[ https://issues.apache.org/jira/browse/KAFKA-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849888#comment-15849888 ]
ASF GitHub Bot commented on KAFKA-4724:
---------------------------------------
GitHub user dguy opened a pull request:
https://github.com/apache/kafka/pull/2486
KAFKA-4724: Clean up of state directories can possibly remove stores that are about to be used by another thread
Delay the cleanup of state directories that are not locked and not owned by the current thread such that we only remove the directory if its last modified is < now - cleanupDelayMs.
This also helps to avoid a race between threads on the same instance, where during rebalance, one thread releases the lock on the state directory, and before another thread can take the lock, the cleanup runs and removes the data.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dguy/kafka KAFKA-4724
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2486.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2486
----
commit 229471706dec638361777f00de195e7538a54c29
Author: Damian Guy <da...@gmail.com>
Date: 2017-02-02T12:58:07Z
delay cleanup of state directories
----
> Clean up of state directories can possibly remove stores that are about to be used by another thread
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4724
> URL: https://issues.apache.org/jira/browse/KAFKA-4724
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.10.2.0
> Reporter: Damian Guy
> Assignee: Damian Guy
> Fix For: 0.10.3.0
>
>
> Currently, each {{StreamThread}} cleans up unused state directories based on the config {{StreamsConfig.STATE_CLEANUP_DELAY_MS_CONFIG}}.
> In situations where there are multiple threads configured this can lead to a race condition where the state directory gets removed by one thread when the task has been assigned (but not yet taken a lock) to another thread in the process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)