[jira] [Comment Edited] (KAFKA-6647) KafkaStreams.cleanUp creates .lock file in directory its trying to clean (Windows OS)

["Rob Leland (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17491083#comment-17491083 ] 

Rob Leland edited comment on KAFKA-6647 at 2/11/22, 6:33 PM:
-------------------------------------------------------------

4 Unit tests are silently swallowing exception attributed to this issue. Search for KAFKA-6647 in source code. If this ticket is truly fixed that should not be needed.

Also in future if there is an OS specific issue you need to code around temporarily please consider taking two steps:
1) Find method to only exclude for Particular OS not all operating system.
2) Agree on mechanism to clean this type of technical debt up since it wod be useful for detecting regressions.

-Rob


 

 


was (Author: rleland):
4 Unit tests are silently swallowing exception attributed to this issue. Search for KAFKA-6647 in source code. If this ticket is truly fixed that should not be needed.

> KafkaStreams.cleanUp creates .lock file in directory its trying to clean (Windows OS)
> -------------------------------------------------------------------------------------
>
>                 Key: KAFKA-6647
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6647
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 1.0.1
>         Environment: windows 10.
> java version "1.8.0_162"
> Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)
> org.apache.kafka:kafka-streams:1.0.1
> Kafka commitId : c0518aa65f25317e
>            Reporter: George Bloggs
>            Assignee: Guozhang Wang
>            Priority: Minor
>             Fix For: 2.6.0, 2.5.1
>
>
> When calling kafkaStreams.cleanUp() before starting a stream the StateDirectory.cleanRemovedTasks() method contains this check:
> {code:java}
> ... Line 240
>                   if (lock(id, 0)) {
>                         long now = time.milliseconds();
>                         long lastModifiedMs = taskDir.lastModified();
>                         if (now > lastModifiedMs + cleanupDelayMs) {
>                             log.info("{} Deleting obsolete state directory {} for task {} as {}ms has elapsed (cleanup delay is {}ms)", logPrefix(), dirName, id, now - lastModifiedMs, cleanupDelayMs);
>                             Utils.delete(taskDir);
>                         }
>                     }
> {code}
> The check for lock(id,0) will create a .lock file in the directory that subsequently is going to be deleted. If the .lock file already exists from a previous run the attempt to delete the .lock file fails with AccessDeniedException.
> This leaves the .lock file in the taskDir. Calling Utils.delete(taskDir) will then attempt to remove the taskDir path calling Files.delete(path).
> The call to files.delete(path) in postVisitDirectory will then fail java.nio.file.DirectoryNotEmptyException as the failed attempt to delete the .lock file left the directory not empty. (o.a.k.s.p.internals.StateDirectory       : stream-thread [restartedMain] Failed to lock the state directory due to an unexpected exception)
> This seems to then cause issues using streams from a topic to an inMemory store.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)