[jira] [Created] (KAFKA-20820) Broker fails to start when log.dirs is a symbolic link

["Gergely Harmadás (Jira)" <ji...@apache.org>]

Gergely Harmadás created KAFKA-20820:
----------------------------------------

             Summary: Broker fails to start when log.dirs is a symbolic link
                 Key: KAFKA-20820
                 URL: https://issues.apache.org/jira/browse/KAFKA-20820
             Project: Kafka
          Issue Type: Bug
            Reporter: Gergely Harmadás
            Assignee: Gergely Harmadás


The issue already came up earlier on the mailing list [https://lists.apache.org/thread/ro8mbk70xstywz5yw7mlm4yoswcfqlp5]

The bug can be reproduced by pointing log.dirs to a symlinked path, e.g.:

_/kafka/1/dd -> /navencrypt/1/kafka/kafka/1/dd_

The storage format step attempts to write the required meta.properties file to the configured log directory and fails with:
{code:java}
Error while writing meta.properties file /kafka/3/dd:
java.nio.file.FileAlreadyExistsException: /kafka/3/dd{code}
The broker later exits with:
{noformat}
java.lang.RuntimeException: No readable meta.properties files found.{noformat}
Based on my research the root cause is  a bug in JDK's Files.createDirectories which was addressed in [https://bugs.openjdk.org/browse/JDK-8294193] and shipped in JDK 20.

Specific code location in Kafka [https://github.com/apache/kafka/blob/8fc539cb383c34e44bdb31721e0643faf453b8e6/metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java#L452]

To conclude, Kafka 4.x is committed to use JDK17 and there is no real restriction on why symlinked paths could not be used for log dirs. Also this bug prevents to use certain tools like NavEncrypt which could handle encryption at rest.

I would propose a simple fix to also make symlinked log.dirs possible when on JDK17.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)