You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Yunyung (via GitHub)" <gi...@apache.org> on 2025/04/27 18:44:56 UTC

[PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Yunyung opened a new pull request, #19579:
URL: https://github.com/apache/kafka/pull/19579

   Follow up https://github.com/apache/kafka/pull/19460/files#r2062664349


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2063791648


##########
core/src/main/scala/kafka/server/BrokerServer.scala:
##########
@@ -709,7 +709,7 @@ class BrokerServer(
         None
       }
 
-      val rlm = new RemoteLogManager(config.remoteLogManagerConfig, config.brokerId, config.logDirs.head, clusterId, time,
+      val rlm = new RemoteLogManager(config.remoteLogManagerConfig, config.brokerId, config.logDirs.asScala.head, clusterId, time,

Review Comment:
   It looks like we can use `getFirst` to avoid redundant conversion.
   
   ```suggestion
         val rlm = new RemoteLogManager(config.remoteLogManagerConfig, config.brokerId, config.logDirs.getFirst, clusterId, time,
   ```



##########
core/src/test/scala/integration/kafka/server/KRaftClusterTest.scala:
##########
@@ -1551,7 +1551,7 @@ class KRaftClusterTest {
         // Copy foo-0 to targetParentDir
         // This is so that we can rename the main replica to a future down below
         val parentDir = log.parentDir
-        val targetParentDir = broker0.config.logDirs.filter(_ != parentDir).head
+        val targetParentDir = broker0.config.logDirs.asScala.filter(_ != parentDir).head
         val targetDirFile = new File(targetParentDir, log.dir.getName)

Review Comment:
   ```suggestion
           val targetParentDir = broker0.config.logDirs.stream().filter(l => !l.equals(parentDir)).findFirst()
           assertTrue(targetParentDir.isPresent)
           val targetDirFile = new File(targetParentDir.get(), log.dir.getName)
   ```



##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -241,7 +241,7 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
   def metadataLogDir: String = {
     Option(getString(MetadataLogConfig.METADATA_LOG_DIR_CONFIG)) match {
       case Some(dir) => dir
-      case None => logDirs.head
+      case None => logDirs.asScala.head

Review Comment:
   ditto



##########
core/src/test/scala/unit/kafka/server/AlterReplicaLogDirsRequestTest.scala:
##########
@@ -91,10 +91,10 @@ class AlterReplicaLogDirsRequestTest extends BaseRequestTest {
   @ParameterizedTest
   @ValueSource(strings = Array("kraft"))
   def testAlterReplicaLogDirsRequestErrorCode(quorum: String): Unit = {
-    val offlineDir = new File(brokers.head.config.logDirs.tail.head).getAbsolutePath
-    val validDir1 = new File(brokers.head.config.logDirs(1)).getAbsolutePath
-    val validDir2 = new File(brokers.head.config.logDirs(2)).getAbsolutePath
-    val validDir3 = new File(brokers.head.config.logDirs(3)).getAbsolutePath
+    val offlineDir = new File(brokers.head.config.logDirs.asScala.tail.head).getAbsolutePath

Review Comment:
   ```suggestion
       val offlineDir = new File(brokers.head.config.logDirs.getLast).getAbsolutePath
   ```



##########
core/src/test/scala/unit/kafka/server/DescribeLogDirsRequestTest.scala:
##########
@@ -42,8 +42,8 @@ class DescribeLogDirsRequestTest extends BaseRequestTest {
   @ParameterizedTest
   @ValueSource(strings = Array("kraft"))
   def testDescribeLogDirsRequest(quorum: String): Unit = {
-    val onlineDir = new File(brokers.head.config.logDirs.head).getAbsolutePath
-    val offlineDir = new File(brokers.head.config.logDirs.tail.head).getAbsolutePath
+    val onlineDir = new File(brokers.head.config.logDirs.asScala.head).getAbsolutePath
+    val offlineDir = new File(brokers.head.config.logDirs.asScala.tail.head).getAbsolutePath

Review Comment:
   ```suggestion
       val onlineDir = new File(brokers.head.config.logDirs.getFirst).getAbsolutePath
       val offlineDir = new File(brokers.head.config.logDirs.getLast).getAbsolutePath
   ```



##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -1237,7 +1237,7 @@ class ReplicaManager(val config: KafkaConfig,
   def describeLogDirs(partitions: Set[TopicPartition]): List[DescribeLogDirsResponseData.DescribeLogDirsResult] = {
     val logsByDir = logManager.allLogs.groupBy(log => log.parentDir)
 
-    config.logDirs.toSet.map { logDir: String =>
+    config.logDirs.asScala.toSet.map { logDir: String =>

Review Comment:
   Can we change `describeLogDirs` to return `java.util.List`? So we don't need to convert `logDirs` to Scala and convert result back to Java again in `KafkaApis`.



##########
core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:
##########
@@ -1500,7 +1500,7 @@ class KafkaConfigTest {
 
     val config = KafkaConfig.fromProps(props)
     assertEquals(metadataDir, config.metadataLogDir)
-    assertEquals(Seq(dataDir), config.logDirs)
+    assertEquals(Seq(dataDir), config.logDirs.asScala)

Review Comment:
   ```suggestion
       assertEquals(util.List.of(dataDir), config.logDirs)
   ```



##########
core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:
##########
@@ -1165,7 +1165,7 @@ class KafkaConfigTest {
     assertEquals(1, config.brokerId)
     assertEquals(Seq("PLAINTEXT://127.0.0.1:1122"), config.effectiveAdvertisedBrokerListeners.map(JTestUtils.endpointToString))
     assertEquals(Map("127.0.0.1" -> 2, "127.0.0.2" -> 3), config.maxConnectionsPerIpOverrides)
-    assertEquals(List("/tmp1", "/tmp2"), config.logDirs)
+    assertEquals(List("/tmp1", "/tmp2"), config.logDirs.asScala.toList)

Review Comment:
   ```suggestion
       assertEquals(util.List.of("/tmp1", "/tmp2"), config.logDirs)
   ```



##########
core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:
##########
@@ -1518,7 +1518,7 @@ class KafkaConfigTest {
 
     val config = KafkaConfig.fromProps(props)
     assertEquals(dataDir1, config.metadataLogDir)
-    assertEquals(Seq(dataDir1, dataDir2), config.logDirs)
+    assertEquals(Seq(dataDir1, dataDir2), config.logDirs.asScala)

Review Comment:
   ```suggestion
       assertEquals(util.List.of(dataDir1, dataDir2), config.logDirs)
   ```



##########
core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala:
##########
@@ -62,8 +64,8 @@ class LogRecoveryTest extends QuorumTestHarness {
 
   var admin: Admin = _
   var producer: KafkaProducer[Integer, String] = _
-  def hwFile1 = new OffsetCheckpointFile(new File(configProps1.logDirs.head, ReplicaManager.HighWatermarkFilename), null)
-  def hwFile2 = new OffsetCheckpointFile(new File(configProps2.logDirs.head, ReplicaManager.HighWatermarkFilename), null)
+  def hwFile1 = new OffsetCheckpointFile(new File(configProps1.logDirs.asScala.head, ReplicaManager.HighWatermarkFilename), null)
+  def hwFile2 = new OffsetCheckpointFile(new File(configProps2.logDirs.asScala.head, ReplicaManager.HighWatermarkFilename), null)

Review Comment:
   ```suggestion
     def hwFile1 = new OffsetCheckpointFile(new File(configProps1.logDirs.getFirst, ReplicaManager.HighWatermarkFilename), null)
     def hwFile2 = new OffsetCheckpointFile(new File(configProps2.logDirs.getFirst, ReplicaManager.HighWatermarkFilename), null)
   ```



##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/server/PartitionCreationBench.java:
##########
@@ -115,8 +115,7 @@ public void setup() {
         this.time = Time.SYSTEM;
         this.failureChannel = new LogDirFailureChannel(brokerProperties.logDirs().size());
         final BrokerTopicStats brokerTopicStats = new BrokerTopicStats(false);
-        final List<File> files =
-                CollectionConverters.asJava(brokerProperties.logDirs()).stream().map(File::new).collect(Collectors.toList());
+        final List<File> files = brokerProperties.logDirs().stream().map(File::new).collect(Collectors.toList());

Review Comment:
   ```suggestion
           final List<File> files = brokerProperties.logDirs().stream().map(File::new).toList();
   ```



##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/server/CheckpointBench.java:
##########
@@ -108,8 +108,7 @@ public void setup() {
         this.metrics = new Metrics();
         this.time = new MockTime();
         this.failureChannel = new LogDirFailureChannel(brokerProperties.logDirs().size());
-        final List<File> files =
-            CollectionConverters.asJava(brokerProperties.logDirs()).stream().map(File::new).collect(Collectors.toList());
+        final List<File> files = brokerProperties.logDirs().stream().map(File::new).collect(Collectors.toList());

Review Comment:
   ```suggestion
           final List<File> files = brokerProperties.logDirs().stream().map(File::new).toList();
   ```



##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -2771,7 +2771,7 @@ class ReplicaManagerTest {
     props.asScala ++= extraProps.asScala
     val config = KafkaConfig.fromProps(props)
     val logConfig = new LogConfig(new Properties)
-    val logDir = new File(new File(config.logDirs.head), s"$topic-$topicPartition")
+    val logDir = new File(new File(config.logDirs.asScala.head), s"$topic-$topicPartition")

Review Comment:
   ```suggestion
       val logDir = new File(new File(config.logDirs.getFirst), s"$topic-$topicPartition")
   ```



##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -214,7 +214,7 @@ class ReplicaManagerTest {
       partition.createLogIfNotExists(isNew = false, isFutureReplica = false,
         new LazyOffsetCheckpoints(rm.highWatermarkCheckpoints.asJava), None)
       rm.checkpointHighWatermarks()
-      config.logDirs.map(s => Paths.get(s, ReplicaManager.HighWatermarkFilename))
+      config.logDirs.asScala.map(s => Paths.get(s, ReplicaManager.HighWatermarkFilename))
         .foreach(checkpointFile => assertTrue(Files.exists(checkpointFile),

Review Comment:
   ```suggestion
         config.logDirs.stream().map(s => Paths.get(s, ReplicaManager.HighWatermarkFilename))
           .forEach(checkpointFile => assertTrue(Files.exists(checkpointFile),
   ```



##########
core/src/test/scala/unit/kafka/server/HighwatermarkPersistenceTest.scala:
##########
@@ -195,7 +197,7 @@ class HighwatermarkPersistenceTest {
   }
 
   private def hwmFor(replicaManager: ReplicaManager, topic: String, partition: Int): Long = {
-    replicaManager.highWatermarkCheckpoints(new File(replicaManager.config.logDirs.head).getAbsolutePath).read().getOrDefault(
+    replicaManager.highWatermarkCheckpoints(new File(replicaManager.config.logDirs.asScala.head).getAbsolutePath).read().getOrDefault(

Review Comment:
   ```suggestion
       replicaManager.highWatermarkCheckpoints(new File(replicaManager.config.logDirs.getFirst).getAbsolutePath).read().getOrDefault(
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "Yunyung (via GitHub)" <gi...@apache.org>.
Yunyung commented on PR #19579:
URL: https://github.com/apache/kafka/pull/19579#issuecomment-2869081857

   > @Yunyung please check the failed tests
   
   Done. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #19579:
URL: https://github.com/apache/kafka/pull/19579#issuecomment-2868974596

   @Yunyung please check the failed tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "Yunyung (via GitHub)" <gi...@apache.org>.
Yunyung commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2064009541


##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/ListOffsetsIntegrationTest.java:
##########
@@ -243,8 +241,7 @@ private void verifyListOffsets(String topic, int expectedMaxTimestampOffset) thr
         // case 2: test the offsets from recovery path.
         // server will rebuild offset index according to log files if the index files are nonexistent
         Set<String> indexFiles = clusterInstance.brokers().values().stream().flatMap(broker ->
-                CollectionConverters.asJava(broker.config().logDirs()).stream()
-        ).collect(Collectors.toUnmodifiableSet());
+            broker.config().logDirs().stream()).collect(Collectors.toUnmodifiableSet());

Review Comment:
   Done. Thanks.



##########
storage/src/test/java/org/apache/kafka/tiered/storage/integration/TransactionsWithTieredStoreTest.java:
##########
@@ -75,7 +76,7 @@ public Properties topicConfig() {
     public void maybeWaitForAtLeastOneSegmentUpload(scala.collection.Seq<TopicPartition> topicPartitions) {
         CollectionConverters.asJava(topicPartitions).forEach(topicPartition -> {
             List<BrokerLocalStorage> localStorages = CollectionConverters.asJava(brokers()).stream()
-                    .map(b -> new BrokerLocalStorage(b.config().brokerId(), CollectionConverters.asJava(b.config().logDirs().toSet()), STORAGE_WAIT_TIMEOUT_SEC))
+                    .map(b -> new BrokerLocalStorage(b.config().brokerId(), new HashSet<>(b.config().logDirs()), STORAGE_WAIT_TIMEOUT_SEC))

Review Comment:
   Done.



##########
server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java:
##########
@@ -66,8 +68,16 @@ public abstract class AbstractKafkaConfig extends AbstractConfig {
         AddPartitionsToTxnConfig.CONFIG_DEF
     ));
 
+    public final List<String> logDirs;
+
+    @SuppressWarnings("this-escape")
     public AbstractKafkaConfig(ConfigDef definition, Map<?, ?> originals, Map<String, ?> configProviderProps, boolean doLog) {
         super(definition, originals, configProviderProps, doLog);
+        this.logDirs = Csv.parseCsvList(Optional.ofNullable(getString(ServerLogConfigs.LOG_DIRS_CONFIG)).orElse(getString(ServerLogConfigs.LOG_DIR_CONFIG)));

Review Comment:
   Sounds fair. Simple, not hot path, low cost.



##########
storage/src/test/java/org/apache/kafka/tiered/storage/TieredStorageTestHarness.java:
##########
@@ -154,7 +155,7 @@ public static List<LocalTieredStorage> remoteStorageManagers(Seq<KafkaBroker> br
 
     public static List<BrokerLocalStorage> localStorages(Seq<KafkaBroker> brokers) {
         return CollectionConverters.asJava(brokers).stream()
-                .map(b -> new BrokerLocalStorage(b.config().brokerId(), CollectionConverters.asJava(b.config().logDirs().toSet()),
+                .map(b -> new BrokerLocalStorage(b.config().brokerId(), new HashSet<>(b.config().logDirs()),

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #19579:
URL: https://github.com/apache/kafka/pull/19579


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2062814664


##########
server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java:
##########
@@ -66,8 +68,16 @@ public abstract class AbstractKafkaConfig extends AbstractConfig {
         AddPartitionsToTxnConfig.CONFIG_DEF
     ));
 
+    public final List<String> logDirs;
+
+    @SuppressWarnings("this-escape")
     public AbstractKafkaConfig(ConfigDef definition, Map<?, ?> originals, Map<String, ?> configProviderProps, boolean doLog) {
         super(definition, originals, configProviderProps, doLog);
+        this.logDirs = Csv.parseCsvList(Optional.ofNullable(getString(ServerLogConfigs.LOG_DIRS_CONFIG)).orElse(getString(ServerLogConfigs.LOG_DIR_CONFIG)));

Review Comment:
   `KafkaConfig` overwrite the getter to reflect the newest config, so `getString` cause NPE as the sub class (`KafkaConfig`) is not initialized. 
   
   The logDirs() method isn't frequently used, so we can parse it only when it's needed to avoid initialization issues. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "TaiJuWu (via GitHub)" <gi...@apache.org>.
TaiJuWu commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2083357389


##########
checkstyle/import-control-clients-integration-tests.xml:
##########
@@ -21,7 +21,6 @@
 <import-control pkg="org.apache.kafka">
     <allow pkg="java"/>
     <allow pkg="org.junit"/>
-    <allow pkg="scala" />

Review Comment:
   Should we change to `<disallow pkg="scala">` ?
   In order to reminder other developers, we should not import scala in this module?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #19579:
URL: https://github.com/apache/kafka/pull/19579#issuecomment-2875378086

   @Yunyung please fix the conflicts


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2062806639


##########
storage/src/test/java/org/apache/kafka/tiered/storage/TieredStorageTestHarness.java:
##########
@@ -154,7 +155,7 @@ public static List<LocalTieredStorage> remoteStorageManagers(Seq<KafkaBroker> br
 
     public static List<BrokerLocalStorage> localStorages(Seq<KafkaBroker> brokers) {
         return CollectionConverters.asJava(brokers).stream()
-                .map(b -> new BrokerLocalStorage(b.config().brokerId(), CollectionConverters.asJava(b.config().logDirs().toSet()),
+                .map(b -> new BrokerLocalStorage(b.config().brokerId(), new HashSet<>(b.config().logDirs()),

Review Comment:
   `Set.copyOf`



##########
storage/src/test/java/org/apache/kafka/tiered/storage/integration/TransactionsWithTieredStoreTest.java:
##########
@@ -75,7 +76,7 @@ public Properties topicConfig() {
     public void maybeWaitForAtLeastOneSegmentUpload(scala.collection.Seq<TopicPartition> topicPartitions) {
         CollectionConverters.asJava(topicPartitions).forEach(topicPartition -> {
             List<BrokerLocalStorage> localStorages = CollectionConverters.asJava(brokers()).stream()
-                    .map(b -> new BrokerLocalStorage(b.config().brokerId(), CollectionConverters.asJava(b.config().logDirs().toSet()), STORAGE_WAIT_TIMEOUT_SEC))
+                    .map(b -> new BrokerLocalStorage(b.config().brokerId(), new HashSet<>(b.config().logDirs()), STORAGE_WAIT_TIMEOUT_SEC))

Review Comment:
   `Set.copyOf`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "Yunyung (via GitHub)" <gi...@apache.org>.
Yunyung commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2064010048


##########
server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java:
##########
@@ -66,8 +68,16 @@ public abstract class AbstractKafkaConfig extends AbstractConfig {
         AddPartitionsToTxnConfig.CONFIG_DEF
     ));
 
+    public final List<String> logDirs;
+
+    @SuppressWarnings("this-escape")
     public AbstractKafkaConfig(ConfigDef definition, Map<?, ?> originals, Map<String, ?> configProviderProps, boolean doLog) {
         super(definition, originals, configProviderProps, doLog);
+        this.logDirs = Csv.parseCsvList(Optional.ofNullable(getString(ServerLogConfigs.LOG_DIRS_CONFIG)).orElse(getString(ServerLogConfigs.LOG_DIR_CONFIG)));

Review Comment:
   Sounds fair. Simple, not hot path, low cost. 
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "TaiJuWu (via GitHub)" <gi...@apache.org>.
TaiJuWu commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2083357389


##########
checkstyle/import-control-clients-integration-tests.xml:
##########
@@ -21,7 +21,6 @@
 <import-control pkg="org.apache.kafka">
     <allow pkg="java"/>
     <allow pkg="org.junit"/>
-    <allow pkg="scala" />

Review Comment:
   Should we change to `<disallow pkg="scala">` ?
   In order to reminder other developers we should not import scala in this module?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2062700142


##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/ListOffsetsIntegrationTest.java:
##########
@@ -243,8 +241,7 @@ private void verifyListOffsets(String topic, int expectedMaxTimestampOffset) thr
         // case 2: test the offsets from recovery path.
         // server will rebuild offset index according to log files if the index files are nonexistent
         Set<String> indexFiles = clusterInstance.brokers().values().stream().flatMap(broker ->
-                CollectionConverters.asJava(broker.config().logDirs()).stream()
-        ).collect(Collectors.toUnmodifiableSet());
+            broker.config().logDirs().stream()).collect(Collectors.toUnmodifiableSet());

Review Comment:
   You can simply use the `toList` collector.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2062700142


##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/admin/ListOffsetsIntegrationTest.java:
##########
@@ -243,8 +241,7 @@ private void verifyListOffsets(String topic, int expectedMaxTimestampOffset) thr
         // case 2: test the offsets from recovery path.
         // server will rebuild offset index according to log files if the index files are nonexistent
         Set<String> indexFiles = clusterInstance.brokers().values().stream().flatMap(broker ->
-                CollectionConverters.asJava(broker.config().logDirs()).stream()
-        ).collect(Collectors.toUnmodifiableSet());
+            broker.config().logDirs().stream()).collect(Collectors.toUnmodifiableSet());

Review Comment:
   You can simply use `toList`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "Yunyung (via GitHub)" <gi...@apache.org>.
Yunyung commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2080661082


##########
core/src/main/scala/kafka/server/BrokerServer.scala:
##########
@@ -709,7 +709,7 @@ class BrokerServer(
         None
       }
 
-      val rlm = new RemoteLogManager(config.remoteLogManagerConfig, config.brokerId, config.logDirs.head, clusterId, time,
+      val rlm = new RemoteLogManager(config.remoteLogManagerConfig, config.brokerId, config.logDirs.asScala.head, clusterId, time,

Review Comment:
   Thanks. I’m using get(0) instead, since List.getFirst() is available starting with Java 21.



##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -214,7 +214,7 @@ class ReplicaManagerTest {
       partition.createLogIfNotExists(isNew = false, isFutureReplica = false,
         new LazyOffsetCheckpoints(rm.highWatermarkCheckpoints.asJava), None)
       rm.checkpointHighWatermarks()
-      config.logDirs.map(s => Paths.get(s, ReplicaManager.HighWatermarkFilename))
+      config.logDirs.asScala.map(s => Paths.get(s, ReplicaManager.HighWatermarkFilename))
         .foreach(checkpointFile => assertTrue(Files.exists(checkpointFile),

Review Comment:
   Nice. Thanks.



##########
core/src/test/scala/integration/kafka/server/KRaftClusterTest.scala:
##########
@@ -1551,7 +1551,7 @@ class KRaftClusterTest {
         // Copy foo-0 to targetParentDir
         // This is so that we can rename the main replica to a future down below
         val parentDir = log.parentDir
-        val targetParentDir = broker0.config.logDirs.filter(_ != parentDir).head
+        val targetParentDir = broker0.config.logDirs.asScala.filter(_ != parentDir).head
         val targetDirFile = new File(targetParentDir, log.dir.getName)

Review Comment:
   Nice. Thanks.



##########
core/src/test/scala/unit/kafka/server/AlterReplicaLogDirsRequestTest.scala:
##########
@@ -91,10 +91,10 @@ class AlterReplicaLogDirsRequestTest extends BaseRequestTest {
   @ParameterizedTest
   @ValueSource(strings = Array("kraft"))
   def testAlterReplicaLogDirsRequestErrorCode(quorum: String): Unit = {
-    val offlineDir = new File(brokers.head.config.logDirs.tail.head).getAbsolutePath
-    val validDir1 = new File(brokers.head.config.logDirs(1)).getAbsolutePath
-    val validDir2 = new File(brokers.head.config.logDirs(2)).getAbsolutePath
-    val validDir3 = new File(brokers.head.config.logDirs(3)).getAbsolutePath
+    val offlineDir = new File(brokers.head.config.logDirs.asScala.tail.head).getAbsolutePath

Review Comment:
   I checked, tail.head means getting the second element (since tail removes the first element), So get(1) gets the same element we originally wanted.
   
   But I'm not sure if the original test was wrong, because `val validDir1 = new File(brokers.head.config.logDirs(1)).getAbsolutePath` also takes the second element...
   
   I'll track this to see whether it's a bug.



##########
core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:
##########
@@ -1165,7 +1165,7 @@ class KafkaConfigTest {
     assertEquals(1, config.brokerId)
     assertEquals(Seq("PLAINTEXT://127.0.0.1:1122"), config.effectiveAdvertisedBrokerListeners.map(JTestUtils.endpointToString))
     assertEquals(Map("127.0.0.1" -> 2, "127.0.0.2" -> 3), config.maxConnectionsPerIpOverrides)
-    assertEquals(List("/tmp1", "/tmp2"), config.logDirs)
+    assertEquals(List("/tmp1", "/tmp2"), config.logDirs.asScala.toList)

Review Comment:
   Nice. Done.



##########
core/src/test/scala/unit/kafka/server/DescribeLogDirsRequestTest.scala:
##########
@@ -42,8 +42,8 @@ class DescribeLogDirsRequestTest extends BaseRequestTest {
   @ParameterizedTest
   @ValueSource(strings = Array("kraft"))
   def testDescribeLogDirsRequest(quorum: String): Unit = {
-    val onlineDir = new File(brokers.head.config.logDirs.head).getAbsolutePath
-    val offlineDir = new File(brokers.head.config.logDirs.tail.head).getAbsolutePath
+    val onlineDir = new File(brokers.head.config.logDirs.asScala.head).getAbsolutePath
+    val offlineDir = new File(brokers.head.config.logDirs.asScala.tail.head).getAbsolutePath

Review Comment:
   > I checked, tail.head means getting the second element (since tail removes the first element)...
   
   ditto



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (WIP) MINOR: Move logDirs config out of KafkaConfig [kafka]

Posted by "Yunyung (via GitHub)" <gi...@apache.org>.
Yunyung commented on code in PR #19579:
URL: https://github.com/apache/kafka/pull/19579#discussion_r2080810066


##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -1237,7 +1237,7 @@ class ReplicaManager(val config: KafkaConfig,
   def describeLogDirs(partitions: Set[TopicPartition]): List[DescribeLogDirsResponseData.DescribeLogDirsResult] = {
     val logsByDir = logManager.allLogs.groupBy(log => log.parentDir)
 
-    config.logDirs.toSet.map { logDir: String =>
+    config.logDirs.asScala.toSet.map { logDir: String =>

Review Comment:
   Great. Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org