You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "kamalcph (via GitHub)" <gi...@apache.org> on 2023/07/27 16:57:42 UTC

[GitHub] [kafka] kamalcph opened a new pull request, #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

kamalcph opened a new pull request, #14114:
URL: https://github.com/apache/kafka/pull/14114

   **Topic -> Broker Synonym**
   
   - local.retention.bytes -> log.local.retention.bytes 
   - local.retention.ms -> log.local.retention.ms
   
   We cannot add synonym for `remote.storage.enable` topic level config as it depends on [KIP-950](https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement)
   
   *More detailed description of your change,
   if necessary. The PR title and PR message become
   the squashed commit message, so use a separate
   comment to ping reviewers.*
   
   *Summary of testing strategy (including rationale)
   for the feature or bug fix. Unit and/or integration
   tests are expected for any behaviour change and
   system tests should be considered for larger changes.*
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] satishd merged pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd merged PR #14114:
URL: https://github.com/apache/kafka/pull/14114


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kamalcph commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1280887247


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {

Review Comment:
   We cannot refer KafkaConfig from LogConfig class.
   
   Filed KAFKA-15290 to initialise/stop the remote log components for a topic when `remote.storage.enable` is set to true/false. In [TopicConfigHandler](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ConfigHandler.scala#L55C13-L55C13), we can add this validation but the config might already saved in the ConfigRepository.
   
   Do you have suggestion on how to add this validation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] satishd commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1278245511


##########
core/src/test/scala/unit/kafka/log/LogConfigTest.scala:
##########
@@ -62,13 +63,17 @@ class LogConfigTest {
     kafkaProps.put(KafkaConfig.LogRollTimeJitterHoursProp, "2")
     kafkaProps.put(KafkaConfig.LogRetentionTimeHoursProp, "2")
     kafkaProps.put(KafkaConfig.LogMessageFormatVersionProp, "0.11.0")
+    kafkaProps.put(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_MS_PROP, "300000")

Review Comment:
   As these values are long types. Can we have a value more than Integer.MAX?



##########
core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:
##########
@@ -713,6 +714,99 @@ class DynamicBrokerConfigTest {
     config.updateCurrentConfig(new KafkaConfig(props))
     assertFalse(config.nonInternalValues.containsKey(KafkaConfig.MetadataLogSegmentMinBytesProp))
   }
+
+  @Test
+  def testDynamicLogLocalRetentionMsConfig(): Unit = {
+    val props = TestUtils.createBrokerConfig(0, TestUtils.MockZkConnect, port = 8181)
+    props.put(KafkaConfig.LogRetentionTimeMillisProp, "1000")

Review Comment:
   As these values are long types. Can we have a value more than Integer.MAX? Same comment in this class for other retention properties.



##########
core/src/test/scala/unit/kafka/log/LogConfigTest.scala:
##########
@@ -62,13 +63,17 @@ class LogConfigTest {
     kafkaProps.put(KafkaConfig.LogRollTimeJitterHoursProp, "2")
     kafkaProps.put(KafkaConfig.LogRetentionTimeHoursProp, "2")
     kafkaProps.put(KafkaConfig.LogMessageFormatVersionProp, "0.11.0")
+    kafkaProps.put(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_MS_PROP, "300000")
+    kafkaProps.put(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_BYTES_PROP, "1024")

Review Comment:
   As these values are long types. Can we have a value more than Integer.MAX?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kamalcph commented on pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on PR #14114:
URL: https://github.com/apache/kafka/pull/14114#issuecomment-1659690038

   > @kamalcph , there's compilation error, could you take a look? https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14114/8/pipeline
   
   rebased the branch against trunk. This [commit](https://github.com/apache/kafka/commit/938fee2b1fec52fa336f68118da120190bff4600) fixes the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1280353910


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -122,6 +122,18 @@ public final class RemoteLogManagerConfig {
             "is full, fetch requests are served with an error.";
     public static final int DEFAULT_REMOTE_LOG_READER_MAX_PENDING_TASKS = 100;
 
+    public static final String LOG_LOCAL_RETENTION_MS_PROP = "log.local.retention.ms";
+    public static final String LOG_LOCAL_RETENTION_MS_DOC = "The number of milliseconds to keep the local log segment before it gets deleted. " +

Review Comment:
   perhaps "before it gets eligible for deletion"
   
   (same for retention bytes)



##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -122,6 +122,18 @@ public final class RemoteLogManagerConfig {
             "is full, fetch requests are served with an error.";
     public static final int DEFAULT_REMOTE_LOG_READER_MAX_PENDING_TASKS = 100;
 
+    public static final String LOG_LOCAL_RETENTION_MS_PROP = "log.local.retention.ms";
+    public static final String LOG_LOCAL_RETENTION_MS_DOC = "The number of milliseconds to keep the local log segment before it gets deleted. " +

Review Comment:
   should we perhaps add here that this configuration is only used when remote storage is enabled at a system level?
   
   (same for retention bytes)



##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {

Review Comment:
   could we please add an assertion and a test that remote storage cannot be enabled for a topic if the remote storage is not enabled at system level? Please let me know if I am missing something and we already have this validation check somewhere else.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon commented on pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14114:
URL: https://github.com/apache/kafka/pull/14114#issuecomment-1659574317

   @kamalcph , there's compilation error, could you take a look?
   https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14114/8/pipeline


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1281717116


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {

Review Comment:
   sure. separate ticket works.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kamalcph commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1278982175


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -424,6 +387,18 @@ public int initFileSize() {
             return 0;
     }
 
+    public boolean remoteStorageEnable() {
+        return remoteLogConfig.remoteStorageEnable;
+    }
+
+    public long localRetentionMs() {
+        return remoteLogConfig.localRetentionMs == LogConfig.DEFAULT_LOCAL_RETENTION_MS ? retentionMs : remoteLogConfig.localRetentionMs;
+    }
+
+    public long localRetentionBytes() {
+        return remoteLogConfig.localRetentionBytes == LogConfig.DEFAULT_LOCAL_RETENTION_BYTES ? retentionSize : remoteLogConfig.localRetentionBytes;

Review Comment:
   yes, correct. The caller should get the real local log retention values when set to -2.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kamalcph commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1281701381


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {

Review Comment:
   Thanks for the suggestion! Can we take this validation separately? If yes, will file a ticket and work on it.
   
   This patch is about adding broker-level config synonyms. Including this validation in the PR will increase the scope and delay the PR from landing it. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kamalcph commented on pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on PR #14114:
URL: https://github.com/apache/kafka/pull/14114#issuecomment-1654008152

   @satishd @divijvaidya @showuon 
   
   PTAL when you get chance!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kamalcph commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1288729452


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {

Review Comment:
   Opened #14176 to address this comment. PTAL.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1280113062


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {
 
-        public final boolean remoteStorageEnable;
+        private final boolean remoteStorageEnable;
+        private final long localRetentionMs;
+        private final long localRetentionBytes;
 
-        public final long localRetentionMs;
-        public final long localRetentionBytes;
-
-        private RemoteLogConfig(LogConfig config, long retentionMs, long retentionSize) {
+        private RemoteLogConfig(LogConfig config) {
             this.remoteStorageEnable = config.getBoolean(TopicConfig.REMOTE_LOG_STORAGE_ENABLE_CONFIG);
-
-            long localLogRetentionMs = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG);
-
-            // -2 indicates to derive value from retentionMs property.
-            if (localLogRetentionMs == -2)
-                this.localRetentionMs = retentionMs;
-            else {
-                // Added validation here to check the effective value should not be more than RetentionMs.
-                if (localLogRetentionMs == -1 && retentionMs != -1)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG, localLogRetentionMs,
-                        "Value must not be -1 as " + TopicConfig.RETENTION_MS_CONFIG + " value is set as " + retentionMs);
-
-                if (localLogRetentionMs > retentionMs)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG, localLogRetentionMs,
-                        "Value must not be more than property: " + TopicConfig.RETENTION_MS_CONFIG + " value.");
-
-                this.localRetentionMs = localLogRetentionMs;
-            }
-
-            long localLogRetentionBytes = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG);
-
-            // -2 indicates to derive value from retentionSize property.
-            if (localLogRetentionBytes == -2)
-                this.localRetentionBytes = retentionSize;
-            else {
-                // Added validation here to check the effective value should not be more than RetentionBytes.
-                if (localLogRetentionBytes == -1 && retentionSize != -1)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG, localLogRetentionBytes,
-                        "Value must not be -1 as " + TopicConfig.RETENTION_BYTES_CONFIG + " value is set as " + retentionSize);
-
-                if (localLogRetentionBytes > retentionSize)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG, localLogRetentionBytes,
-                        "Value must not be more than property: " + TopicConfig.RETENTION_BYTES_CONFIG + " value.");
-
-                this.localRetentionBytes = localLogRetentionBytes;
-            }
+            this.localRetentionMs = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG);
+            this.localRetentionBytes = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG);

Review Comment:
   OK, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kamalcph commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1280095629


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {
 
-        public final boolean remoteStorageEnable;
+        private final boolean remoteStorageEnable;
+        private final long localRetentionMs;
+        private final long localRetentionBytes;
 
-        public final long localRetentionMs;
-        public final long localRetentionBytes;
-
-        private RemoteLogConfig(LogConfig config, long retentionMs, long retentionSize) {
+        private RemoteLogConfig(LogConfig config) {
             this.remoteStorageEnable = config.getBoolean(TopicConfig.REMOTE_LOG_STORAGE_ENABLE_CONFIG);
-
-            long localLogRetentionMs = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG);
-
-            // -2 indicates to derive value from retentionMs property.
-            if (localLogRetentionMs == -2)
-                this.localRetentionMs = retentionMs;
-            else {
-                // Added validation here to check the effective value should not be more than RetentionMs.
-                if (localLogRetentionMs == -1 && retentionMs != -1)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG, localLogRetentionMs,
-                        "Value must not be -1 as " + TopicConfig.RETENTION_MS_CONFIG + " value is set as " + retentionMs);
-
-                if (localLogRetentionMs > retentionMs)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG, localLogRetentionMs,
-                        "Value must not be more than property: " + TopicConfig.RETENTION_MS_CONFIG + " value.");
-
-                this.localRetentionMs = localLogRetentionMs;
-            }
-
-            long localLogRetentionBytes = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG);
-
-            // -2 indicates to derive value from retentionSize property.
-            if (localLogRetentionBytes == -2)
-                this.localRetentionBytes = retentionSize;
-            else {
-                // Added validation here to check the effective value should not be more than RetentionBytes.
-                if (localLogRetentionBytes == -1 && retentionSize != -1)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG, localLogRetentionBytes,
-                        "Value must not be -1 as " + TopicConfig.RETENTION_BYTES_CONFIG + " value is set as " + retentionSize);
-
-                if (localLogRetentionBytes > retentionSize)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG, localLogRetentionBytes,
-                        "Value must not be more than property: " + TopicConfig.RETENTION_BYTES_CONFIG + " value.");
-
-                this.localRetentionBytes = localLogRetentionBytes;
-            }
+            this.localRetentionMs = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG);
+            this.localRetentionBytes = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG);

Review Comment:
   Those values will be replaced when accessing them in LogConfig.java#L394 and LogConfig.java#398. The intention is to preserve the original value as it is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kamalcph commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1280887247


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {

Review Comment:
   We cannot infer KafkaConfig from LogConfig class.
   
   Filed KAFKA-15290 to initialise/stop the remote log components for a topic when `remote.storage.enable` is set to true/false. In [TopicConfigHandler](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ConfigHandler.scala#L55C13-L55C13), we can add this validation but the config might already saved in the ConfigRepository.
   
   Do you have suggestion on how to add this validation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1278885944


##########
core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:
##########
@@ -713,6 +714,99 @@ class DynamicBrokerConfigTest {
     config.updateCurrentConfig(new KafkaConfig(props))
     assertFalse(config.nonInternalValues.containsKey(KafkaConfig.MetadataLogSegmentMinBytesProp))
   }
+
+  @Test
+  def testDynamicLogLocalRetentionMsConfig(): Unit = {
+    val props = TestUtils.createBrokerConfig(0, TestUtils.MockZkConnect, port = 8181)
+    props.put(KafkaConfig.LogRetentionTimeMillisProp, "2592000000")
+    val config = KafkaConfig(props)
+    val dynamicLogConfig = new DynamicLogConfig(mock(classOf[LogManager]), mock(classOf[KafkaServer]))
+    config.dynamicConfig.initialize(None)
+    config.dynamicConfig.addBrokerReconfigurable(dynamicLogConfig)
+
+    val newProps = new Properties()
+    newProps.put(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_MS_PROP, "2160000000")
+    // update default config
+    config.dynamicConfig.validate(newProps, perBrokerConfig = false)
+    config.dynamicConfig.updateDefaultConfig(newProps)
+    assertEquals(2160000000L, config.logLocalRetentionMs)
+
+    // update per broker config
+    config.dynamicConfig.validate(newProps, perBrokerConfig = true)
+    newProps.put(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_MS_PROP, "2150000000")
+    config.dynamicConfig.updateBrokerConfig(0, newProps)
+    assertEquals(2150000000L, config.logLocalRetentionMs)
+  }
+
+  @Test
+  def testDynamicLogLocalRetentionSizeConfig(): Unit = {
+    val props = TestUtils.createBrokerConfig(0, TestUtils.MockZkConnect, port = 8181)
+    props.put(KafkaConfig.LogRetentionBytesProp, "4294967296")
+    val config = KafkaConfig(props)
+    val dynamicLogConfig = new DynamicLogConfig(mock(classOf[LogManager]), mock(classOf[KafkaServer]))
+    config.dynamicConfig.initialize(None)
+    config.dynamicConfig.addBrokerReconfigurable(dynamicLogConfig)
+
+    val newProps = new Properties()
+    newProps.put(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_BYTES_PROP, "4294967295")
+    // update default config
+    config.dynamicConfig.validate(newProps, perBrokerConfig = false)
+    config.dynamicConfig.updateDefaultConfig(newProps)
+    assertEquals(4294967295L, config.logLocalRetentionBytes)
+
+    // update per broker config
+    config.dynamicConfig.validate(newProps, perBrokerConfig = true)
+    newProps.put(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_BYTES_PROP, "4294967294")
+    config.dynamicConfig.updateBrokerConfig(0, newProps)
+    assertEquals(4294967294L, config.logLocalRetentionBytes)
+  }
+
+  @Test
+  def testDynamicLogLocalRetentionSkipsOnInvalidConfig(): Unit = {
+    val props = TestUtils.createBrokerConfig(0, TestUtils.MockZkConnect, port = 8181)
+    props.put(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_MS_PROP, "1000")
+    props.put(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_BYTES_PROP, "1024")
+    val config = KafkaConfig(props)
+    config.dynamicConfig.initialize(None)
+
+    // Check for invalid localRetentionMs < -2
+    verifyConfigUpdateWithInvalidConfig(config, props, Map.empty, Map(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_MS_PROP -> "-3"))
+    // Check for invalid localRetentionBytes < -2
+    verifyConfigUpdateWithInvalidConfig(config, props, Map.empty, Map(RemoteLogManagerConfig.LOG_LOCAL_RETENTION_BYTES_PROP -> "-3"))
+  }
+
+  @Test
+  def testDynamicLogLocalRetentionThrowsOnIncorrectConfig(): Unit = {
+    // Check for incorrect case of logLocalRetentionMs > retentionMs
+    verifyIncorrectLogLocalRetentionProps(2000L, 2, 100, 1000L)
+    // Check for incorrect case of logLocalRetentionBytes > retentionBytes
+    verifyIncorrectLogLocalRetentionProps(500L, 200, 100, 1000L)
+    // Check for incorrect case of logLocalRetentionMs (-1 viz unlimited) > retentionMs,
+    verifyIncorrectLogLocalRetentionProps(-1, 200, 100, 1000L)
+    // Check for incorrect case of logLocalRetentionBytes(-1 viz unlimited) > retentionBytes
+    verifyIncorrectLogLocalRetentionProps(2000L, -1, 100, 1000L)
+  }
+
+  def verifyIncorrectLogLocalRetentionProps(logLocalRetentionMs: Long,
+                                            logLocalRetentionBytes: Long,
+                                            retentionBytes: Long,
+                                            retentionMs: Long): Unit = {

Review Comment:
   nit: It's hard to review this validation method with current parameter order. Could you put the same type of config in pair? ex:
   ```
    def verifyIncorrectLogLocalRetentionProps(logLocalRetentionMs: Long,  retentionMs: Long,  logLocalRetentionBytes: Long, retentionBytes: Long) 
   ```



##########
core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:
##########
@@ -1784,4 +1788,22 @@ class KafkaConfigTest {
     props.put(KafkaConfig.ConsumerGroupHeartbeatIntervalMsProp, "25")
     assertThrows(classOf[IllegalArgumentException], () => KafkaConfig.fromProps(props))
   }
+
+  @Test
+  def testMultipleLogDirectoriesNotSupportedWithRemoteLogStorage(): Unit = {
+    val props = TestUtils.createBrokerConfig(0, TestUtils.MockZkConnect, port = 8181)
+    props.put(RemoteLogManagerConfig.REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP, String.valueOf(true))
+    props.put(KafkaConfig.LogDirsProp, "/tmp/a,/tmp/b")
+
+    val caught = assertThrows(classOf[ConfigException], () => KafkaConfig.fromProps(props))
+    assertTrue(caught.getMessage.contains("Multiple log directories `/tmp/a,/tmp/b` are not supported when remote log storage is enabled"))
+  }
+
+  @Test
+  def testSingleLogDirectoryWithRemoteLogStorage(): Unit = {
+    val props = TestUtils.createBrokerConfig(0, TestUtils.MockZkConnect, port = 8181)
+    props.put(RemoteLogManagerConfig.REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP, String.valueOf(true))
+    props.put(KafkaConfig.LogDirsProp, "/tmp/a")
+    KafkaConfig.fromProps(props)

Review Comment:
   nit: `assertDoesNotThrow(() ->  KafkaConfig.fromProps(props));`



##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -215,7 +227,19 @@ public final class RemoteLogManagerConfig {
                                   DEFAULT_REMOTE_LOG_READER_MAX_PENDING_TASKS,
                                   atLeast(1),
                                   MEDIUM,
-                                  REMOTE_LOG_READER_MAX_PENDING_TASKS_DOC);
+                                  REMOTE_LOG_READER_MAX_PENDING_TASKS_DOC)
+                  .defineInternal(LOG_LOCAL_RETENTION_MS_PROP,
+                          LONG,
+                          DEFAULT_LOG_LOCAL_RETENTION_MS,
+                          atLeast(DEFAULT_LOG_LOCAL_RETENTION_MS),
+                          MEDIUM,
+                          LOG_LOCAL_RETENTION_MS_DOC)
+                  .defineInternal(LOG_LOCAL_RETENTION_BYTES_PROP,
+                          LONG,
+                          DEFAULT_LOG_LOCAL_RETENTION_BYTES,

Review Comment:
   Could we try to align with above lines?



##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -424,6 +387,18 @@ public int initFileSize() {
             return 0;
     }
 
+    public boolean remoteStorageEnable() {
+        return remoteLogConfig.remoteStorageEnable;
+    }
+
+    public long localRetentionMs() {
+        return remoteLogConfig.localRetentionMs == LogConfig.DEFAULT_LOCAL_RETENTION_MS ? retentionMs : remoteLogConfig.localRetentionMs;
+    }
+
+    public long localRetentionBytes() {
+        return remoteLogConfig.localRetentionBytes == LogConfig.DEFAULT_LOCAL_RETENTION_BYTES ? retentionSize : remoteLogConfig.localRetentionBytes;

Review Comment:
   When will we use them? I think they should be called when retrieved the value since we don't expect to get value of `-2`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] kamalcph commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1281723208


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {

Review Comment:
   Filed KAFKA-15295 to add this validation. Thanks for reviewing the PR!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1280072894


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {
 
-        public final boolean remoteStorageEnable;
+        private final boolean remoteStorageEnable;
+        private final long localRetentionMs;
+        private final long localRetentionBytes;
 
-        public final long localRetentionMs;
-        public final long localRetentionBytes;
-
-        private RemoteLogConfig(LogConfig config, long retentionMs, long retentionSize) {
+        private RemoteLogConfig(LogConfig config) {
             this.remoteStorageEnable = config.getBoolean(TopicConfig.REMOTE_LOG_STORAGE_ENABLE_CONFIG);
-
-            long localLogRetentionMs = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG);
-
-            // -2 indicates to derive value from retentionMs property.
-            if (localLogRetentionMs == -2)
-                this.localRetentionMs = retentionMs;
-            else {
-                // Added validation here to check the effective value should not be more than RetentionMs.
-                if (localLogRetentionMs == -1 && retentionMs != -1)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG, localLogRetentionMs,
-                        "Value must not be -1 as " + TopicConfig.RETENTION_MS_CONFIG + " value is set as " + retentionMs);
-
-                if (localLogRetentionMs > retentionMs)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG, localLogRetentionMs,
-                        "Value must not be more than property: " + TopicConfig.RETENTION_MS_CONFIG + " value.");
-
-                this.localRetentionMs = localLogRetentionMs;
-            }
-
-            long localLogRetentionBytes = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG);
-
-            // -2 indicates to derive value from retentionSize property.
-            if (localLogRetentionBytes == -2)
-                this.localRetentionBytes = retentionSize;
-            else {
-                // Added validation here to check the effective value should not be more than RetentionBytes.
-                if (localLogRetentionBytes == -1 && retentionSize != -1)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG, localLogRetentionBytes,
-                        "Value must not be -1 as " + TopicConfig.RETENTION_BYTES_CONFIG + " value is set as " + retentionSize);
-
-                if (localLogRetentionBytes > retentionSize)
-                    throw new ConfigException(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG, localLogRetentionBytes,
-                        "Value must not be more than property: " + TopicConfig.RETENTION_BYTES_CONFIG + " value.");
-
-                this.localRetentionBytes = localLogRetentionBytes;
-            }
+            this.localRetentionMs = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_MS_CONFIG);
+            this.localRetentionBytes = config.getLong(TopicConfig.LOCAL_LOG_RETENTION_BYTES_CONFIG);

Review Comment:
   Since we removed this line above:
   ```
               if (localLogRetentionBytes == -2)
                   this.localRetentionBytes = retentionSize;
   ```
   So, when will we replace `-2` into the retentionSize?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] showuon commented on pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14114:
URL: https://github.com/apache/kafka/pull/14114#issuecomment-1659723172

   Thanks! Let's wait for the CI build.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on a diff in pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14114:
URL: https://github.com/apache/kafka/pull/14114#discussion_r1281624661


##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java:
##########
@@ -102,49 +102,14 @@ public String topicWarningMessage(String topicName) {
 
     public static class RemoteLogConfig {

Review Comment:
   > Do you have suggestion on how to add this validation?
   
   Places such as `KafkaApis.scala`, `ControllerApis.scala` and `ZkAdminManager` already have access to the `KafkaConfig` instance. When `createTopic` or `incrementAlterConfig` API is called, they call `LogConfig.validate()` today. Instead, they should do something like:
   ```
   val defaultConfigs = kafkaConfig.extractLogConfigMap
   LogConfig.validate(newConfigs, defaultConfigs) // new method
   ```
   
   You will need to make the following changes:
   
   1. Add the following line to `KafkaConfig.extractLogConfigMap()`
   `logProps.put(RemoteLogManagerConfig.REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP, remoteLogManagerConfig.enableRemoteStorageSystem())`. 
   
   2. At places where we call `LogConfig.validate()` such as `AdminZkClient.validateTopicConfig()`, use the new LogConfig.validate() method. You can also pass KafkaConfig to AdminZkClient via ZkAdminManager -> AdminZkClient.validateTopicConfig(). Similarly for KRaft. 
   
   3. Inside the new `LogConfig.validate()` method, add the check that we want i.e. if system level remote storage is not enabled, then we cannot enable topic level remote config.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] satishd commented on pull request #14114: KAFKA-12969: Add broker level config synonyms for topic level tiered storage configs

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #14114:
URL: https://github.com/apache/kafka/pull/14114#issuecomment-1663512246

   A few unrelated tests failed in Jenkins jobs, merging it to trunk.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org