You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "tinaselenge (via GitHub)" <gi...@apache.org> on 2023/08/21 11:55:59 UTC

[GitHub] [kafka] tinaselenge opened a new pull request, #14266: KAFKA-15294: Publish remote storage configs

tinaselenge opened a new pull request, #14266:
URL: https://github.com/apache/kafka/pull/14266

   - Make RemoteLogManagerConfigs public
   - Add tasks to generate html docs for the configs
   - Include config docs in the main site
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] tinaselenge commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1300374040


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -146,111 +146,111 @@ public final class RemoteLogManagerConfig {
     public static final ConfigDef CONFIG_DEF = new ConfigDef();
 
     static {
-        CONFIG_DEF.defineInternal(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
+        CONFIG_DEF.define(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
                                   BOOLEAN,
                                   DEFAULT_REMOTE_LOG_STORAGE_SYSTEM_ENABLE,
                                   null,
                                   MEDIUM,
                                   REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_STORAGE_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CLASS_NAME,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
                                   STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
+                  .define(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
+                  .define(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
                                   INT,
                                   DEFAULT_REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES,
                                   atLeast(0),
                                   LOW,
                                   REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_DOC)
-                  .defineInternal(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,
+                  .define(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,

Review Comment:
   Thank you for the feedback @divijvaidya. I wasn't able to generate docs for internal configs due to `Internal configuration won't show up in the docs and aren't intended for general use`. I can address the other comments and mark this PR as draft until KAFKA-14912 is merged, what do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] satishd commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1305229020


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -56,28 +56,28 @@ public final class RemoteLogManagerConfig {
 
 
     public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP = "remote.log.storage.system.enable";
-    public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC = "Whether to enable tier storage functionality in a broker or not. Valid values " +
-            "are `true` or `false` and the default value is false. When it is true broker starts all the services required for tiered storage functionality.";
+    public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC = "Whether to enable remote storage functionality in a broker or not. Valid values " +

Review Comment:
   Thanks @divijvaidya for the clarification. Let us continue having "tiered storage" as mentioned earlier. cc @tinaselenge 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #14266:
URL: https://github.com/apache/kafka/pull/14266#issuecomment-1687769802

   > I think we have always referred to the functionality as Tiered Storage rather than Tier Storage
   
   Across Kafka code base including the config we refer to it as "remote storage" such as `RemoteStorageException`, `REMOTE_STORAGE_MANAGER` etc. Due to this, let's stick to "Remote Storage" as the terminology.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #14266:
URL: https://github.com/apache/kafka/pull/14266#issuecomment-1689491667

   > the remote log manager configs are left as internal until the required implementation is complete.
   
   Some of these configs are implemented and some aren't. We want to expose the configurations which are already implemented. As an example, `REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP` should be exposed in the documentation since it is implemented but `REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP` isn't.
   
   As part of this PR, I would suggest that you look through all properties in RemoteLogManagerConfig, see which ones are actually being used in the code, make them external (i.e. change from `defineInternal` to `define`) and keep `defineInternal` for the ones which are not being used.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya merged pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya merged PR #14266:
URL: https://github.com/apache/kafka/pull/14266


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] tinaselenge commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1301308504


##########
docs/configuration.html:
##########
@@ -286,7 +286,12 @@ <h4 class="anchor-heading"><a id="mirrormakerheartbeatconfigs" class="anchor-lin
   Below is the configuration of MirrorMaker 2 heartbeat connector for checking connectivity between connectors and clusters.
   <!--#include virtual="generated/mirror_heartbeat_config.html" -->
 
-  <h3 class="anchor-heading"><a id="systemproperties" class="anchor-link"></a><a href="#systemproperties">3.9 System Properties</a></h3>
+  <h3 class="anchor-heading"><a id="remotestorageconfigs" class="anchor-link"></a><a href="#remotestorageconfigs">3.9 Tier Storage Configs</a></h3>
+  Below is the configuration of Tier Storage.

Review Comment:
   I noticed that some of the config docs seem to refer to tier storage. Perhaps I should update those?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1301294921


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -146,111 +146,111 @@ public final class RemoteLogManagerConfig {
     public static final ConfigDef CONFIG_DEF = new ConfigDef();
 
     static {
-        CONFIG_DEF.defineInternal(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
+        CONFIG_DEF.define(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
                                   BOOLEAN,
                                   DEFAULT_REMOTE_LOG_STORAGE_SYSTEM_ENABLE,
                                   null,
                                   MEDIUM,
                                   REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_STORAGE_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CLASS_NAME,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
                                   STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
+                  .define(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
+                  .define(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
                                   INT,
                                   DEFAULT_REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES,
                                   atLeast(0),
                                   LOW,
                                   REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_DOC)
-                  .defineInternal(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,
+                  .define(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,

Review Comment:
   > I wasn't able to generate docs for internal configs
   
   which is perfectly fine. We don't want to publish docs for internal configs since they are not being used.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1300083853


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -146,111 +146,111 @@ public final class RemoteLogManagerConfig {
     public static final ConfigDef CONFIG_DEF = new ConfigDef();
 
     static {
-        CONFIG_DEF.defineInternal(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
+        CONFIG_DEF.define(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
                                   BOOLEAN,
                                   DEFAULT_REMOTE_LOG_STORAGE_SYSTEM_ENABLE,
                                   null,
                                   MEDIUM,
                                   REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_STORAGE_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CLASS_NAME,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
                                   STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
+                  .define(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
+                  .define(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
                                   INT,
                                   DEFAULT_REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES,
                                   atLeast(0),
                                   LOW,
                                   REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_DOC)
-                  .defineInternal(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,
+                  .define(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,

Review Comment:
   This is unused so far until https://issues.apache.org/jira/browse/KAFKA-14912 completes. Hence, this should be internal. Can you please verify the same for rest of the configs as well?



##########
docs/toc.html:
##########
@@ -52,7 +52,8 @@
                 <li><a href="#streamsconfigs">3.6 Kafka Streams Configs</a>
                 <li><a href="#adminclientconfigs">3.7 AdminClient Configs</a>
                 <li><a href="#mirrormakerconfigs">3.8 MirrorMaker Configs</a>
-                <li><a href="#systemproperties">3.9 System Properties</a>
+                <li><a href="#remotestorageconfigs">3.9 Tier Storage Configs</a></li>

Review Comment:
   "Remote Storage Configs"
   
   Also please move this to 3.10 (instead of 3.9) so that existing hyperlinks for "System Properties" continue working properly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1303972505


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -56,28 +56,28 @@ public final class RemoteLogManagerConfig {
 
 
     public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP = "remote.log.storage.system.enable";
-    public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC = "Whether to enable tier storage functionality in a broker or not. Valid values " +
-            "are `true` or `false` and the default value is false. When it is true broker starts all the services required for tiered storage functionality.";
+    public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC = "Whether to enable remote storage functionality in a broker or not. Valid values " +

Review Comment:
   Hi @satishd 
   It is on my [suggestion](https://github.com/apache/kafka/pull/14266#issuecomment-1687769802) that @tinaselenge made this change. I suggested that because all our publicly facing configurations are named as "remote storage" and same for code related files such as RemoteLogManager. Is there a reason we want to use Tiered storage nomenclature?
   
   I don't have a strong opinion on either but want to ensure that we stick to one nomenclature in the code base and public docs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] tinaselenge commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1301651698


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -146,111 +146,111 @@ public final class RemoteLogManagerConfig {
     public static final ConfigDef CONFIG_DEF = new ConfigDef();
 
     static {
-        CONFIG_DEF.defineInternal(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
+        CONFIG_DEF.define(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
                                   BOOLEAN,
                                   DEFAULT_REMOTE_LOG_STORAGE_SYSTEM_ENABLE,
                                   null,
                                   MEDIUM,
                                   REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_STORAGE_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CLASS_NAME,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
                                   STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
+                  .define(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
+                  .define(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
                                   INT,
                                   DEFAULT_REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES,
                                   atLeast(0),
                                   LOW,
                                   REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_DOC)
-                  .defineInternal(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,
+                  .define(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,

Review Comment:
   Ah, understood. I thought the issue is about making the internal configs public and doc them. Since some of the key configs such as enabling remote storage functionality is missing, I thought it would look good odd. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] satishd commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1303795865


##########
docs/configuration.html:
##########
@@ -303,6 +303,11 @@ <h4><a id="org.apache.kafka.disallowed.login.modules"></a><a id="systempropertie
       </tbody></table>
     </li>
   </ul>
+
+  <h3 class="anchor-heading"><a id="remotestorageconfigs" class="anchor-link"></a><a href="#remotestorageconfigs">3.10 Remote Storage Configs</a></h3>

Review Comment:
   This feature is known as "Tiered Storage". Let us not change that as it will cause confusion to users. 



##########
storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManagerConfig.java:
##########
@@ -49,26 +49,26 @@ public final class TopicBasedRemoteLogMetadataManagerConfig {
     public static final String REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS_PROP = "remote.log.metadata.initialization.retry.interval.ms";
 
     public static final int DEFAULT_REMOTE_LOG_METADATA_TOPIC_PARTITIONS = 50;
-    public static final long DEFAULT_REMOTE_LOG_METADATA_TOPIC_RETENTION_MILLIS = -1L;
+    public static final long DEFAULT_REMOTE_LOG_METADATA_TOPIC_RETENTION_MS = -1L;
     public static final short DEFAULT_REMOTE_LOG_METADATA_TOPIC_REPLICATION_FACTOR = 3;
     public static final long DEFAULT_REMOTE_LOG_METADATA_CONSUME_WAIT_MS = 2 * 60 * 1000L;
     public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_MAX_TIMEOUT_MS = 2 * 60 * 1000L;
     public static final long DEFAULT_REMOTE_LOG_METADATA_INITIALIZATION_RETRY_INTERVAL_MS = 5 * 1000L;
 
-    public static final String REMOTE_LOG_METADATA_TOPIC_REPLICATION_FACTOR_DOC = "Replication factor of remote log metadata Topic.";
-    public static final String REMOTE_LOG_METADATA_TOPIC_PARTITIONS_DOC = "The number of partitions for remote log metadata Topic.";
-    public static final String REMOTE_LOG_METADATA_TOPIC_RETENTION_MS_DOC = "Remote log metadata topic log retention in milli seconds." +
+    public static final String REMOTE_LOG_METADATA_TOPIC_REPLICATION_FACTOR_DOC = "Replication factor of remote log metadata topic.";
+    public static final String REMOTE_LOG_METADATA_TOPIC_PARTITIONS_DOC = "The number of partitions for remote log metadata topic.";
+    public static final String REMOTE_LOG_METADATA_TOPIC_RETENTION_MS_DOC = "Retention of remote log metadata topic in milliseconds. " +
             "Default: -1, that means unlimited. Users can configure this value based on their use cases. " +
             "To avoid any data loss, this value should be more than the maximum retention period of any topic enabled with " +
-            "tiered storage in the cluster.";
-    public static final String REMOTE_LOG_METADATA_CONSUME_WAIT_MS_DOC = "The amount of time in milli seconds to wait for the local consumer to " +
+            "remote storage in the cluster.";

Review Comment:
   Ditto for tiered.



##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -56,28 +56,28 @@ public final class RemoteLogManagerConfig {
 
 
     public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP = "remote.log.storage.system.enable";
-    public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC = "Whether to enable tier storage functionality in a broker or not. Valid values " +
-            "are `true` or `false` and the default value is false. When it is true broker starts all the services required for tiered storage functionality.";
+    public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC = "Whether to enable remote storage functionality in a broker or not. Valid values " +

Review Comment:
   Let us keep this as "tiered storage"



##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -56,28 +56,28 @@ public final class RemoteLogManagerConfig {
 
 
     public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP = "remote.log.storage.system.enable";
-    public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC = "Whether to enable tier storage functionality in a broker or not. Valid values " +
-            "are `true` or `false` and the default value is false. When it is true broker starts all the services required for tiered storage functionality.";
+    public static final String REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC = "Whether to enable remote storage functionality in a broker or not. Valid values " +
+            "are `true` or `false` and the default value is false. When it is true broker starts all the services required for the remote storage functionality.";

Review Comment:
   Let us keep this as "tiered storage"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] clolov commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1301284451


##########
docs/toc.html:
##########
@@ -52,7 +52,8 @@
                 <li><a href="#streamsconfigs">3.6 Kafka Streams Configs</a>
                 <li><a href="#adminclientconfigs">3.7 AdminClient Configs</a>
                 <li><a href="#mirrormakerconfigs">3.8 MirrorMaker Configs</a>
-                <li><a href="#systemproperties">3.9 System Properties</a>
+                <li><a href="#remotestorageconfigs">3.9 Tier Storage Configs</a></li>

Review Comment:
   ```suggestion
                   <li><a href="#remotestorageconfigs">3.9 Tiered Storage Configs</a></li>
   ```



##########
docs/configuration.html:
##########
@@ -286,7 +286,12 @@ <h4 class="anchor-heading"><a id="mirrormakerheartbeatconfigs" class="anchor-lin
   Below is the configuration of MirrorMaker 2 heartbeat connector for checking connectivity between connectors and clusters.
   <!--#include virtual="generated/mirror_heartbeat_config.html" -->
 
-  <h3 class="anchor-heading"><a id="systemproperties" class="anchor-link"></a><a href="#systemproperties">3.9 System Properties</a></h3>
+  <h3 class="anchor-heading"><a id="remotestorageconfigs" class="anchor-link"></a><a href="#remotestorageconfigs">3.9 Tier Storage Configs</a></h3>
+  Below is the configuration of Tier Storage.

Review Comment:
   ```suggestion
     <h3 class="anchor-heading"><a id="remotestorageconfigs" class="anchor-link"></a><a href="#remotestorageconfigs">3.9 Tiered Storage Configs</a></h3>
     Below are the configuration properties for Tiered Storage.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1301603620


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -146,111 +146,111 @@ public final class RemoteLogManagerConfig {
     public static final ConfigDef CONFIG_DEF = new ConfigDef();
 
     static {
-        CONFIG_DEF.defineInternal(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
+        CONFIG_DEF.define(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
                                   BOOLEAN,
                                   DEFAULT_REMOTE_LOG_STORAGE_SYSTEM_ENABLE,
                                   null,
                                   MEDIUM,
                                   REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_STORAGE_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CLASS_NAME,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
                                   STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
+                  .define(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
+                  .define(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
                                   INT,
                                   DEFAULT_REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES,
                                   atLeast(0),
                                   LOW,
                                   REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_DOC)
-                  .defineInternal(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,
+                  .define(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,

Review Comment:
   Let me clarify. We do indeed want to make the configs public but we only want to make them externally visible if they are implement. Since https://issues.apache.org/jira/browse/KAFKA-14912 is not implemented yet, we can keep it as internal but still have rest of the configs as public.
   
   This will ensure that 3.6 contains public documentation for things which are being released.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] divijvaidya commented on pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #14266:
URL: https://github.com/apache/kafka/pull/14266#issuecomment-1695276501

   Test failures are unrelated:
   ```
   [Build / JDK 17 and Scala 2.13 / kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/kafka.server/DynamicBrokerReconfigurationTest/Build___JDK_17_and_Scala_2_13___testThreadPoolResize__/)
   [Build / JDK 17 and Scala 2.13 / org.apache.kafka.controller.QuorumControllerTest.testBalancePartitionLeaders()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/org.apache.kafka.controller/QuorumControllerTest/Build___JDK_17_and_Scala_2_13___testBalancePartitionLeaders__/)
   [Build / JDK 17 and Scala 2.13 / org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_17_and_Scala_2_13___initializationError/)
   [Build / JDK 17 and Scala 2.13 / org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/org.apache.kafka.trogdor.coordinator/CoordinatorTest/Build___JDK_17_and_Scala_2_13___testTaskRequestWithOldStartMsGetsUpdated__/)
   [Build / JDK 11 and Scala 2.13 / kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/kafka.server/DynamicBrokerReconfigurationTest/Build___JDK_11_and_Scala_2_13___testThreadPoolResize__/)
   [Build / JDK 11 and Scala 2.13 / org.apache.kafka.controller.QuorumControllerTest.testBalancePartitionLeaders()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/org.apache.kafka.controller/QuorumControllerTest/Build___JDK_11_and_Scala_2_13___testBalancePartitionLeaders__/)
   [Build / JDK 11 and Scala 2.13 / org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_11_and_Scala_2_13___initializationError/)
   [Build / JDK 20 and Scala 2.13 / kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/kafka.server/DynamicBrokerReconfigurationTest/Build___JDK_20_and_Scala_2_13___testThreadPoolResize__/)
   [Build / JDK 20 and Scala 2.13 / org.apache.kafka.controller.QuorumControllerTest.testBalancePartitionLeaders()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/org.apache.kafka.controller/QuorumControllerTest/Build___JDK_20_and_Scala_2_13___testBalancePartitionLeaders__/)
   [Build / JDK 20 and Scala 2.13 / org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14266/6/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)
   ```
   
   Merging to 3.6


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] tinaselenge commented on pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on PR #14266:
URL: https://github.com/apache/kafka/pull/14266#issuecomment-1693130061

   Thank you @divijvaidya @satishd for reviewing the PR :) . I believe I have addressed the comments. Please let me know if there is any further comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] tinaselenge commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1301308504


##########
docs/configuration.html:
##########
@@ -286,7 +286,12 @@ <h4 class="anchor-heading"><a id="mirrormakerheartbeatconfigs" class="anchor-lin
   Below is the configuration of MirrorMaker 2 heartbeat connector for checking connectivity between connectors and clusters.
   <!--#include virtual="generated/mirror_heartbeat_config.html" -->
 
-  <h3 class="anchor-heading"><a id="systemproperties" class="anchor-link"></a><a href="#systemproperties">3.9 System Properties</a></h3>
+  <h3 class="anchor-heading"><a id="remotestorageconfigs" class="anchor-link"></a><a href="#remotestorageconfigs">3.9 Tier Storage Configs</a></h3>
+  Below is the configuration of Tier Storage.

Review Comment:
   I noticed that some of the config docs seem to refer to tier storage. Perhaps I should update those?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] tinaselenge commented on a diff in pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14266:
URL: https://github.com/apache/kafka/pull/14266#discussion_r1301326223


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -146,111 +146,111 @@ public final class RemoteLogManagerConfig {
     public static final ConfigDef CONFIG_DEF = new ConfigDef();
 
     static {
-        CONFIG_DEF.defineInternal(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
+        CONFIG_DEF.define(REMOTE_LOG_STORAGE_SYSTEM_ENABLE_PROP,
                                   BOOLEAN,
                                   DEFAULT_REMOTE_LOG_STORAGE_SYSTEM_ENABLE,
                                   null,
                                   MEDIUM,
                                   REMOTE_LOG_STORAGE_SYSTEM_ENABLE_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_STORAGE_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CONFIG_PREFIX_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
+                  .define(REMOTE_STORAGE_MANAGER_CLASS_PATH_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_STORAGE_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_PROP,
                                   STRING,
                                   DEFAULT_REMOTE_LOG_METADATA_MANAGER_CLASS_NAME,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
+                  .define(REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_PROP,
                                   STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_CLASS_PATH_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
+                  .define(REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_PROP, STRING,
                                   null,
                                   new ConfigDef.NonEmptyString(),
                                   MEDIUM,
                                   REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME_DOC)
-                  .defineInternal(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
+                  .define(REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_PROP,
                                   INT,
                                   DEFAULT_REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES,
                                   atLeast(0),
                                   LOW,
                                   REMOTE_LOG_METADATA_CUSTOM_METADATA_MAX_BYTES_DOC)
-                  .defineInternal(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,
+                  .define(REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_PROP,

Review Comment:
   ok, since the intention of KAFKA-15294 is to make configs public which we are not ready to do yet, I think we could mark this PR draft not to get merged. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] tinaselenge commented on pull request #14266: KAFKA-15294: Publish remote storage configs

Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on PR #14266:
URL: https://github.com/apache/kafka/pull/14266#issuecomment-1688206105

   @divijvaidya @clolov thank you for the feedback. I believe I have addressed the comments and as discussed, the remote log manager configs are left as internal until the required implementation is complete. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org