You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "showuon (via GitHub)" <gi...@apache.org> on 2023/09/13 08:25:55 UTC
[GitHub] [kafka] showuon opened a new pull request, #14382: KAFKA-15442: add a section in doc for tiered storage
showuon opened a new pull request, #14382:
URL: https://github.com/apache/kafka/pull/14382
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1717567763
> Will the RemoteLogManagerConfig and TBRLMMConfig will be auto-exported in the doc config section?
Yes, that's been handled in another [PR](https://github.com/apache/kafka/pull/14266).
> Can we provide an example of creating a topic, uploading a segment, reading data from remote segment using the LocalTieredStorage as RSM and TBRLMM as RLMM?
That sounds good. But since we don't have much time left for v3.6.0 RC build, we can add them after release. Doc updates can directly update in kafka-site repo. I'll add them in a separate PR.
> Can we include the steps for the quick-start?
I don't have strong opinion on this question. IMO, we can at least put the quick start guide in the 6.11 section I added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1719338018
Failed tests are unrelated:
```
Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.integration.ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector
Build / JDK 11 and Scala 2.13 / kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.testCreateTokenForOtherUserFails(String).quorum=kraft
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kamalcph commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1717366033
My suggestion is to add one more section with LocalTieredStorage and TBRLMM, then add all the basic configurations that are required for quick start:
```properties
# Remote Storage Manager Config
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage
remote.log.storage.manager.class.path=
remote.log.storage.manager.impl.prefix=rsm.config.
rsm.config.dir=/tmp/kafka-remote-storage
# Remote Log Metadata Manager Config that uses Kafka topic to store metadata
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.class.path=
remote.log.metadata.manager.listener.name=PLAINTEXT
remote.log.metadata.manager.impl.prefix=rlmm.config.
rlmm.config.remote.log.metadata.topic.num.partitions=50
rlmm.config.remote.log.metadata.topic.replication.factor=3
rlmm.config.remote.log.metadata.topic.retention.ms=-1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tinaselenge commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1325541025
##########
docs/ops.html:
##########
@@ -3859,6 +3859,98 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage Overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.
+ Please check <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a> for more information.
+</p>
+
+<p><b>Note: Tiered storage is considered as an early access feature, and is not recommended for use in production environments</b></p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">Broker Configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ and <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> and <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">Topic Configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">Configurations Example</a></h4>
+
+<p>Here is a sample configuration to enable tiered storage feature in broker side:
+<pre>
+# Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092
+remote.log.storage.system.enable=true
+# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage.
+# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager
+# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers.
+remote.log.metadata.manager.listener.name=PLAINTEXT
+</pre>
+</p>
+
+<p>After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
+<pre>bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000
+</pre>
+</p>
+
+<p>Then, after the active segment is rolled, the old segment should be moved to the remote storage and get deleted.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_limitation" class="anchor-link"></a><a href="#tiered_storage_limitation">Limitations</a></h4>
+
+<p>While the early access release of Tiered Storage offers the opportunity to try out this new feature, it is important to be aware of the following limitations:
+<ul>
+ <li>No support for clusters with multiple log directories (i.e. JBOD feature)</li>
+ <li>No support for compacted topics</li>
+ <li>Cannot disable tiered storage at the topic level</li>
Review Comment:
Does this mean, once enabled for a topic, you cannot disable it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1718998104
@tinaselenge , thanks for the comments. PR updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1325394656
##########
docs/ops.html:
##########
@@ -3859,6 +3859,98 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage Overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.
+ Please check <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a> for more information.
+</p>
+
+<p><b>Note: Tiered storage is considered as an early access feature, and is not recommended for use in production environments</b></p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">Broker Configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ and <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> and <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">Topic Configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">Configurations Example</a></h4>
+
+<p>Here is a sample configuration to enable tiered storage feature in broker side:
+<pre>
+# Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092
+remote.log.storage.system.enable=true
+# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage.
+# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager
+# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers.
+remote.log.metadata.manager.listener.name=PLAINTEXT
+</pre>
+</p>
+
+<p>After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
+<pre>bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000
+</pre>
+</p>
+
+<p>Then, after the active segment is rolled, the old segment should be moved to the remote storage and get deleted.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_limitation" class="anchor-link"></a><a href="#tiered_storage_limitation">Limitation</a></h4>
Review Comment:
Good suggestion. Updated. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1324238229
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ or <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> or <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">topic configurations</a></h5>
Review Comment:
`topic configurations` -> `Topic Configurations`
Camel Case is followed for section headings in the docs page.
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ or <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> or <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">topic configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">configurations example</a></h4>
+
+<p>Here is a sample configuration to enable tiered storage feature in broker side:
+<pre>
+# Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092
+remote.log.storage.system.enable=true
+# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage.
+# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager
+# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers.
+remote.log.metadata.manager.listener.name=PLAINTEXT
+</pre>
+</p>
+
+<p>After broker is startup, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
+<pre>bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000
+</pre>
+</p>
+
+<p>Then, after the active segment is rolled, the old segment should be moved to remote storage and get deleted.
+ Checking the remote storage to verify it.
Review Comment:
`Checking the remote storage to verify it.`
What do you mean by it? Are we asking users to check the remote storage whether the segment is moved to remote storage or not?
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
Review Comment:
`broker configurations` -> `Broker Configurations`
Camel Case is followed for section headings in the docs page.
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ or <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> or <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">topic configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">configurations example</a></h4>
Review Comment:
`configurations example` -> `Configurations Example`
Camel Case is followed for section headings in the docs page.
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
Review Comment:
`Tiered Storage overview` -> `Tiered Storage Overview`
Camel Case is followed for section headings in the docs page.
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ or <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> or <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">topic configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">configurations example</a></h4>
+
+<p>Here is a sample configuration to enable tiered storage feature in broker side:
+<pre>
+# Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092
+remote.log.storage.system.enable=true
+# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage.
+# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager
+# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers.
+remote.log.metadata.manager.listener.name=PLAINTEXT
+</pre>
+</p>
+
+<p>After broker is startup, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
Review Comment:
After broker is startup -> After a broker is started
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1324263153
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
Review Comment:
1. `Configuring` -> `Configure`
2. <code>remote.log.storage.manager.class.name</code>
and <code>remote.log.storage.manager.class.path</code> instead of <code>remote.log.storage.manager.class.name</code>
or <code>remote.log.storage.manager.class.path</code>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kamalcph commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1717351303
@showuon
Thanks for drafting the initial version of the doc! Will the RemoteLogManagerConfig and TBRLMMConfig will be auto-exported in the doc config section?
Can we provide an example of creating a topic, uploading a segment, reading data from remote segment using the LocalTieredStorage as RSM and TBRLMM as RLMM?
LocalTieredStorage resides inside the `test` folder. To use it, the user has to build the `kafka-storage-<version>-test.jar` binary from source code and add that lib to the classpath. Then, enable `INCLUDE_TEST_JARS` to true via export. Can we include the steps for the quick-start?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tinaselenge commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1325541475
##########
docs/ops.html:
##########
@@ -3859,6 +3859,98 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage Overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.
+ Please check <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a> for more information.
+</p>
+
+<p><b>Note: Tiered storage is considered as an early access feature, and is not recommended for use in production environments</b></p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">Broker Configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ and <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> and <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">Topic Configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
Review Comment:
```suggestion
When tiered storage is enabled for a topic, there are 2 additional log retention configurations to set:
```
##########
docs/ops.html:
##########
@@ -3859,6 +3859,98 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage Overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.
+ Please check <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a> for more information.
+</p>
+
+<p><b>Note: Tiered storage is considered as an early access feature, and is not recommended for use in production environments</b></p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">Broker Configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ and <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> and <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">Topic Configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
Review Comment:
```suggestion
<code>remote.storage.enable</code> is the switch to determine if a topic wants to use tiered storage or not. By default it is set to false.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tinaselenge commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1325542037
##########
docs/ops.html:
##########
@@ -3859,6 +3859,98 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage Overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.
+ Please check <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a> for more information.
+</p>
+
+<p><b>Note: Tiered storage is considered as an early access feature, and is not recommended for use in production environments</b></p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">Broker Configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
Review Comment:
```suggestion
is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" enables this feature.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1324438818
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ or <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> or <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">topic configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">configurations example</a></h4>
+
+<p>Here is a sample configuration to enable tiered storage feature in broker side:
+<pre>
+# Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092
+remote.log.storage.system.enable=true
+# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage.
+# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager
+# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers.
+remote.log.metadata.manager.listener.name=PLAINTEXT
+</pre>
+</p>
+
+<p>After broker is startup, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
+<pre>bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000
+</pre>
+</p>
+
+<p>Then, after the active segment is rolled, the old segment should be moved to remote storage and get deleted.
+ Checking the remote storage to verify it.
Review Comment:
Oh second thought, I think we don't need this line. I'll delete it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1717185561
@satishd @kamalcph @divijvaidya @clolov , call for help to make the doc better. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1324263153
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
Review Comment:
`Configuring` -> `Configure`
<code>remote.log.storage.manager.class.name</code>
and <code>remote.log.storage.manager.class.path</code> instead of <code>remote.log.storage.manager.class.name</code>
or <code>remote.log.storage.manager.class.path</code>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1325573311
##########
docs/ops.html:
##########
@@ -3859,6 +3859,98 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage Overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.
+ Please check <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a> for more information.
+</p>
+
+<p><b>Note: Tiered storage is considered as an early access feature, and is not recommended for use in production environments</b></p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">Broker Configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ and <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> and <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">Topic Configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">Configurations Example</a></h4>
+
+<p>Here is a sample configuration to enable tiered storage feature in broker side:
+<pre>
+# Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092
+remote.log.storage.system.enable=true
+# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage.
+# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager
+# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers.
+remote.log.metadata.manager.listener.name=PLAINTEXT
+</pre>
+</p>
+
+<p>After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
+<pre>bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000
+</pre>
+</p>
+
+<p>Then, after the active segment is rolled, the old segment should be moved to the remote storage and get deleted.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_limitation" class="anchor-link"></a><a href="#tiered_storage_limitation">Limitations</a></h4>
+
+<p>While the early access release of Tiered Storage offers the opportunity to try out this new feature, it is important to be aware of the following limitations:
+<ul>
+ <li>No support for clusters with multiple log directories (i.e. JBOD feature)</li>
+ <li>No support for compacted topics</li>
+ <li>Cannot disable tiered storage at the topic level</li>
Review Comment:
Yes, because we don't have a better way now to handle the data in the remote storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1717671724
>That sounds good. But since we don't have much time left for v3.6.0 RC build, we can add them after release. Doc updates can directly update in kafka-site repo. I'll add them in a separate PR.
+1 on that. We can improve the doc in a followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1717677707
>That sounds good. But since we don't have much time left for v3.6.0 RC build, we can add them after release. Doc updates can directly update in kafka-site repo. I'll add them in a separate PR.
+1 on that. We can improve the doc in a followup.
@kamalcph @showuon We can add quick start steps in the docs later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tinaselenge commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "tinaselenge (via GitHub)" <gi...@apache.org>.
tinaselenge commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1718963224
Left very minor comments which I'm happy if you want to address in subsequent PR. LGTM otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #14382:
URL: https://github.com/apache/kafka/pull/14382#issuecomment-1718792709
@satishd , in the latest commit, I added:
1. a note to say this is an early access feature and should not use in production env.
2. a limitation section to describe the limitation in v3.6.0
3. a notable change entry in v3.6.0
The screenshot is updated in the PR description. Please take a look again. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1324262175
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ or <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> or <code>remote.log.metadata.manager.class.path</code>.
Review Comment:
<code>remote.log.metadata.manager.class.name</code> and <code>remote.log.metadata.manager.class.path</code> instead of <code>remote.log.metadata.manager.class.name</code> or <code>remote.log.metadata.manager.class.path</code>
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
Review Comment:
<code>remote.log.storage.manager.class.name</code>
and <code>remote.log.storage.manager.class.path</code> instead of <code>remote.log.storage.manager.class.name</code>
or <code>remote.log.storage.manager.class.path</code>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon merged pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon merged PR #14382:
URL: https://github.com/apache/kafka/pull/14382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1324437600
##########
docs/ops.html:
##########
@@ -3859,6 +3859,80 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">broker configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ or <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> or <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">topic configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">configurations example</a></h4>
+
+<p>Here is a sample configuration to enable tiered storage feature in broker side:
+<pre>
+# Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092
+remote.log.storage.system.enable=true
+# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage.
+# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager
+# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers.
+remote.log.metadata.manager.listener.name=PLAINTEXT
+</pre>
+</p>
+
+<p>After broker is startup, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
+<pre>bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000
+</pre>
+</p>
+
+<p>Then, after the active segment is rolled, the old segment should be moved to remote storage and get deleted.
+ Checking the remote storage to verify it.
Review Comment:
Yes, that's my thought. Let me rephrase it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1325382836
##########
docs/ops.html:
##########
@@ -3859,6 +3859,98 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage Overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.
+ Please check <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a> for more information.
+</p>
+
+<p><b>Note: Tiered storage is considered as an early access feature, and is not recommended for use in production environments</b></p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">Broker Configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ and <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> and <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">Topic Configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">Configurations Example</a></h4>
+
+<p>Here is a sample configuration to enable tiered storage feature in broker side:
+<pre>
+# Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092
+remote.log.storage.system.enable=true
+# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage.
+# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager
+# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers.
+remote.log.metadata.manager.listener.name=PLAINTEXT
+</pre>
+</p>
+
+<p>After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
+<pre>bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000
+</pre>
+</p>
+
+<p>Then, after the active segment is rolled, the old segment should be moved to the remote storage and get deleted.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_limitation" class="anchor-link"></a><a href="#tiered_storage_limitation">Limitation</a></h4>
Review Comment:
Please change `Limitation` -> `Limitations`
Both in the text and the links.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #14382: KAFKA-15442: add a section in doc for tiered storage
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14382:
URL: https://github.com/apache/kafka/pull/14382#discussion_r1325382836
##########
docs/ops.html:
##########
@@ -3859,6 +3859,98 @@ <h3>Finalizing the migration</h3>
# Other configs ...</pre>
+
+<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>
+
+<h4 class="anchor-heading"><a id="tiered_storage_overview" class="anchor-link"></a><a href="#tiered_storage_overview">Tiered Storage Overview</a></h4>
+
+<p>Kafka data is mostly consumed in a streaming fashion using tail reads. Tail reads leverage OS's page cache to serve the data instead of disk reads.
+ Older data is typically read from the disk for backfill or failure recovery purposes and is infrequent.</p>
+
+<p>In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote.
+ The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments.
+ The new remote tier uses external storage systems, such as HDFS or S3, to store the completed log segments.
+ Please check <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a> for more information.
+</p>
+
+<p><b>Note: Tiered storage is considered as an early access feature, and is not recommended for use in production environments</b></p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config" class="anchor-link"></a><a href="#tiered_storage_config">Configuration</a></h4>
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_broker" class="anchor-link"></a><a href="#tiered_storage_config_broker">Broker Configurations</a></h5>
+
+<p>By default, Kafka server will not enable tiered storage feature. <code>remote.log.storage.system.enable</code>
+ is the property to control whether to enable tiered storage functionality in a broker or not. Setting it to "true" to enable this feature.
+</p>
+
+<p><code>RemoteStorageManager</code> is an interface to provide the lifecycle of remote log segments and indexes. Kafka server
+ doesn't provide out-of-the-box implementation of RemoteStorageManager. Configuring <code>remote.log.storage.manager.class.name</code>
+ and <code>remote.log.storage.manager.class.path</code> to specify the implementation of RemoteStorageManager.
+</p>
+
+<p><code>RemoteLogMetadataManager</code> is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics.
+ By default, Kafka provides an implementation with storage as an internal topic. This implementation can be changed by configuring
+ <code>remote.log.metadata.manager.class.name</code> and <code>remote.log.metadata.manager.class.path</code>.
+ When adopting the default kafka internal topic based implementation, <code>remote.log.metadata.manager.listener.name</code>
+ is a mandatory property to specify which listener the clients created by the default RemoteLogMetadataManager implementation.
+</p>
+
+
+<h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-link"></a><a href="#tiered_storage_config_topic">Topic Configurations</a></h5>
+
+<p>After correctly configuring broker side configurations for tiered storage feature, there are still configurations in topic level needed to be set.
+ <code>remote.storage.enable</code> is the switch to determine if this topic want to use tiered storage or not. By default it is set as false.
+ After enabling <code>remote.storage.enable</code> property, the next thing to consider is the log retention.
+ When tiered storage is enabled in the topic, there will be 2 additional log retention configuration to set:
+
+<ul>
+ <li><code>local.retention.ms</code></li>
+ <li><code>retention.ms</code></li>
+ <li><code>local.retention.bytes</code></li>
+ <li><code>retention.bytes</code></li>
+</ul>
+
+ The configuration prefixed with <code>local</code> are to specify the time/size the "local" log file can accept before moving to remote storage, and then get deleted.
+ If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">Configurations Example</a></h4>
+
+<p>Here is a sample configuration to enable tiered storage feature in broker side:
+<pre>
+# Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092
+remote.log.storage.system.enable=true
+# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage.
+# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager
+# Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers.
+remote.log.metadata.manager.listener.name=PLAINTEXT
+</pre>
+</p>
+
+<p>After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
+<pre>bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000
+</pre>
+</p>
+
+<p>Then, after the active segment is rolled, the old segment should be moved to the remote storage and get deleted.
+</p>
+
+<h4 class="anchor-heading"><a id="tiered_storage_limitation" class="anchor-link"></a><a href="#tiered_storage_limitation">Limitation</a></h4>
Review Comment:
Please change `Limitation` -> `Limitations`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org