[GitHub] srkukarni commented on a change in pull request #1941: Tiered Storage documentation

[GitBox <gi...@apache.org>]

srkukarni commented on a change in pull request #1941: Tiered Storage documentation
URL: https://github.com/apache/incubator-pulsar/pull/1941#discussion_r194191655
 
 

 ##########
 File path: site/docs/latest/cookbooks/tiered-storage.md
 ##########
 @@ -0,0 +1,119 @@
+---
+title: Tiered Storage
+tags: [admin, tiered-storage]
+---
+
+Pulsar's Tiered Storage feature allows older backlog data to be offloaded to long term storage, thereby freeing up space in BookKeeper and reducing storage costs. 
+
+## When should I use Tiered Storage?
+
+Tiered storage should be used when you have a topic for which you want to keep a very long backlog for a long time. For example, if you have a topic containing user actions which you use to train your recommendation systems, you may want to keep that data for a long time, so that if you change your recommendation algorithm you can rerun it against your full user history.
+
+## The offloading mechanism
+
+A topic in Pulsar is backed by a log, known as a managed ledger. This log is composed of an ordered list of segments. Pulsar only every writes to the final segment of the log. All previous segments are sealed. The data within the segment is immutable. This is known as a segment oriented architecture.
+
+{% include figure.html src="/img/pulsar-tiered-storage.png" alt="Tiered Storage" width="80" %}
+
+The Tiered Storage offloading mechanism takes advantage of this segment oriented architecture. When offloading is requested, the segments of the log are copied, one-by-one, to tiered storage. All segments of the log, apart from the segment currently being written to can be offloaded.
+
+## Amazon S3
+
+Tiered storage currently supports S3 for long term storage. The broker must be configured with the S3 bucket into which the offloaded data will be placed and the AWS region where that bucket exists.
+
+The configured S3 bucket must exist before attempting to offload. If it does not offload, will fail.
+
+Pulsar users multipart objects to update the segment data. It is possible that a broker could crash while uploading the data. We recommend you add a lifecycle rule your S3 bucket to expire incomplete multipart upload after a day or two to avoid getting charged for incomplete uploads.
+
+### Configuring the broker
+
+Offloading is configured in ```broker.conf```. 
+
+At a minimum, the user must configure the driver, the region and the bucket.
+
+```conf
+managedLedgerOffloadDriver=S3
+s3ManagedLedgerOffloadRegion=eu-west-3
+s3ManagedLedgerOffloadBucket=pulsar-topic-offload
+```
+
+It is also possible to specify the s3 endpoint directly, using ```s3ManagedLedgerOffloadServiceEndpoint```. This is useful if you are using a none AWS storage service which provides an S3 compatible API. 
 
 Review comment:
   a none -> another

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services