[GitHub] merlimat commented on a change in pull request #1172: Improve retention and expiry documentation (WIP)

[GitBox <gi...@apache.org>]

merlimat commented on a change in pull request #1172: Improve retention and expiry documentation (WIP)
URL: https://github.com/apache/incubator-pulsar/pull/1172#discussion_r165829396
 
 

 ##########
 File path: site/docs/latest/getting-started/ConceptsAndArchitecture.md
 ##########
 @@ -266,13 +266,35 @@ In BookKeeper, *journal* files contain BookKeeper transaction logs. Before makin
 
 A future version of BookKeeper will support *non-persistent messaging* and thus multiple durability modes at the topic level. This will enable you to set the durability mode at the topic level, replacing the `persistent` in topic names with a `non-persistent` indicator.
 
+## Message retention and expiry
+
+By default, Pulsar message {% popover brokers %}:
+
+* immediately delete *all* messages that have been {% popover acknowledged %} by a {% popover consumer %}, and
+* [persistently store](#persistent-storage) all unacknowledged messages in a message backlog.
+
+Pulsar has two features, however, that enable you to override this default behavior:
+
+* Message **retention** enables you to store messages that have been acknowledged by a consumer
+* Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged
+
+{% include admonition.html type="info" content='All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](../../advanced/RetentionExpiry) admin documentation.' %}
+
+The diagram below illustrates both concepts:
+
+{% img /img/retention-expiry.png 80 %}
+
+With message retention, shown at the top, a retention policy applied to all topics in a {% popover namespace %} dicates that some messages are durably stored in Pulsar even though they've already been acknowledged. Acknowledged messages that are not covered by the retention policy are deleted.
+
+With message expiry, shown at the bottom, some messages are deleted, even though they haven't been acknowledged, because they've expired according to the TTL applied to the namespace (for example because a TTL of 5 minutes has been applied and the messages haven't been acknowledged but are 10 minutes old).
+
 ## Replication
 
 Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](../../admin/GeoReplication) in Pulsar enables you to do that.
 
 ## Multi-tenancy
 
-Pulsar was created from the ground up as a {% popover multi-tenant %} system. To support multi-tenancy, Pulsar has a concept of {% popover properties %}. Properties can be spread across {% popover clusters %} and can each have their own [authentication and authorization](../../admin/Authz) scheme applied to them. They are also the administrative unit at which [storage quotas](TODO), [message TTL](TODO), and [isolation policies](TODO) can be managed.
+Pulsar was created from the ground up as a {% popover multi-tenant %} system. To support multi-tenancy, Pulsar has a concept of {% popover properties %}. Properties can be spread across {% popover clusters %} and can each have their own [authentication and authorization](../../admin/Authz) scheme applied to them. They are also the administrative unit at which [storage quotas](TODO), [message TTL](../../advanced/RetentionExpiry#time-to-live-ttl), and [isolation policies](TODO) can be managed.
 
 The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure:
 
 
 Review comment:
   One comment on the diagram below, for message expiry, can we use a different color for the section "Within the applied TTL"? The reason is that, technically, the TTL is just auto-acknowledging messages that were not acked withing the configured time.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services