[incubator-pulsar] branch master updated: Compaction documentation follow-up (#1871)

[si...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
     new 12864e8  Compaction documentation follow-up (#1871)
12864e8 is described below

commit 12864e8a3dd95977aef406a6499dc30bfc5432c0
Author: Luc Perkins <lu...@gmail.com>
AuthorDate: Wed Jun 20 16:39:59 2018 -0700

    Compaction documentation follow-up (#1871)
    
    * clarification around compacted vs non-compacted consumers
    
    * clarify that no-key msgs are ignored
    
    * theoretical clarifications
    
    * clarify language around listening/changes
---
 site/docs/latest/cookbooks/compaction.md                    | 13 ++++++++++---
 site/docs/latest/getting-started/ConceptsAndArchitecture.md | 10 ++++------
 2 files changed, 14 insertions(+), 9 deletions(-)

diff --git a/site/docs/latest/cookbooks/compaction.md b/site/docs/latest/cookbooks/compaction.md
index 3b11400..035dfd4 100644
--- a/site/docs/latest/cookbooks/compaction.md
+++ b/site/docs/latest/cookbooks/compaction.md
@@ -11,13 +11,18 @@ To use compaction:
 * Compaction can be configured to run [automatically](#automatic), or you can manually [trigger](#trigger) compaction using the Pulsar administrative API.
 * Your {% popover consumers %} must be [configured](#config) to read from compacted topics ([Java consumers](#java), for example, have a `readCompacted` setting that must be set to `true`). If this configuration is not set, consumers will still be able to read from the non-compacted topic.
 
+{% include admonition.html type="warning" content="Compaction only works on messages that have keys (as in the stock ticker example the stock symbol serves as the key for each message). Keys can thus be thought of as the axis along which compaction is applied. Messages that don't have keys are simply ignored by compaction." %}
+
 ## When should I use compacted topics? {#when}
 
-The classic example of a topic that could benefit from compaction would be a stock ticker topic through which {% popover consumers %} can access up-to-date values for specific stocks. On a stock ticker topic you only care about the most recent value of each stock; "historical values" don't matter, so there's no need to read through outdated data when processing a topic's messages.
+The classic example of a topic that could benefit from compaction would be a stock ticker topic through which {% popover consumers %} can access up-to-date values for specific stocks. Imagine a scneario in which messages carrying stock value data use the stock symbol as the key (`GOOG`, `AAPL`, `TWTR`, etc.). Compacting this topic would give consumers on the topic two options:
+
+* They can read from the "original," non-compacted topic in case they need access to "historical" values, i.e. the entirety of the topic's messages.
+* They can read from the compacted topic if they only want to see the most up-to-date messages.
 
-{% include admonition.html type="info" content="For topics where older values are important, for example when you need to process a long series of messages in order, many of which have the same key, compaction is unnecessary and could possibly even be harmful." %}
+Thus, if you're using a Pulsar topic called `stock-values`, some consumers could have access to all messages in the topic (perhaps because they're performing some kind of number crunching of all values in the last hour) while the consumers used to power the real-time stock ticker only see the compacted topic (and thus aren't forced to process outdated messages). Which variant of the topic any given consumer pulls messages from is determined by the consumer's [configuration](#config).
 
-{% include admonition.html type="warning" content="Compaction only works on topics where each message has a key (as in the stock ticker example, where the stock symbol serves as the key). Keys can be thought of as the axis along which compaction is applied." %}
+{% include admonition.html type="success" content="One of the benefits of compaction in Pulsar is that you aren't forced to choose between compacted and non-compacted topics, as the compaction process leaves the original topic as-is and essentially adds an alternate topic. In other words, you can run compaction on a topic and consumers that need access to the non-compacted version of the topic will not be adversely affected." %}
 
 ## Configuring compaction to run automatically {#automatic}
 
@@ -48,6 +53,8 @@ $ bin/pulsar compact-topic \
   --topic persistent://my-tenant-namespace/my-topic
 ```
 
+{% include admonition.html type="info" content="Running compaction in its own process is recommended when you want to avoid interfering with the broker's performance. Broker performance should only be affected, however, when running compaction on topics with a large keyspace (i.e when there are many keys on the topic). The first phase of the compaction process keeps a copy of each key in the topic, which can create memory pressure as the number of keys grows. Using the `pulsar-admin topi [...]
+
 The `pulsar compact-topic` command communicates with [ZooKeeper](https://zookeeper.apache.org) directly. In order to establish communication with ZooKeeper, though, the `pulsar` CLI tool will need to have a valid [broker configuration](../../Configuration#broker). You can either supply a proper configuration in `conf/broker.conf` or specify a non-default location for the configuration:
 
 ```bash
diff --git a/site/docs/latest/getting-started/ConceptsAndArchitecture.md b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
index 96c0be2..e07c5f9 100644
--- a/site/docs/latest/getting-started/ConceptsAndArchitecture.md
+++ b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
@@ -551,7 +551,7 @@ For some use cases consumers don't need a complete "image" of the topic log. The
 
 Pulsar's topic compaction feature:
 
-* Allos for much more efficient "rewind" through topic logs
+* Allows for faster "rewind" through topic logs
 * Applies only to [persistent topics](#persistent-storage)
 * Triggered automatically when the backlog reaches a certain size or can be triggered manually via the command line. See the [Topic compaction cookbook](../../cookbooks/compaction)
 * Is conceptually and operationally distinct from [retention and expiry](#message-retention-and-expiry). Topic compaction *does*, however, respect retention. If retention has removed a message from the message backlog of a topic, the message will also not be readable from the compacted topic ledger.
@@ -561,13 +561,11 @@ Pulsar's topic compaction feature:
 
 ### How topic compaction works
 
-When topic compaction is triggered [via the CLI](../../cookbooks/compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the {% popover broker %} responsible will keep a record of the latest occurrence of that key. When this iterative process is finished, the broker will create a [BookKeeper ledger](#ledgers) to store the compacted topic.
+When topic compaction is triggered [via the CLI](../../cookbooks/compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key.
 
-After that, the broker will make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the new BookKeeper ledger (the one that was created when compaction was manually initiated). If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered delet [...]
+After that, the broker will create a new [BookKeeper ledger](#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the conc [...]
 
-{% include admonition.html type="info" title="Compaction leaves the original topic intact" %}
-
-In addition to performing compaction, Pulsar {% popover brokers %} listen for changes on each topic's metadata. If the ledger for the topic changes:
+After the initial compaction operation, the Pulsar {% popover broker %} that owns the topic is notified whenever any future changes are made to the compaction horizon and compacted backlog. When such changes occur:
 
 * Clients (consumers and readers) that have read compacted enabled will attempt to read messages from a topic and either:
   * Read from the topic like normal (if the message ID is greater than or equal to the compaction horizon) or