You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/08/12 09:33:24 UTC

[GitHub] [druid] petermarshallio opened a new pull request #11584: Docs - query caching

petermarshallio opened a new pull request #11584:
URL: https://github.com/apache/druid/pull/11584


   OTBO https://the-asf.slack.com/archives/CJ8D1JTB8/p1597781107153900
   
   - Expands on the information about segment-level caching on historicals with links to config lines
   - Emphasises that cache invalidation strategy and why this matters to which cache you use
   - Calls out strategies as quote boxes
   
   @techdocsmith this PR can be the starting point for us to work on how to arrange this info - as discussed.
   
   This PR has:
   - [X] been self-reviewed.
   - [ ] been tested in a test Druid cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] petermarshallio commented on a change in pull request #11584: Docs - query caching

Posted by GitBox <gi...@apache.org>.

petermarshallio commented on a change in pull request #11584:
URL: https://github.com/apache/druid/pull/11584#discussion_r755062147



##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.
+
+It allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html) as instructed by the lead [coordinator](../design/coordinator.html).  Meanwhile, real-time segments, on the other hand, continue to have results computed at query time.
 
-When your queries include data from segments that are mutable and undergoing real-time ingestion, use a segment cache. In this case Druid caches query results for immutable historical segments when possible. It re-computes results for the real-time segments at query time.
+Per-segment cached results also have the potential to be merged into the results of later queries where there is a similar basic shape (filters, aggregations, etc.) yet cover a different period of time, for example.
 
-For example, you have queries that frequently include incoming data from a Kafka or Kinesis stream alongside unchanging segments. Per-segment caching lets Druid cache results from older immutable segments and merge them with updated data. Whole-query caching would not be helpful in this scenario because the new data from real-time ingestion continually invalidates the cache.
+Per-segment caching is controlled by the parameters `useCache` and `populateCache`.

Review comment:
       I've moved this into the Using Caching page.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] techdocsmith commented on a change in pull request #11584: Docs - query caching

Posted by GitBox <gi...@apache.org>.

techdocsmith commented on a change in pull request #11584:
URL: https://github.com/apache/druid/pull/11584#discussion_r832742802



##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies or "pulls" segment files from Deep Storage to local disk in an area called the *segment cache*.  Set the `druid.segmentCache.locations` to configure the size and location of the segment cache on each Historical process. See [Historical general configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+See the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size) for more information.
 
-When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.md). Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.
+The [Coordinator](../design/coordinator.html) controls the assignment of segments to Historicals and the balance of segments between Historicals. Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral entries in Zookeeper in a [load queue path](../configuration/index.html#path-configuration). Each Historical process maintains a connection to Zookeeper, watching those paths for segment information.
+
+For more information about how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.html).

Review comment:
       ```suggestion
   For more information about how the Coordinator assigns segments to Historical processes, see [Coordinator](../design/coordinator.html).
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] techdocsmith commented on a change in pull request #11584: Docs - query caching

Posted by GitBox <gi...@apache.org>.

techdocsmith commented on a change in pull request #11584:
URL: https://github.com/apache/druid/pull/11584#discussion_r720581464



##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+For more information on tuning this value, see the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size).
 
-When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.md). Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.
+The [Coordinator](../design/coordinator.html) leads the assignment of segments to - and balance between - Historical processes.  [Zookeeper](../dependencies/zookeeper.md) is central to this collaboration; Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral Zookeeper entries under a [load queue path](../configuration/index.html#path-configuration) and each Historical process maintains a connection to Zookeeper, watching those paths for segment information.

Review comment:
       ```suggestion
   The [Coordinator](../design/coordinator.html) controls the assignment of segments to Historicals and the balance of segments between Historicals. Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral entries in Zookeeper in a [load queue path](../configuration/index.html#path-configuration). Each Historical process maintains a connection to Zookeeper, watching those paths for segment information.
   ```
   What does this knowledge of the internals of segment assignment gain me. What am I supposed to do with this knowledge?

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.

Review comment:
       ```suggestion
   The primary form of caching in Druid is a *per-segment results cache*.  This cache stores partial query results on a per-segment basis and is enabled on Historical services by default.
   ```
   nit: avoid "this" as a pronoun

##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+For more information on tuning this value, see the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size).
 
-When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.md). Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.
+The [Coordinator](../design/coordinator.html) leads the assignment of segments to - and balance between - Historical processes.  [Zookeeper](../dependencies/zookeeper.md) is central to this collaboration; Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral Zookeeper entries under a [load queue path](../configuration/index.html#path-configuration) and each Historical process maintains a connection to Zookeeper, watching those paths for segment information.
+
+For more information about how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.html).
+
+When a Historical process sees a new Zookeeper load queue entry, it checks its own segment cache. If no information about the segment exists there, the Historical process first retrieves metadata from Zookeeper about the segment, including where the segment is located in Deep Storage and how it needs to decompress and process it.
+
+For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.html). 
+
+Once a Historical process has completed pulling down and processing a segment from Deep Storage, the segment is advertised as being available for queries.  This announcement by the Historical is made via Zookeeper, this time under a [served segments path](../configuration/index.html#path-configuration). At this point, the segment is considered available for querying by the Broker.

Review comment:
       ```suggestion
   After a Historical process pulls down and processes a segment from Deep Storage, Druid advertises the segment as being available for queries from the Broker.  This announcement by the Historical is made via Zookeeper, in a [served segments path](../configuration/index.html#path-configuration).
   ```
   Why do I need to know that the announcement is going through Zookeper? 

##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+For more information on tuning this value, see the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size).
 
-When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.md). Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.
+The [Coordinator](../design/coordinator.html) leads the assignment of segments to - and balance between - Historical processes.  [Zookeeper](../dependencies/zookeeper.md) is central to this collaboration; Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral Zookeeper entries under a [load queue path](../configuration/index.html#path-configuration) and each Historical process maintains a connection to Zookeeper, watching those paths for segment information.
+
+For more information about how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.html).
+
+When a Historical process sees a new Zookeeper load queue entry, it checks its own segment cache. If no information about the segment exists there, the Historical process first retrieves metadata from Zookeeper about the segment, including where the segment is located in Deep Storage and how it needs to decompress and process it.
+
+For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.html). 
+
+Once a Historical process has completed pulling down and processing a segment from Deep Storage, the segment is advertised as being available for queries.  This announcement by the Historical is made via Zookeeper, this time under a [served segments path](../configuration/index.html#path-configuration). At this point, the segment is considered available for querying by the Broker.
+
+For more information about how the Broker determines what data is available for queries, please see [Broker](broker.html).
+
+On startup, a Historical process searches through its segment cache and, in order for Historicals to be queried as soon as possible, immediately advertises all segments it finds there.

Review comment:
       ```suggestion
   To make data from the segment cache available for querying as soon as possible, Historical services search the local segment cache upon startup and advertise the segments found there.
   ```

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.

Review comment:
       Druid can store cache data on the local JVM heap or in an external distributed key/value store. For example memcached. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage. 
   
   The caffeine-based cache resides within the JVM heap and is directly measurable.  Heap usage grows up to the maximum configured size. After that Druid flushes the least recently used segment results and replaces them with newer results.
   ```
   I think of flushing from a cache not "evicting" maybe I'm confused.

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.

Review comment:
       ```suggestion
   - [Whole-query caching](#whole-query-caching) stores final query results.
   ```
   avoid italics for emphasis. Not sure we need emphasis here.

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.

Review comment:
       ```suggestion
   Druid invalidates any cache the moment any underlying data change to avoid returning stale results. This is especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
   ```
   This is not an admonition. It's documentation of how the feature works. If we need to influence customer behavior, it may be worth it to reiterate it elsewhere.

##########
File path: docs/operations/basic-cluster-tuning.md
##########
@@ -90,11 +90,9 @@ Tuning the cluster so that each Historical can accept 50 queries and 10 non-quer
 
 #### Segment Cache Size
 
-`druid.segmentCache.locations` specifies locations where segment data can be stored on the Historical. The sum of available disk space across these locations is set as the default value for property: `druid.server.maxSize`, which controls the total size of segment data that can be assigned by the Coordinator to a Historical.
+Avoid allocating a Historical an excessive amount of segment data.  As the value of (`free system memory` / total size of all `druid.segmentCache.locations`) increases, a greater proportion of segments can be kept in the [memory-mapped segment cache](../design/historical.md#loading-and-serving-segments-from-cache), allowing for better query performance.

Review comment:
       ```suggestion
   For better query performance, do not allocate segment data to a Historical in excess of the system free memory.  When `free system memory` is greater than or equal to `druid.segmentCache.locations`, the more segment data the Historical can be hold in the memory-mapped segment cache.
   ```

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.
+
+It allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html) as instructed by the lead [coordinator](../design/coordinator.html).  Meanwhile, real-time segments, on the other hand, continue to have results computed at query time.

Review comment:
       ```suggestion
   The per-segment results cache allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html). Real-time segments, on the other hand, continue to have results computed at query time.
   ```
   Why does it matter that the Coordinator instructs them to do this? How do I use that information in a practical sense?

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.
+
+It allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html) as instructed by the lead [coordinator](../design/coordinator.html).  Meanwhile, real-time segments, on the other hand, continue to have results computed at query time.
 
-When your queries include data from segments that are mutable and undergoing real-time ingestion, use a segment cache. In this case Druid caches query results for immutable historical segments when possible. It re-computes results for the real-time segments at query time.
+Per-segment cached results also have the potential to be merged into the results of later queries where there is a similar basic shape (filters, aggregations, etc.) yet cover a different period of time, for example.
 
-For example, you have queries that frequently include incoming data from a Kafka or Kinesis stream alongside unchanging segments. Per-segment caching lets Druid cache results from older immutable segments and merge them with updated data. Whole-query caching would not be helpful in this scenario because the new data from real-time ingestion continually invalidates the cache.
+Per-segment caching is controlled by the parameters `useCache` and `populateCache`.

Review comment:
       Why do we need this here? This should be covered in the how to section.

##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+For more information on tuning this value, see the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size).
 
-When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.md). Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.
+The [Coordinator](../design/coordinator.html) leads the assignment of segments to - and balance between - Historical processes.  [Zookeeper](../dependencies/zookeeper.md) is central to this collaboration; Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral Zookeeper entries under a [load queue path](../configuration/index.html#path-configuration) and each Historical process maintains a connection to Zookeeper, watching those paths for segment information.
+
+For more information about how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.html).
+
+When a Historical process sees a new Zookeeper load queue entry, it checks its own segment cache. If no information about the segment exists there, the Historical process first retrieves metadata from Zookeeper about the segment, including where the segment is located in Deep Storage and how it needs to decompress and process it.
+
+For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.html). 
+
+Once a Historical process has completed pulling down and processing a segment from Deep Storage, the segment is advertised as being available for queries.  This announcement by the Historical is made via Zookeeper, this time under a [served segments path](../configuration/index.html#path-configuration). At this point, the segment is considered available for querying by the Broker.
+
+For more information about how the Broker determines what data is available for queries, please see [Broker](broker.html).
+
+On startup, a Historical process searches through its segment cache and, in order for Historicals to be queried as soon as possible, immediately advertises all segments it finds there.
 
 ### Loading and serving segments from cache
 
-Recall that when a Historical process notices a new segment entry in its load queue path, the Historical process first checks a configurable cache directory on its local disk to see if the segment had been previously downloaded. If a local cache entry already exists, the Historical process will directly read the segment binary files from disk and load the segment.
+A technique called [memory mapping](https://en.wikipedia.org/wiki/Mmap) is used for the segment cache, consuming memory from the underlying operating system so that parts of segment files can be held in memory, increasing query performance at the data level.  The in-memory segment cache is therefore affected by, for example, the size of the Historical JVM, heap / direct memory buffers, and other processes on the operating system itself.
+
+At query time, if the required part of a segment file is available in the memory mapped cache (also known as the "page cache"), it will be re-used and read directly from memory.  If it is not, that part of the segment will be read from disk.  When this happens, there is potential for this new data to evict other segment data from memory.  Consequently, the closer that free operating system memory is to `druid.server.maxSize`, the faster historical processes typically respond at query time since segment data is very likely to be available in memory.  Conversely, the lower the free operating system memory, the more likely a Historical is to read segments from disk.

Review comment:
       ```suggestion
   At query time, if the required part of a segment file is available in the memory mapped cache or "page cache", the Historical re-uses it and reads it directly from memory.  If it is not in the memory-mapped cache, the Historical reads that part of the segment from disk. In this case, there is potential for new data to flush other segment data from memory. This means that if free operating system memory is close to `druid.server.maxSize`, the more likely that segment data will be available in memory and reduce query times.  Conversely, the lower the free operating system memory, the more likely a Historical is to read segments from disk.
   ```
   This seems like the most important point to me. Like it should be in some sort of operational best practice. For example I want to know when I'm setting up my system that I want to be able to store as much of the data in memory as possible to make queries fast.

##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+For more information on tuning this value, see the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size).
 
-When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.md). Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.
+The [Coordinator](../design/coordinator.html) leads the assignment of segments to - and balance between - Historical processes.  [Zookeeper](../dependencies/zookeeper.md) is central to this collaboration; Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral Zookeeper entries under a [load queue path](../configuration/index.html#path-configuration) and each Historical process maintains a connection to Zookeeper, watching those paths for segment information.
+
+For more information about how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.html).
+
+When a Historical process sees a new Zookeeper load queue entry, it checks its own segment cache. If no information about the segment exists there, the Historical process first retrieves metadata from Zookeeper about the segment, including where the segment is located in Deep Storage and how it needs to decompress and process it.
+
+For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.html). 
+
+Once a Historical process has completed pulling down and processing a segment from Deep Storage, the segment is advertised as being available for queries.  This announcement by the Historical is made via Zookeeper, this time under a [served segments path](../configuration/index.html#path-configuration). At this point, the segment is considered available for querying by the Broker.
+
+For more information about how the Broker determines what data is available for queries, please see [Broker](broker.html).
+
+On startup, a Historical process searches through its segment cache and, in order for Historicals to be queried as soon as possible, immediately advertises all segments it finds there.
 
 ### Loading and serving segments from cache
 
-Recall that when a Historical process notices a new segment entry in its load queue path, the Historical process first checks a configurable cache directory on its local disk to see if the segment had been previously downloaded. If a local cache entry already exists, the Historical process will directly read the segment binary files from disk and load the segment.
+A technique called [memory mapping](https://en.wikipedia.org/wiki/Mmap) is used for the segment cache, consuming memory from the underlying operating system so that parts of segment files can be held in memory, increasing query performance at the data level.  The in-memory segment cache is therefore affected by, for example, the size of the Historical JVM, heap / direct memory buffers, and other processes on the operating system itself.

Review comment:
       ```suggestion
   The segment cache uses [memory mapping](https://en.wikipedia.org/wiki/Mmap). The cache consumes memory from the underlying operating system so Historicals can hold parts of segment files in memory to increase query performance at the data level.  The in-memory segment cache is affected by the size of the Historical JVM, heap / direct memory buffers, and other processes on the operating system itself.
   ```

##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+For more information on tuning this value, see the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size).

Review comment:
       ```suggestion
   See the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size) for more information.
   ```

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.
+
+It allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html) as instructed by the lead [coordinator](../design/coordinator.html).  Meanwhile, real-time segments, on the other hand, continue to have results computed at query time.
 
-When your queries include data from segments that are mutable and undergoing real-time ingestion, use a segment cache. In this case Druid caches query results for immutable historical segments when possible. It re-computes results for the real-time segments at query time.
+Per-segment cached results also have the potential to be merged into the results of later queries where there is a similar basic shape (filters, aggregations, etc.) yet cover a different period of time, for example.
 
-For example, you have queries that frequently include incoming data from a Kafka or Kinesis stream alongside unchanging segments. Per-segment caching lets Druid cache results from older immutable segments and merge them with updated data. Whole-query caching would not be helpful in this scenario because the new data from real-time ingestion continually invalidates the cache.
+Per-segment caching is controlled by the parameters `useCache` and `populateCache`.
+
+> **Use per-segment caching with real-time data**
+>
+> For example, you could have queries that address fresh data arriving from Kafka, and which additionally covers intervals in segments that are loaded on historicals.  Per-segment caching allows Druid to cache results from the historical segments confidently, and to merge them with real-time results from the stream.  [Whole-query caching](#whole-query-caching), on the other hand, would not be helpful in this scenario because new data from real-time ingestion will continually invalidate the _entire_ result.
 
 ### Whole-query caching
 
-If real-time ingestion invalidating the cache is not an issue for your queries, you can use **whole-query caching** on the Broker to increase query efficiency. The Broker performs whole-query caching operations before sending fan out queries to Historicals. Therefore Druid no longer needs to merge the per-segment results on the Broker.
+Here, entire results of individual queries are cached, meaning Druid no longer needs to merge the per-segment results on the Broker.
+
+Whole-query result caching is controlled by the parameters `useResultLevelCache` and `populateResultLevelCache` and runtime properties `druid.broker.cache.*`.

Review comment:
       This should be covered in the "how to" "Using caching.

##########
File path: docs/operations/basic-cluster-tuning.md
##########
@@ -90,11 +90,9 @@ Tuning the cluster so that each Historical can accept 50 queries and 10 non-quer
 
 #### Segment Cache Size
 
-`druid.segmentCache.locations` specifies locations where segment data can be stored on the Historical. The sum of available disk space across these locations is set as the default value for property: `druid.server.maxSize`, which controls the total size of segment data that can be assigned by the Coordinator to a Historical.
+Avoid allocating a Historical an excessive amount of segment data.  As the value of (`free system memory` / total size of all `druid.segmentCache.locations`) increases, a greater proportion of segments can be kept in the [memory-mapped segment cache](../design/historical.md#loading-and-serving-segments-from-cache), allowing for better query performance.
 
-Segments are memory-mapped by Historical processes using any available free system memory (i.e., memory not used by the Historical JVM and heap/direct memory buffers or other processes on the system). Segments that are not currently in memory will be paged from disk when queried.
-
-Therefore, the size of cache locations set within `druid.segmentCache.locations` should be such that a Historical is not allocated an excessive amount of segment data. As the value of (`free system memory` / total size of all `druid.segmentCache.locations`) increases, a greater proportion of segments can be kept in memory, allowing for better query performance. The total segment data size assigned to a Historical can be overridden with `druid.server.maxSize`, but this is not required for most of the use cases.
+The total segment data size assigned to a Historical is calculated using `druid.segmentCache.locations` but can be overridden with `druid.server.maxSize`.  This is not required for most of the use cases.

Review comment:
       ```suggestion
   Druid uses the `druid.segmentCache.locations` to calculate the total segment data size assigned to a Historical. For some rarer use cases, you can override this behavior with `druid.server.maxSize` property.
   ```

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.
+
+It allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html) as instructed by the lead [coordinator](../design/coordinator.html).  Meanwhile, real-time segments, on the other hand, continue to have results computed at query time.
 
-When your queries include data from segments that are mutable and undergoing real-time ingestion, use a segment cache. In this case Druid caches query results for immutable historical segments when possible. It re-computes results for the real-time segments at query time.
+Per-segment cached results also have the potential to be merged into the results of later queries where there is a similar basic shape (filters, aggregations, etc.) yet cover a different period of time, for example.

Review comment:
       ```suggestion
   Druid may potentially merge per-segment cached results with the results of later queries that use a similar basic shape with similar filters, aggregations, etc. For example, if the query is identical except that it covers a different time period.

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.
+
+It allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html) as instructed by the lead [coordinator](../design/coordinator.html).  Meanwhile, real-time segments, on the other hand, continue to have results computed at query time.
 
-When your queries include data from segments that are mutable and undergoing real-time ingestion, use a segment cache. In this case Druid caches query results for immutable historical segments when possible. It re-computes results for the real-time segments at query time.
+Per-segment cached results also have the potential to be merged into the results of later queries where there is a similar basic shape (filters, aggregations, etc.) yet cover a different period of time, for example.
 
-For example, you have queries that frequently include incoming data from a Kafka or Kinesis stream alongside unchanging segments. Per-segment caching lets Druid cache results from older immutable segments and merge them with updated data. Whole-query caching would not be helpful in this scenario because the new data from real-time ingestion continually invalidates the cache.
+Per-segment caching is controlled by the parameters `useCache` and `populateCache`.
+
+> **Use per-segment caching with real-time data**
+>
+> For example, you could have queries that address fresh data arriving from Kafka, and which additionally covers intervals in segments that are loaded on historicals.  Per-segment caching allows Druid to cache results from the historical segments confidently, and to merge them with real-time results from the stream.  [Whole-query caching](#whole-query-caching), on the other hand, would not be helpful in this scenario because new data from real-time ingestion will continually invalidate the _entire_ result.

Review comment:
       ```suggestion
   Use per-segment caching with real-time data. For example, your queries request data actively arriving from Kafka alongside intervals in segments that are loaded on Historicals.  Druid can merge cached results from Historical segments with real-time results from the stream.  [Whole-query caching](#whole-query-caching), on the other hand, is not helpful in this scenario because new data from real-time ingestion will continually invalidate the entire cached result.
   ```

##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+For more information on tuning this value, see the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size).
 
-When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.md). Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.
+The [Coordinator](../design/coordinator.html) leads the assignment of segments to - and balance between - Historical processes.  [Zookeeper](../dependencies/zookeeper.md) is central to this collaboration; Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral Zookeeper entries under a [load queue path](../configuration/index.html#path-configuration) and each Historical process maintains a connection to Zookeeper, watching those paths for segment information.
+
+For more information about how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.html).
+
+When a Historical process sees a new Zookeeper load queue entry, it checks its own segment cache. If no information about the segment exists there, the Historical process first retrieves metadata from Zookeeper about the segment, including where the segment is located in Deep Storage and how it needs to decompress and process it.
+
+For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.html). 

Review comment:
       ```suggestion
   For more information about segment metadata and Druid segments in general, see [Segments](../design/segments.html). 
   ```

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).

Review comment:
       ```suggestion
   - [Per-segment caching](#per-segment-caching) stores partial query results for a specific segment. It is enabled by default.
   ```
   avoid parens

##########
File path: docs/querying/caching.md
##########
@@ -69,9 +89,11 @@ For instance, whole-query caching is a good option when you have queries that in
 
 - On Brokers for small production clusters with less than five servers. 
 
-     Do not use per-segment caches on the Broker for large production clusters. When `druid.broker.cache.populateCache` is `true` and query context parameter `populateCache` _is not_ `false`, Historicals return results on a per-segment basis without merging results locally thus negatively impacting cluster scalability.
+> **Avoid using per-segment cache at the Broker for large production clusters**
+>
+> When the Broker cache is enabled (`druid.broker.cache.populateCache` is `true`) and `populateCache` _is not_ `false` in the [query context](../querying/query-context.html), individual Historicals will _not_ merge individual segment-level results, and instead pass these back to the lead Broker.  The Broker must then carry out a large merge from _all_ segments on its own.

Review comment:
       ```suggestion
   Avoid using per-segment cache at the Broker for large production clusters. When the Broker cache is enabled (`druid.broker.cache.populateCache` is `true`) and `populateCache` _is not_ `false` in the [query context](../querying/query-context.html), individual Historicals will _not_ merge individual segment-level results, and instead pass these back to the lead Broker.  The Broker must then carry out a large merge from _all_ segments on its own.
   ```

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.
+
+It allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html) as instructed by the lead [coordinator](../design/coordinator.html).  Meanwhile, real-time segments, on the other hand, continue to have results computed at query time.
 
-When your queries include data from segments that are mutable and undergoing real-time ingestion, use a segment cache. In this case Druid caches query results for immutable historical segments when possible. It re-computes results for the real-time segments at query time.
+Per-segment cached results also have the potential to be merged into the results of later queries where there is a similar basic shape (filters, aggregations, etc.) yet cover a different period of time, for example.
 
-For example, you have queries that frequently include incoming data from a Kafka or Kinesis stream alongside unchanging segments. Per-segment caching lets Druid cache results from older immutable segments and merge them with updated data. Whole-query caching would not be helpful in this scenario because the new data from real-time ingestion continually invalidates the cache.
+Per-segment caching is controlled by the parameters `useCache` and `populateCache`.
+
+> **Use per-segment caching with real-time data**
+>
+> For example, you could have queries that address fresh data arriving from Kafka, and which additionally covers intervals in segments that are loaded on historicals.  Per-segment caching allows Druid to cache results from the historical segments confidently, and to merge them with real-time results from the stream.  [Whole-query caching](#whole-query-caching), on the other hand, would not be helpful in this scenario because new data from real-time ingestion will continually invalidate the _entire_ result.
 
 ### Whole-query caching
 
-If real-time ingestion invalidating the cache is not an issue for your queries, you can use **whole-query caching** on the Broker to increase query efficiency. The Broker performs whole-query caching operations before sending fan out queries to Historicals. Therefore Druid no longer needs to merge the per-segment results on the Broker.
+Here, entire results of individual queries are cached, meaning Druid no longer needs to merge the per-segment results on the Broker.

Review comment:
       ```suggestion
   With *whole-query caching*, Druid caches the entire results of individual queries. In this case the Broker no longer needs to merge the per-segment results.
   ```
   

##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).

Review comment:
       ```suggestion
   Each Historical process copies or "pulls" segment files from Deep Storage to local disk in an area called the *segment cache*.  Set the `druid.segmentCache.locations` to configure the size and location of the segment cache on each Historical process. See [Historical general configuration](../configuration/index.html#historical-general-configuration).
   ```
   

##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.
+
+It allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html) as instructed by the lead [coordinator](../design/coordinator.html).  Meanwhile, real-time segments, on the other hand, continue to have results computed at query time.
 
-When your queries include data from segments that are mutable and undergoing real-time ingestion, use a segment cache. In this case Druid caches query results for immutable historical segments when possible. It re-computes results for the real-time segments at query time.
+Per-segment cached results also have the potential to be merged into the results of later queries where there is a similar basic shape (filters, aggregations, etc.) yet cover a different period of time, for example.
 
-For example, you have queries that frequently include incoming data from a Kafka or Kinesis stream alongside unchanging segments. Per-segment caching lets Druid cache results from older immutable segments and merge them with updated data. Whole-query caching would not be helpful in this scenario because the new data from real-time ingestion continually invalidates the cache.
+Per-segment caching is controlled by the parameters `useCache` and `populateCache`.
+
+> **Use per-segment caching with real-time data**
+>
+> For example, you could have queries that address fresh data arriving from Kafka, and which additionally covers intervals in segments that are loaded on historicals.  Per-segment caching allows Druid to cache results from the historical segments confidently, and to merge them with real-time results from the stream.  [Whole-query caching](#whole-query-caching), on the other hand, would not be helpful in this scenario because new data from real-time ingestion will continually invalidate the _entire_ result.
 
 ### Whole-query caching
 
-If real-time ingestion invalidating the cache is not an issue for your queries, you can use **whole-query caching** on the Broker to increase query efficiency. The Broker performs whole-query caching operations before sending fan out queries to Historicals. Therefore Druid no longer needs to merge the per-segment results on the Broker.
+Here, entire results of individual queries are cached, meaning Druid no longer needs to merge the per-segment results on the Broker.
+
+Whole-query result caching is controlled by the parameters `useResultLevelCache` and `populateResultLevelCache` and runtime properties `druid.broker.cache.*`.
 
-For instance, whole-query caching is a good option when you have queries that include data from a batch ingestion task that runs every few hours or once a day. Per-segment caching would be less efficient in this case because it requires Druid to merge the per-segment results for each query, even when the results are cached.
+> **Use whole-query caching when segments are stable and you are not using real-time ingestion**
+>
+> If real-time ingestion invalidating the cache is not an issue for your queries, you can use *whole-query caching* on the Broker to increase query efficiency. The Broker performs whole-query caching operations before sending fan out queries to Historicals.
+>
+> For instance, whole-query caching is a good option when you have queries that include data from a batch ingestion task that runs every few hours or once a day. [Per-segment caching](#per-segment-caching) would be less efficient in this case because it requires Druid to merge the per-segment results for each query, even when the results are cached.

Review comment:
       ```suggestion
   Use whole-query caching when segments are stable and you are not using real-time ingestion. If real-time ingestion invalidating the cache is not an issue for your queries, you can use *whole-query caching* on the Broker to increase query efficiency.
   
   For instance, whole-query caching is a good option when you have queries that include data from a batch ingestion task that runs every few hours or once a day. [Per-segment caching](#per-segment-caching) would be less efficient in this case because it requires Druid to merge the per-segment results for each query, even when the results are cached.
   ```
   Why do I care that the Broker performs the caching before the fan out? How does that modify my behavior? 

##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+For more information on tuning this value, see the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size).
 
-When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.md). Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.
+The [Coordinator](../design/coordinator.html) leads the assignment of segments to - and balance between - Historical processes.  [Zookeeper](../dependencies/zookeeper.md) is central to this collaboration; Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral Zookeeper entries under a [load queue path](../configuration/index.html#path-configuration) and each Historical process maintains a connection to Zookeeper, watching those paths for segment information.
+
+For more information about how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.html).
+
+When a Historical process sees a new Zookeeper load queue entry, it checks its own segment cache. If no information about the segment exists there, the Historical process first retrieves metadata from Zookeeper about the segment, including where the segment is located in Deep Storage and how it needs to decompress and process it.

Review comment:
       ```suggestion
   When a Historical process detects a new entry in the Zookeeper load queue, it checks its own segment cache. If no information about the segment exists there, the Historical process first retrieves metadata from Zookeeper about the segment, including where the segment is located in Deep Storage and how it needs to decompress and process it.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] petermarshallio commented on a change in pull request #11584: Docs - query caching

Posted by GitBox <gi...@apache.org>.

petermarshallio commented on a change in pull request #11584:
URL: https://github.com/apache/druid/pull/11584#discussion_r722176610



##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.
 
 ### Per-segment caching
 
-The primary form of caching in Druid is the **per-segment cache** which stores query results on a per-segment basis. It is enabled on Historical services by default.
+The primary form of caching in Druid is a *per-segment results cache*.  This stores partial query results on a per-segment basis and is enabled on Historical services by default.
+
+It allows Druid to maintain a low-eviction-rate cache for segments that do not change, especially important for those segments that [historical](../design/historical.html) processes pull into their local _segment cache_ from [deep storage](../dependencies/deep-storage.html) as instructed by the lead [coordinator](../design/coordinator.html).  Meanwhile, real-time segments, on the other hand, continue to have results computed at query time.

Review comment:
       Agreed... this was just a flourish to distinguish between pulled segments and ones that are accumulating at ingestion time.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] petermarshallio commented on a change in pull request #11584: Docs - query caching

Posted by GitBox <gi...@apache.org>.

petermarshallio commented on a change in pull request #11584:
URL: https://github.com/apache/druid/pull/11584#discussion_r722167390



##########
File path: docs/design/historical.md
##########
@@ -39,17 +39,31 @@ org.apache.druid.cli.Main server historical
 
 ### Loading and serving segments
 
-Each Historical process maintains a constant connection to Zookeeper and watches a configurable set of Zookeeper paths for new segment information. Historical processes do not communicate directly with each other or with the Coordinator processes but instead rely on Zookeeper for coordination.
+Each Historical process copies ("pulls") segment files from [Deep Storage](../dependencies/deep-storage.md) to local disk in an area called the *segment cache*.  The size and location of the segment cache on each Historical process is set using `druid.segmentCache.locations` in [configuration](../configuration/index.html#historical-general-configuration).
 
-The [Coordinator](../design/coordinator.md) process is responsible for assigning new segments to Historical processes. Assignment is done by creating an ephemeral Zookeeper entry under a load queue path associated with a Historical process. For more information on how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.md).
+For more information on tuning this value, see the [Tuning Guide](../operations/basic-cluster-tuning.html#segment-cache-size).
 
-When a Historical process notices a new load queue entry in its load queue path, it will first check a local disk directory (cache) for the information about segment. If no information about the segment exists in the cache, the Historical process will download metadata about the new segment to serve from Zookeeper. This metadata includes specifications about where the segment is located in deep storage and about how to decompress and process the segment. For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.md). Once a Historical process completes processing a segment, the segment is announced in Zookeeper under a served segments path associated with the process. At this point, the segment is available for querying.
+The [Coordinator](../design/coordinator.html) leads the assignment of segments to - and balance between - Historical processes.  [Zookeeper](../dependencies/zookeeper.md) is central to this collaboration; Historical processes do not communicate directly with each other, nor do they communicate directly with the Coordinator.  Instead, the Coordinator creates ephemeral Zookeeper entries under a [load queue path](../configuration/index.html#path-configuration) and each Historical process maintains a connection to Zookeeper, watching those paths for segment information.
+
+For more information about how the Coordinator assigns segments to Historical processes, please see [Coordinator](../design/coordinator.html).
+
+When a Historical process sees a new Zookeeper load queue entry, it checks its own segment cache. If no information about the segment exists there, the Historical process first retrieves metadata from Zookeeper about the segment, including where the segment is located in Deep Storage and how it needs to decompress and process it.
+
+For more information about segment metadata and Druid segments in general, please see [Segments](../design/segments.html). 
+
+Once a Historical process has completed pulling down and processing a segment from Deep Storage, the segment is advertised as being available for queries.  This announcement by the Historical is made via Zookeeper, this time under a [served segments path](../configuration/index.html#path-configuration). At this point, the segment is considered available for querying by the Broker.
+
+For more information about how the Broker determines what data is available for queries, please see [Broker](broker.html).
+
+On startup, a Historical process searches through its segment cache and, in order for Historicals to be queried as soon as possible, immediately advertises all segments it finds there.
 
 ### Loading and serving segments from cache
 
-Recall that when a Historical process notices a new segment entry in its load queue path, the Historical process first checks a configurable cache directory on its local disk to see if the segment had been previously downloaded. If a local cache entry already exists, the Historical process will directly read the segment binary files from disk and load the segment.
+A technique called [memory mapping](https://en.wikipedia.org/wiki/Mmap) is used for the segment cache, consuming memory from the underlying operating system so that parts of segment files can be held in memory, increasing query performance at the data level.  The in-memory segment cache is therefore affected by, for example, the size of the Historical JVM, heap / direct memory buffers, and other processes on the operating system itself.
+
+At query time, if the required part of a segment file is available in the memory mapped cache (also known as the "page cache"), it will be re-used and read directly from memory.  If it is not, that part of the segment will be read from disk.  When this happens, there is potential for this new data to evict other segment data from memory.  Consequently, the closer that free operating system memory is to `druid.server.maxSize`, the faster historical processes typically respond at query time since segment data is very likely to be available in memory.  Conversely, the lower the free operating system memory, the more likely a Historical is to read segments from disk.

Review comment:
       👍🏻 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] petermarshallio commented on a change in pull request #11584: Docs - query caching

Posted by GitBox <gi...@apache.org>.

petermarshallio commented on a change in pull request #11584:
URL: https://github.com/apache/druid/pull/11584#discussion_r722171431



##########
File path: docs/operations/basic-cluster-tuning.md
##########
@@ -90,11 +90,9 @@ Tuning the cluster so that each Historical can accept 50 queries and 10 non-quer
 
 #### Segment Cache Size
 
-`druid.segmentCache.locations` specifies locations where segment data can be stored on the Historical. The sum of available disk space across these locations is set as the default value for property: `druid.server.maxSize`, which controls the total size of segment data that can be assigned by the Coordinator to a Historical.
+Avoid allocating a Historical an excessive amount of segment data.  As the value of (`free system memory` / total size of all `druid.segmentCache.locations`) increases, a greater proportion of segments can be kept in the [memory-mapped segment cache](../design/historical.md#loading-and-serving-segments-from-cache), allowing for better query performance.

Review comment:
       Should `can be hold` be `be held` or `can hold` ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] petermarshallio commented on a change in pull request #11584: Docs - query caching

Posted by GitBox <gi...@apache.org>.

petermarshallio commented on a change in pull request #11584:
URL: https://github.com/apache/druid/pull/11584#discussion_r722174661



##########
File path: docs/querying/caching.md
##########
@@ -32,30 +32,50 @@ If you're unfamiliar with Druid architecture, review the following topics before
 
 For instructions to configure query caching see [Using query caching](./using-caching.md).
 
+Cache monitoring, including the hit rate and number of evictions, is available in [Druid metrics](../operations/metrics.html#cache).
+
+Query-level caching is in addition to [data-level caching](../design/historical.md) on Historicals.
+
 ## Cache types
 
-Druid supports the following types of caches:
+Druid supports two types of query caching:
 
-- **Per-segment** caching which stores _partial results_ of a query for a specific segment. Per-segment caching is enabled on Historicals by default.
-- **Whole-query** caching which stores all results for a query.
+- [Per-segment caching](#per-segment-caching) stores _partial_ query results for a specific segment (enabled by default).
+- [Whole-query caching](#whole-query-caching) stores _final_ query results.
 
-To avoid returning stale results, Druid invalidates the cache the moment any underlying data changes for both types of cache.
+> **Druid invalidates _any_ cache the moment any underlying data changes**
+>
+> This ensures that Druid does not return stale results, especially important for `table` datasources that have highly-variable underlying data segments, including real-time data segments.
 
-Druid can store cache data on the local JVM heap or in an external distributed key/value store. The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). Maximum cache storage defaults to the minimum value of 1 GiB or the ten percent of the maximum runtime memory for the JVM with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.
+> **Druid can store cache data on the local JVM heap or in an external distributed key/value store (e.g. memcached)**
+>
+> The default is a local cache based upon [Caffeine](https://github.com/ben-manes/caffeine). The default maximum cache storage size is the minimum of 1 GiB / ten percent of maximum runtime memory for the JVM, with no cache expiration. See [Cache configuration](../configuration/index.md#cache-configuration) for information on how to configure cache storage.  When using caffeine, the cache is inside the JVM heap and is directly measurable.  Heap usage will grow up to the maximum configured size, and then the least recently used segment results will be evicted and replaced with newer results.

Review comment:
       So I'm not an expert here, and I suspect we may need to get an eng to look at this to be absolutely 100%, but I believe a flush method is more associated with putting what's in cache into the thing that it's caching, whereas an evict is about deletion of objects from the cache.
   > An eviction policy decides which objects should be deleted at any given time. This policy directly affects the cache's hit rate — a crucial characteristic of caching libraries.
   https://www.baeldung.com/java-caching-caffeine
   
   I know it's only words (!!!) but I think it might be good to be right on this point to avoid confusion.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org