You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by fj...@apache.org on 2019/05/17 21:29:19 UTC
[incubator-druid] branch 0.15.0-incubating updated: some more doc
improvements (#7675) (#7686)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch 0.15.0-incubating
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/0.15.0-incubating by this push:
new 4663795 some more doc improvements (#7675) (#7686)
4663795 is described below
commit 46637958749ad70e523c58adcfefecc6dcaaac81
Author: Jonathan Wei <jo...@users.noreply.github.com>
AuthorDate: Fri May 17 14:29:13 2019 -0700
some more doc improvements (#7675) (#7686)
---
docs/content/design/index.md | 41 ++++++++++++++++++++-------------
docs/content/design/segments.md | 2 +-
docs/content/ingestion/index.md | 2 +-
docs/content/toc.md | 51 ++++++++++++++++++++++-------------------
4 files changed, 55 insertions(+), 41 deletions(-)
diff --git a/docs/content/design/index.md b/docs/content/design/index.md
index ec7e38a..191a7d6 100644
--- a/docs/content/design/index.md
+++ b/docs/content/design/index.md
@@ -24,18 +24,23 @@ title: "Apache Druid (incubating) Design"
# What is Druid?<a id="what-is-druid"></a>
-Apache Druid (incubating) is a data store designed for high-performance slice-and-dice analytics
-("[OLAP](http://en.wikipedia.org/wiki/Online_analytical_processing)"-style) on large data sets. Druid is most often
-used as a data store for powering GUI analytical applications, or as a backend for highly-concurrent APIs that need
-fast aggregations. Common application areas for Druid include:
+Apache Druid (incubating) is a real-time analytics database designed for fast slice-and-dice analytics
+("[OLAP](http://en.wikipedia.org/wiki/Online_analytical_processing)" queries) on large data sets. Druid is most often
+used as a database for powering use cases where real-time ingest, fast query performance, and high uptime are important.
+As such, Druid is commonly used for powering GUIs of analytical applications, or as a backend for highly-concurrent APIs
+that need fast aggregations. Druid works best with event-oriented data.
-- Clickstream analytics
-- Network flow analytics
+Common application areas for Druid include:
+
+- Clickstream analytics (web and mobile analytics)
+- Network telemetry analytics (network performance monitoring)
- Server metrics storage
+- Supply chain analytics (manufacturing metrics)
- Application performance metrics
-- Digital marketing analytics
+- Digital marketing/advertising analytics
- Business intelligence / OLAP
+Druid's core architecture combines ideas from data warehouses, timeseries databases, and logsearch systems. Some of
Druid's key features are:
1. **Columnar storage format.** Druid uses column-oriented storage, meaning it only needs to load the exact columns
@@ -45,7 +50,7 @@ column is stored optimized for its particular data type, which supports fast sca
offer ingest rates of millions of records/sec, retention of trillions of records, and query latencies of sub-second to a
few seconds.
3. **Massively parallel processing.** Druid can process a query in parallel across the entire cluster.
-4. **Realtime or batch ingestion.** Druid can ingest data either realtime (ingested data is immediately available for
+4. **Realtime or batch ingestion.** Druid can ingest data either real-time (ingested data is immediately available for
querying) or in batches.
5. **Self-healing, self-balancing, easy to operate.** As an operator, to scale the cluster out or in, simply add or
remove servers and the cluster will rebalance itself automatically, in the background, without any downtime. If any
@@ -59,11 +64,14 @@ Druid servers, replication ensures that queries are still possible while the sys
7. **Indexes for quick filtering.** Druid uses [CONCISE](https://arxiv.org/pdf/1004.0403) or
[Roaring](https://roaringbitmap.org/) compressed bitmap indexes to create indexes that power fast filtering and
searching across multiple columns.
-8. **Approximate algorithms.** Druid includes algorithms for approximate count-distinct, approximate ranking, and
+8. **Time-based partitioning.** Druid first partitions data by time, and can additionally partition based on other fields.
+This means time-based queries will only access the partitions that match the time range of the query. This leads to
+significant performance improvements for time-based data.
+9. **Approximate algorithms.** Druid includes algorithms for approximate count-distinct, approximate ranking, and
computation of approximate histograms and quantiles. These algorithms offer bounded memory usage and are often
substantially faster than exact computations. For situations where accuracy is more important than speed, Druid also
offers exact count-distinct and exact ranking.
-9. **Automatic summarization at ingest time.** Druid optionally supports data summarization at ingestion time. This
+10. **Automatic summarization at ingest time.** Druid optionally supports data summarization at ingestion time. This
summarization partially pre-aggregates your data, and can lead to big costs savings and performance boosts.
# When should I use Druid?<a id="when-to-use-druid"></a>
@@ -85,7 +93,8 @@ Situations where you would likely _not_ want to use Druid include:
- You need low-latency updates of _existing_ records using a primary key. Druid supports streaming inserts, but not streaming updates (updates are done using
background batch jobs).
- You are building an offline reporting system where query latency is not very important.
-- You want to do "big" joins (joining one big fact table to another big fact table).
+- You want to do "big" joins (joining one big fact table to another big fact table) and you are okay with these queries
+taking up to hours to complete.
# Architecture
@@ -157,7 +166,7 @@ The following diagram shows how queries and data flow through this architecture,
Druid data is stored in "datasources", which are similar to tables in a traditional RDBMS. Each datasource is
partitioned by time and, optionally, further partitioned by other attributes. Each time range is called a "chunk" (for
example, a single day, if your datasource is partitioned by day). Within a chunk, data is partitioned into one or more
-"segments". Each segment is a single file, typically comprising up to a few million rows of data. Since segments are
+["segments"](../design/segments.html). Each segment is a single file, typically comprising up to a few million rows of data. Since segments are
organized into time chunks, it's sometimes helpful to think of segments as living on a timeline like the following:
<img src="../../img/druid-timeline.png" width="800" />
@@ -183,10 +192,10 @@ cluster.
# Query processing
-Queries first enter the Broker, where the Broker will identify which segments have data that may pertain to that query.
+Queries first enter the [Broker](../design/broker.html), where the Broker will identify which segments have data that may pertain to that query.
The list of segments is always pruned by time, and may also be pruned by other attributes depending on how your
-datasource is partitioned. The Broker will then identify which Historicals and MiddleManagers are serving those segments
-and send a rewritten subquery to each of those processes. The Historical/MiddleManager processes will take in the
+datasource is partitioned. The Broker will then identify which [Historicals](../design/historical.html) and
+[MiddleManagers](../design/middlemanager.html) are serving those segments and send a rewritten subquery to each of those processes. The Historical/MiddleManager processes will take in the
queries, process them and return results. The Broker receives results and merges them together to get the final answer,
which it returns to the original caller.
@@ -200,4 +209,4 @@ So Druid uses three different techniques to maximize query performance:
- Pruning which segments are accessed for each query.
- Within each segment, using indexes to identify which rows must be accessed.
-- Within each segment, only reading the specific rows and columns that are relevant to a particular query.
\ No newline at end of file
+- Within each segment, only reading the specific rows and columns that are relevant to a particular query.
diff --git a/docs/content/design/segments.md b/docs/content/design/segments.md
index d8d69c1..adc454b 100644
--- a/docs/content/design/segments.md
+++ b/docs/content/design/segments.md
@@ -28,7 +28,7 @@ Apache Druid (incubating) stores its index in *segment files*, which are partiti
time. In a basic setup, one segment file is created for each time
interval, where the time interval is configurable in the
`segmentGranularity` parameter of the `granularitySpec`, which is
-documented [here](../ingestion/ingestion-spec.html#granularityspec). For druid to
+documented [here](../ingestion/ingestion-spec.html#granularityspec). For Druid to
operate well under heavy query load, it is important for the segment
file size to be within the recommended range of 300mb-700mb. If your
segment files are larger than this range, then consider either
diff --git a/docs/content/ingestion/index.md b/docs/content/ingestion/index.md
index 9141cb5..e9909a1 100644
--- a/docs/content/ingestion/index.md
+++ b/docs/content/ingestion/index.md
@@ -33,7 +33,7 @@ title: "Ingestion"
Apache Druid (incubating) data is stored in "datasources", which are similar to tables in a traditional RDBMS. Each datasource is
partitioned by time and, optionally, further partitioned by other attributes. Each time range is called a "chunk" (for
example, a single day, if your datasource is partitioned by day). Within a chunk, data is partitioned into one or more
-"segments". Each segment is a single file, typically comprising up to a few million rows of data. Since segments are
+["segments"](../design/segments.html). Each segment is a single file, typically comprising up to a few million rows of data. Since segments are
organized into time chunks, it's sometimes helpful to think of segments as living on a timeline like the following:
<img src="../../img/druid-timeline.png" width="800" />
diff --git a/docs/content/toc.md b/docs/content/toc.md
index 713a35b..9663ebb 100644
--- a/docs/content/toc.md
+++ b/docs/content/toc.md
@@ -118,42 +118,47 @@ layout: toc
* [ZooKeeper](/docs/VERSION/dependencies/zookeeper.html)
## Operations
- * [API Reference](/docs/VERSION/operations/api-reference.html)
- * [Coordinator](/docs/VERSION/operations/api-reference.html#coordinator)
- * [Overlord](/docs/VERSION/operations/api-reference.html#overlord)
- * [MiddleManager](/docs/VERSION/operations/api-reference.html#middlemanager)
- * [Peon](/docs/VERSION/operations/api-reference.html#peon)
- * [Broker](/docs/VERSION/operations/api-reference.html#broker)
- * [Historical](/docs/VERSION/operations/api-reference.html#historical)
+ * [Management UIs](/docs/VERSION/operations/management-uis.html)
* [Including Extensions](/docs/VERSION/operations/including-extensions.html)
* [Data Retention](/docs/VERSION/operations/rule-configuration.html)
+ * [High Availability](/docs/VERSION/operations/high-availability.html)
+ * [Updating the Cluster](/docs/VERSION/operations/rolling-updates.html)
* [Metrics and Monitoring](/docs/VERSION/operations/metrics.html)
* [Alerts](/docs/VERSION/operations/alerts.html)
- * [Updating the Cluster](/docs/VERSION/operations/rolling-updates.html)
* [Different Hadoop Versions](/docs/VERSION/operations/other-hadoop.html)
- * [High Availability](/docs/VERSION/operations/high-availability.html)
- * [Management UIs](/docs/VERSION/operations/management-uis.html)
- * [Dump Segment Tool](/docs/VERSION/operations/dump-segment.html)
- * [Insert Segment Tool](/docs/VERSION/operations/insert-segment-to-db.html)
- * [Pull Dependencies Tool](/docs/VERSION/operations/pull-deps.html)
- * [Recommendations](/docs/VERSION/operations/recommendations.html)
- * [TLS Support](/docs/VERSION/operations/tls-support.html)
- * [Password Provider](/docs/VERSION/operations/password-provider.html)
* [HTTP Compression](/docs/VERSION/operations/http-compression.html)
* [Basic Cluster Tuning](/docs/VERSION/operations/basic-cluster-tuning.html)
- * [Single-server Deployment Examples](/docs/VERSION/operations/single-server.html)
- * [Clustered Deployment Example](/docs/VERSION/operations/example-cluster.html)
+ * Examples
+ * [Single-server Deployment Examples](/docs/VERSION/operations/single-server.html)
+ * [Clustered Deployment Example](/docs/VERSION/operations/example-cluster.html)
+ * [Recommendations](/docs/VERSION/operations/recommendations.html)
+ * [Performance FAQ](/docs/VERSION/operations/performance-faq.html)
+ * [API Reference](/docs/VERSION/operations/api-reference.html)
+ * [Coordinator](/docs/VERSION/operations/api-reference.html#coordinator)
+ * [Overlord](/docs/VERSION/operations/api-reference.html#overlord)
+ * [MiddleManager](/docs/VERSION/operations/api-reference.html#middlemanager)
+ * [Peon](/docs/VERSION/operations/api-reference.html#peon)
+ * [Broker](/docs/VERSION/operations/api-reference.html#broker)
+ * [Historical](/docs/VERSION/operations/api-reference.html#historical)
+ * Tools
+ * [Dump Segment Tool](/docs/VERSION/operations/dump-segment.html)
+ * [Insert Segment Tool](/docs/VERSION/operations/insert-segment-to-db.html)
+ * [Pull Dependencies Tool](/docs/VERSION/operations/pull-deps.html)
+ * Security
+ * [TLS Support](/docs/VERSION/operations/tls-support.html)
+ * [Password Provider](/docs/VERSION/operations/password-provider.html)
## Configuration
* [Configuration Reference](/docs/VERSION/configuration/index.html)
* [Recommended Configuration File Organization](/docs/VERSION/configuration/index.html#recommended-configuration-file-organization)
* [JVM Configuration Best Practices](/docs/VERSION/configuration/index.html#jvm-configuration-best-practices)
* [Common Configuration](/docs/VERSION/configuration/index.html#common-configurations)
- * [Coordinator](/docs/VERSION/configuration/index.html#coordinator)
- * [Overlord](/docs/VERSION/configuration/index.html#overlord)
- * [MiddleManager & Peons](/docs/VERSION/configuration/index.html#middle-manager-and-peons)
- * [Broker](/docs/VERSION/configuration/index.html#broker)
- * [Historical](/docs/VERSION/configuration/index.html#historical)
+ * Processes
+ * [Coordinator](/docs/VERSION/configuration/index.html#coordinator)
+ * [Overlord](/docs/VERSION/configuration/index.html#overlord)
+ * [MiddleManager & Peons](/docs/VERSION/configuration/index.html#middle-manager-and-peons)
+ * [Historical](/docs/VERSION/configuration/index.html#historical)
+ * [Broker](/docs/VERSION/configuration/index.html#broker)
* [Caching](/docs/VERSION/configuration/index.html#cache-configuration)
* [General Query Configuration](/docs/VERSION/configuration/index.html#general-query-configuration)
* [Configuring Logging](/docs/VERSION/configuration/logging.html)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org