You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by hi...@apache.org on 2019/07/24 22:26:13 UTC
[incubator-druid] branch master updated: More documentation
formatting fixes (#8149)
This is an automated email from the ASF dual-hosted git repository.
himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new c87b47e More documentation formatting fixes (#8149)
c87b47e is described below
commit c87b47e0fa5d5b93593fca43933f6389ebecef0d
Author: Magnus Henoch <ma...@gameanalytics.com>
AuthorDate: Wed Jul 24 23:26:03 2019 +0100
More documentation formatting fixes (#8149)
Add empty lines before bulleted lists and code blocks, to ensure that
they show up properly on the web site. See also #8079.
---
.../extensions-core/approximate-histograms.md | 1 +
.../development/extensions-core/bloom-filter.md | 1 +
.../extensions-core/druid-basic-security.md | 2 ++
docs/content/ingestion/native_tasks.md | 1 +
docs/content/operations/basic-cluster-tuning.md | 2 ++
docs/content/operations/deep-storage-migration.md | 1 +
docs/content/operations/export-metadata.md | 22 ++++++++++++----------
docs/content/operations/metadata-migration.md | 1 +
docs/content/operations/recommendations.md | 1 +
docs/content/querying/aggregations.md | 1 +
docs/content/tutorials/cluster.md | 9 +++++++++
11 files changed, 32 insertions(+), 10 deletions(-)
diff --git a/docs/content/development/extensions-core/approximate-histograms.md b/docs/content/development/extensions-core/approximate-histograms.md
index 30b5f32..2e900d2 100644
--- a/docs/content/development/extensions-core/approximate-histograms.md
+++ b/docs/content/development/extensions-core/approximate-histograms.md
@@ -37,6 +37,7 @@ The Approximate Histogram aggregator is deprecated. Please use <a href="../exten
This aggregator is based on
[http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf](http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf)
to compute approximate histograms, with the following modifications:
+
- some tradeoffs in accuracy were made in the interest of speed (see below)
- the sketch maintains the exact original data as long as the number of
distinct data points is fewer than the resolutions (number of centroids),
diff --git a/docs/content/development/extensions-core/bloom-filter.md b/docs/content/development/extensions-core/bloom-filter.md
index 3d6749a..f0563a0 100644
--- a/docs/content/development/extensions-core/bloom-filter.md
+++ b/docs/content/development/extensions-core/bloom-filter.md
@@ -33,6 +33,7 @@ to use with Druid for cases where an explicit filter is impossible, e.g. filteri
values.
Following are some characteristics of BloomFilters:
+
- BloomFilters are highly space efficient when compared to using a HashSet.
- Because of the probabilistic nature of bloom filters, false positive results are possible (element was not actually
inserted into a bloom filter during construction, but `test()` says true)
diff --git a/docs/content/development/extensions-core/druid-basic-security.md b/docs/content/development/extensions-core/druid-basic-security.md
index e067fdf..4189c3f 100644
--- a/docs/content/development/extensions-core/druid-basic-security.md
+++ b/docs/content/development/extensions-core/druid-basic-security.md
@@ -25,6 +25,7 @@ title: "Basic Security"
# Druid Basic Security
This Apache Druid (incubating) extension adds:
+
- an Authenticator which supports [HTTP Basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication)
- an Authorizer which implements basic role-based access control
@@ -342,6 +343,7 @@ Unassign role {roleName} from user {userName}
Set the permissions of {roleName}. This replaces the previous set of permissions on the role.
Content: List of JSON Resource-Action objects, e.g.:
+
```
[
{
diff --git a/docs/content/ingestion/native_tasks.md b/docs/content/ingestion/native_tasks.md
index c5cd91b..1cf5e01 100644
--- a/docs/content/ingestion/native_tasks.md
+++ b/docs/content/ingestion/native_tasks.md
@@ -55,6 +55,7 @@ the implementation of splittable firehoses. Please note that multiple tasks can
if one of them fails.
You may want to consider the below points:
+
- Since this task doesn't shuffle intermediate data, it isn't available for [perfect rollup](../ingestion/index.html#roll-up-modes).
- The number of tasks for parallel ingestion is decided by `maxNumSubTasks` in the tuningConfig.
Since the supervisor task creates up to `maxNumSubTasks` worker tasks regardless of the available task slots,
diff --git a/docs/content/operations/basic-cluster-tuning.md b/docs/content/operations/basic-cluster-tuning.md
index aa09c07..226cc92 100644
--- a/docs/content/operations/basic-cluster-tuning.md
+++ b/docs/content/operations/basic-cluster-tuning.md
@@ -37,6 +37,7 @@ If you have questions on tuning Druid for specific use cases, or questions on co
#### Heap sizing
The biggest contributions to heap usage on Historicals are:
+
- Partial unmerged query results from segments
- The stored maps for [lookups](../querying/lookups.html).
@@ -63,6 +64,7 @@ Be sure to add `(2 * total size of all loaded lookups)` to your heap size in add
Please see the [General Guidelines for Processing Threads and Buffers](#general-guidelines-for-processing-threads-and-buffers) section for an overview of processing thread/buffer configuration.
On Historicals:
+
- `druid.processing.numThreads` should generally be set to `(number of cores - 1)`: a smaller value can result in CPU underutilization, while going over the number of cores can result in unnecessary CPU contention.
- `druid.processing.buffer.sizeBytes` can be set to 500MB.
- `druid.processing.numMergeBuffers`, a 1:4 ratio of merge buffers to processing threads is a reasonable choice for general use.
diff --git a/docs/content/operations/deep-storage-migration.md b/docs/content/operations/deep-storage-migration.md
index 3fc61e7..55180a0 100644
--- a/docs/content/operations/deep-storage-migration.md
+++ b/docs/content/operations/deep-storage-migration.md
@@ -28,6 +28,7 @@ If you have been running an evaluation Druid cluster using local deep storage an
more production-capable deep storage system such as S3 or HDFS, this document describes the necessary steps.
Migration of deep storage involves the following steps at a high level:
+
- Copying segments from local deep storage to the new deep storage
- Exporting Druid's segments table from metadata
- Rewriting the load specs in the exported segment data to reflect the new deep storage location
diff --git a/docs/content/operations/export-metadata.md b/docs/content/operations/export-metadata.md
index 11c0f76..d1119ee 100644
--- a/docs/content/operations/export-metadata.md
+++ b/docs/content/operations/export-metadata.md
@@ -27,6 +27,7 @@ title: "Export Metadata Tool"
Druid includes an `export-metadata` tool for assisting with migration of cluster metadata and deep storage.
This tool exports the contents of the following Druid metadata tables:
+
- segments
- rules
- config
@@ -37,6 +38,7 @@ Additionally, the tool can rewrite the local deep storage location descriptors i
to point to new deep storage locations (S3, HDFS, and local rewrite paths are supported).
The tool has the following limitations:
+
- Only exporting from Derby metadata is currently supported
- If rewriting load specs for deep storage migration, only migrating from local deep storage is currently supported.
@@ -46,20 +48,19 @@ The `export-metadata` tool provides the following options:
### Connection Properties
-`--connectURI`: The URI of the Derby database, e.g. `jdbc:derby://localhost:1527/var/druid/metadata.db;create=true`
-`--user`: Username
-`--password`: Password
-`--base`: corresponds to the value of `druid.metadata.storage.tables.base` in the configuration, `druid` by default.
+- `--connectURI`: The URI of the Derby database, e.g. `jdbc:derby://localhost:1527/var/druid/metadata.db;create=true`
+- `--user`: Username
+- `--password`: Password
+- `--base`: corresponds to the value of `druid.metadata.storage.tables.base` in the configuration, `druid` by default.
### Output Path
-`--output-path`, `-o`: The output directory of the tool. CSV files for the Druid segments, rules, config, datasource, and supervisors tables will be written to this directory.
+- `--output-path`, `-o`: The output directory of the tool. CSV files for the Druid segments, rules, config, datasource, and supervisors tables will be written to this directory.
### Export Format Options
-`--use-hex-blobs`, `-x`: If set, export BLOB payload columns as hexadecimal strings. This needs to be set if importing back into Derby. Default is false.
-
-`--booleans-as-strings`, `-t`: If set, write boolean values as "true" or "false" instead of "1" and "0". This needs to be set if importing back into Derby. Default is false.
+- `--use-hex-blobs`, `-x`: If set, export BLOB payload columns as hexadecimal strings. This needs to be set if importing back into Derby. Default is false.
+- `--booleans-as-strings`, `-t`: If set, write boolean values as "true" or "false" instead of "1" and "0". This needs to be set if importing back into Derby. Default is false.
### Deep Storage Migration
@@ -69,8 +70,8 @@ By setting the options below, the tool will rewrite the segment load specs to po
This helps users migrate segments stored in local deep storage to S3.
-`--s3bucket`, `-b`: The S3 bucket that will hold the migrated segments
-`--s3baseKey`, `-k`: The base S3 key where the migrated segments will be stored
+- `--s3bucket`, `-b`: The S3 bucket that will hold the migrated segments
+- `--s3baseKey`, `-k`: The base S3 key where the migrated segments will be stored
When copying the local deep storage segments to S3, the rewrite performed by this tool requires that the directory structure of the segments be unchanged.
@@ -142,6 +143,7 @@ java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log
```
In the example command above:
+
- `lib` is the the Druid lib directory
- `extensions` is the Druid extensions directory
- `/tmp/csv` is the output directory. Please make sure that this directory exists.
diff --git a/docs/content/operations/metadata-migration.md b/docs/content/operations/metadata-migration.md
index 95c05ef..9f6b419 100644
--- a/docs/content/operations/metadata-migration.md
+++ b/docs/content/operations/metadata-migration.md
@@ -61,6 +61,7 @@ Update your Druid runtime properties with the new metadata configuration.
Druid provides a `metadata-init` tool for creating Druid's metadata tables. After initializing the Druid database, you can run the commands shown below from the root of the Druid package to initialize the tables.
In the example commands below:
+
- `lib` is the the Druid lib directory
- `extensions` is the Druid extensions directory
- `base` corresponds to the value of `druid.metadata.storage.tables.base` in the configuration, `druid` by default.
diff --git a/docs/content/operations/recommendations.md b/docs/content/operations/recommendations.md
index 61cb871..21100bb 100644
--- a/docs/content/operations/recommendations.md
+++ b/docs/content/operations/recommendations.md
@@ -59,6 +59,7 @@ JVM Flags:
Please note that above flags are general guidelines only. Be cautious and feel free to change them if necessary for the specific deployment.
Additionally, for large jvm heaps, here are a few Garbage Collection efficiency guidelines that have been known to help in some cases.
+
- Mount /tmp on tmpfs ( See http://www.evanjones.ca/jvm-mmap-pause.html )
- On Disk-IO intensive processes (e.g. Historical and MiddleManager), GC and Druid logs should be written to a different disk than where data is written.
- Disable Transparent Huge Pages ( See https://blogs.oracle.com/linux/performance-issues-with-transparent-huge-pages-thp )
diff --git a/docs/content/querying/aggregations.md b/docs/content/querying/aggregations.md
index ba9b80e..66acea0 100644
--- a/docs/content/querying/aggregations.md
+++ b/docs/content/querying/aggregations.md
@@ -337,6 +337,7 @@ The [Approximate Histogram](../development/extensions-core/approximate-histogram
The algorithm used by this deprecated aggregator is highly distribution-dependent and its output is subject to serious distortions when the input does not fit within the algorithm's limitations.
A [study published by the DataSketches team](https://datasketches.github.io/docs/Quantiles/DruidApproxHistogramStudy.html) demonstrates some of the known failure modes of this algorithm:
+
- The algorithm's quantile calculations can fail to provide results for a large range of rank values (all ranks less than 0.89 in the example used in the study), returning all zeroes instead.
- The algorithm can completely fail to record spikes in the tail ends of the distribution
- In general, the histogram produced by the algorithm can deviate significantly from the true histogram, with no bounds on the errors.
diff --git a/docs/content/tutorials/cluster.md b/docs/content/tutorials/cluster.md
index a202d25..f78cf5c 100644
--- a/docs/content/tutorials/cluster.md
+++ b/docs/content/tutorials/cluster.md
@@ -30,6 +30,7 @@ In this document, we'll set up a simple cluster and discuss how it can be furthe
your needs.
This simple cluster will feature:
+
- A Master server to host the Coordinator and Overlord processes
- Two scalable, fault-tolerant Data servers running Historical and MiddleManager processes
- A query server, hosting the Druid Broker and Router processes
@@ -49,6 +50,7 @@ The Coordinator and Overlord processes are responsible for handling the metadata
In this example, we will be deploying the equivalent of one AWS [m5.2xlarge](https://aws.amazon.com/ec2/instance-types/m5/) instance.
This hardware offers:
+
- 8 vCPUs
- 31 GB RAM
@@ -77,6 +79,7 @@ in-memory query cache. These servers benefit greatly from CPU and RAM.
In this example, we will be deploying the equivalent of one AWS [m5.2xlarge](https://aws.amazon.com/ec2/instance-types/m5/) instance.
This hardware offers:
+
- 8 vCPUs
- 31 GB RAM
@@ -323,6 +326,7 @@ You can copy your existing `coordinator-overlord` configs from the single-server
Suppose we are migrating from a single-server deployment that had 32 CPU and 256GB RAM. In the old deployment, the following configurations for Historicals and MiddleManagers were applied:
Historical (Single-server)
+
```
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=8
@@ -330,6 +334,7 @@ druid.processing.numThreads=31
```
MiddleManager (Single-server)
+
```
druid.worker.capacity=8
druid.indexer.fork.property.druid.processing.numMergeBuffers=2
@@ -340,11 +345,13 @@ druid.indexer.fork.property.druid.processing.numThreads=1
In the clustered deployment, we can choose a split factor (2 in this example), and deploy 2 Data servers with 16CPU and 128GB RAM each. The areas to scale are the following:
Historical
+
- `druid.processing.numThreads`: Set to `(num_cores - 1)` based on the new hardware
- `druid.processing.numMergeBuffers`: Divide the old value from the single-server deployment by the split factor
- `druid.processing.buffer.sizeBytes`: Keep this unchanged
MiddleManager:
+
- `druid.worker.capacity`: Divide the old value from the single-server deployment by the split factor
- `druid.indexer.fork.property.druid.processing.numMergeBuffers`: Keep this unchanged
- `druid.indexer.fork.property.druid.processing.buffer.sizeBytes`: Keep this unchanged
@@ -353,6 +360,7 @@ MiddleManager:
The resulting configs after the split:
New Historical (on 2 Data servers)
+
```
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=8
@@ -360,6 +368,7 @@ New Historical (on 2 Data servers)
```
New MiddleManager (on 2 Data servers)
+
```
druid.worker.capacity=4
druid.indexer.fork.property.druid.processing.numMergeBuffers=2
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org