You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by br...@apache.org on 2018/09/06 01:50:50 UTC
[drill-site] branch asf-site updated: edits
This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/drill-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cc74e50 edits
cc74e50 is described below
commit cc74e5055233476bfb4445b4256b74285ab2095d
Author: Bridget Bevens <bb...@maprtech.com>
AuthorDate: Wed Sep 5 18:50:36 2018 -0700
edits
---
.../index.html | 22 ++++++++++------------
feed.xml | 4 ++--
2 files changed, 12 insertions(+), 14 deletions(-)
diff --git a/docs/sort-based-and-hash-based-memory-constrained-operators/index.html b/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
index 59cb98b..19c5885 100644
--- a/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
+++ b/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
@@ -1285,11 +1285,9 @@
</ul>
<p>Drill only uses the External Sort operator to sort data. Drill uses the Hash-Aggregate operator to aggregate data. Alternatively, Drill can sort the data and then use the (lightweight) Streaming-Aggregate operator to aggregate data.
-Drill uses the Hash-Join operator to join data. Alternatively, Drill can use the Nested-Loop-Join or sort the data and then use the (lightweight) Merge-Join. Drill typically uses Hash operators for joining and aggregation, as they perform better than the Sort operator (Hash - O(N) vs. Sort - O(N * log(N))). However, if the Hash operators are disabled, or the data is already sorted, Drill uses the alternative methods previously described.</p>
+Drill uses the Hash-Join operator to join data. Alternatively, Drill can use the Nested-Loop-Join or sort the data and then use the (lightweight) Merge-Join. Drill typically uses Hash operators for joining and aggregation, as they perform better than the Sort operator (Hash - O(N) vs. Sort - O(N * log(N))). However, if you disable the Hash operators, or the data is already sorted, Drill uses the alternative methods previously described.</p>
-<p>The memory configuration in Drill is specified as the memory limit per-query, per-node. The allocated memory is equally divided among all instances of the spillable operators (per query on each node). The number of instances is the number of spillable operators in the query plan multiplied by the maximal degree of parallelism. The maximal degree of parallelism is the number of minor fragments required to perform the work for each instance of a spillable operator. When an instance of a [...]
-
-<p>To see the difference in memory consumption between the operators, run a query and then view the query profile in the Drill Web UI. Optionally, you can disable the Hash operators, which forces Drill to use the Merge-Join and Streaming-Aggregate operators. </p>
+<p>The memory configuration in Drill is specified as the memory limit per-query, per-node. The allocated memory is equally divided among all instances of the spillable operators (per query on each node). The number of instances is the number of spillable operators in the query plan multiplied by the maximal degree of parallelism. The maximal degree of parallelism is the number of minor fragments required to perform the work for each instance of a spillable operator. When an instance of a [...]
<h2 id="spill-to-disk">Spill to Disk</h2>
@@ -1305,7 +1303,7 @@ Drill uses the Hash-Join operator to join data. Alternatively, Drill can use the
<p>Spillable operators write data to a temporary work area on disk when they cannot process all of the data in memory. The default location of the temporary work area is <code>/tmp/drill/spill</code> on the local file system. </p>
-<p>The <code>/tmp/drill/spill</code> directory should suffice for small workloads or examples, however it is highly recommended that you redirect the default spill location to a location with enough disk space to support spilling for large workloads.</p>
+<p>The <code>/tmp/drill/spill</code> directory should suffice for small workloads or examples; however, you should redirect the default spill location to a location with enough disk space to support spilling for large workloads.</p>
<p><strong>Note:</strong> Spilled data may require more space than the table referenced in the query that is spilling the data. For example, when the underlying table is compressed (Parquet), or when the operator received data joined from multiple tables.</p>
@@ -1313,15 +1311,15 @@ Drill uses the Hash-Join operator to join data. Alternatively, Drill can use the
<p><strong>Configuring Spill to Disk</strong> </p>
-<p>The <code>drill-override.conf</code> file, located in the <code>/conf</code> directory, contains options that set the spill locations for the Hash and Sort operators. An administrator can change the file system and directories into which the operators spill data. Refer to the <code>drill-override-example.conf</code> file included in the <code>/conf</code> directory for examples. </p>
+<p>The <code>drill-override.conf</code> file, located in the <code>/conf</code> directory, contains options that set the spill locations for the spillable operators. An administrator can change the file system and directories into which the operators spill data. Refer to the <code>drill-override-example.conf</code> file included in the <code>/conf</code> directory for examples. </p>
<p>The following list describes the spill to disk configuration options: </p>
<ul>
-<li><strong>drill.exec.spill.fs</strong><br>
-Introduced in Drill 1.11. The default file system on the local machine into which the Sort, Hash Aggregate, and Hash Join operators spill data. You can configure this option so that data spills into a distributed file system, such as hdfs. For example, "hdfs:///". The default setting is "file:///".</li>
-<li><strong>drill.exec.spill.directories</strong><br>
-Introduced in Drill 1.11. The list of directories into which the Sort, Hash Aggregate, and Hash Join operators spill data. The list must be an array with directories separated by a comma, for example ["/fs1/drill/spill" , "/fs2/drill/spill" , "/fs3/drill/spill"]. The default setting is ["/tmp/drill/spill"].<br></li>
+<li><p><strong>drill.exec.spill.fs</strong><br>
+Introduced in Drill 1.11. The default file system on the local machine into which the spillable operators spill data. You can configure this option so that data spills into a distributed file system, such as hdfs. For example, "hdfs:///". The default setting is "file:///". </p></li>
+<li><p><strong>drill.exec.spill.directories</strong><br>
+Introduced in Drill 1.11. The list of directories into which the spillable operators spill data. The list must be an array with directories separated by a comma, for example ["/fs1/drill/spill" , "/fs2/drill/spill" , "/fs3/drill/spill"]. The default setting is ["/tmp/drill/spill"]. </p></li>
</ul>
<p><strong>Note:</strong> The following options were available prior to Drill 1.11, but have since been deprecated and replaced with the options described above: </p>
@@ -1334,7 +1332,7 @@ Introduced in Drill 1.11. The list of directories into which the Sort, Hash Aggr
<h2 id="memory-allocation">Memory Allocation</h2>
-<p>Drill evenly splits the available memory among all instances of the spillable operators. When a query is parallelized, the number of operators is multiplied, which reduces the amount of memory given to each instance of the operators during a query. </p>
+<p>Drill evenly splits the available memory among all instances of the spillable operators. When a query is parallelized, the number of operators is multiplied, which reduces the amount of memory given to each instance of the operators during a query. To see the difference in memory consumption between the operators, you can run a query and then view the query profile in the Drill Web UI. Optionally, you can disable the Hash operators, which forces Drill to use the Merge-Join and Streami [...]
<p><strong>Memory Allocation Configuration Options</strong> </p>
@@ -1344,7 +1342,7 @@ Introduced in Drill 1.11. The list of directories into which the Sort, Hash Aggr
<li><p><strong>planner.memory.max_query_memory_per_node</strong><br>
The <code>planner.memory.max_query_memory_per_node</code> option is the minimum amount of memory available to Drill per query on a node. The default of 2 GB typically allows between two and three concurrent queries to run when the JVM is configured to use 8 GB of direct memory (default). When the memory requirement for Drill increases, the default of 2 GB is constraining. You must increase the amount of memory for queries to complete, unless the setting for the <code>planner.memory.perce [...]
<li><p><strong>planner.memory.percent_per_query</strong><br>
-Alternatively, the <code>planner.memory.percent_per_query</code> option sets the memory as a percentage of the total direct memory. The default is 5%. This value is only used when throttling is disabled. Setting the value to 0 disables the option. You can increase or decrease the value, however you should set the percentage well below the JVM direct memory to account for the cases where Drill does not manage memory, such as for the less memory intensive operators. </p>
+Alternatively, the <code>planner.memory.percent_per_query</code> option sets the memory as a percentage of the total direct memory. The default is 5%. This value is only used when throttling is disabled. Setting the value to 0 disables the option. You can increase or decrease the value; however, you should set the percentage well below the JVM direct memory to account for the cases where Drill does not manage memory, such as for the less memory intensive operators. </p>
<ul>
<li><p>The percentage is calculated using the following formula: </p>
diff --git a/feed.xml b/feed.xml
index 3a87690..0e23b7d 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,8 +6,8 @@
</description>
<link>/</link>
<atom:link href="/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Wed, 05 Sep 2018 18:33:55 -0700</pubDate>
- <lastBuildDate>Wed, 05 Sep 2018 18:33:55 -0700</lastBuildDate>
+ <pubDate>Wed, 05 Sep 2018 18:48:40 -0700</pubDate>
+ <lastBuildDate>Wed, 05 Sep 2018 18:48:40 -0700</lastBuildDate>
<generator>Jekyll v2.5.2</generator>
<item>