Repository: drill
Updated Branches:
refs/heads/gh-pages 505bc1425 -> bfe570561
edits to planner.memory.max_query_memory_per_node description
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/bfe57056
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/bfe57056
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/bfe57056
Branch: refs/heads/gh-pages
Commit: bfe570561e271696af1581604c5111608a0b915d
Parents: 505bc14
Author: Bridget Bevens <bb...@maprtech.com>
Authored: Thu Aug 17 15:46:31 2017 -0700
Committer: Bridget Bevens <bb...@maprtech.com>
Committed: Thu Aug 17 15:46:31 2017 -0700
----------------------------------------------------------------------
.../010-configuration-options-introduction.md | 4 ++--
...050-sort-based-and-hash-based-memory-constrained-operators.md | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/bfe57056/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md b/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
index e153c9a..65566f5 100644
--- a/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
+++ b/_docs/configure-drill/configuration-options/010-configuration-options-introduction.md
@@ -1,6 +1,6 @@
---
title: "Configuration Options Introduction"
-date: 2017-08-17 21:46:31 UTC
+date: 2017-08-17 22:46:40 UTC
parent: "Configuration Options"
---
@@ -68,7 +68,7 @@ The sys.options table lists ptions that you can set at the system or session lev
| planner.memory.enable_memory_estimation | FALSE | Toggles the state of memory estimation and re-planning of the query. When enabled, Drill conservatively estimates memory requirements and typically excludes these operators from the plan and negatively impacts performance.
|
| planner.memory.hash_agg_table_factor | 1.1 | A heuristic value for influencing the size of the hash aggregation table.
|
| planner.memory.hash_join_table_factor | 1.1 | A heuristic value for influencing the size of the hash aggregation table.
|
-| planner.memory.max_query_memory_per_node | 2147483648 bytes | Sets the maximum amount of direct memory allocated to the Sort and Hash Aggregate operators during each query on a node. This memory is split between operators. If a query plan contains multiple Sort and/or Hash Aggregate operators, the memory is divided between them. The default setting is very conservative.
|
+| planner.memory.max_query_memory_per_node | 2147483648 bytes | Sets the maximum amount of direct memory allocated to the Sort and Hash Aggregate operators during each query on a node. This memory is split between operators. If a query plan contains multiple Sort and/or Hash Aggregate operators, the memory is divided between them. The default setting is too small for queries on large data sets and should be increased.
|
| planner.memory.non_blocking_operators_memory | 64 | Extra query memory per node for non-blocking operators. This option is currently used only for memory estimation. Range: 0-2048 MB
|
| planner.memory_limit | 268435456 bytes | Defines the maximum amount of direct memory allocated to a query for planning. When multiple queries run concurrently, each query is allocated the amount of memory set by this parameter.Increase the value of this parameter and rerun the query if partition pruning failed due to insufficient memory.
|
| planner.nestedloopjoin_factor | 100 | A heuristic value for influencing the nested loop join.
|
http://git-wip-us.apache.org/repos/asf/drill/blob/bfe57056/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md
----------------------------------------------------------------------
diff --git a/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md b/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md
index 1c9de18..bb62602 100644
--- a/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md
+++ b/_docs/performance-tuning/query-plans-and-tuning/050-sort-based-and-hash-based-memory-constrained-operators.md
@@ -1,6 +1,6 @@
---
title: "Sort-Based and Hash-Based Memory-Constrained Operators"
-date: 2017-08-17 21:20:22 UTC
+date: 2017-08-17 22:46:43 UTC
parent: "Query Plans and Tuning"
---
@@ -9,7 +9,7 @@ Drill uses hash-based and sort-based operators depending on the query characteri
When planning a query with sort- and hash-based operations, Drill evaluates the available memory multiplied by a configurable reduction constant (for parallelization purposes) and then limits the operations to the maximum of this amount of memory. Drill spills data to disk if the sort and hash aggregate operations cannot be performed in memory. Alternatively, you can disable large hash operations if they do not fit in memory on your system. When disabled, Drill creates alternative plans. You can also modify the minimum hash table size, increasing the size for very large aggregations or joins when you have large amounts of memory for Drill to use. If you have large data sets, you can increase the hash table size to improve performance.
##Memory Options
-The `planner.memory.max_query_memory_per_node` option sets the maximum amount of direct memory allocated to the Sort and Hash Aggregate operators during each query on a node. The default limit is 2147483648 bytes (2GB), which is quite conservative. This memory is split between operators. If a query plan contains multiple Sort and/or Hash Aggregate operators, the memory is divided between them.
+The `planner.memory.max_query_memory_per_node` option sets the maximum amount of direct memory allocated to the Sort and Hash Aggregate operators during each query on a node. The default limit is set to 2147483648 bytes (2GB), which is too small for queries on large data sets and should be increased. This memory is split between operators. If a query plan contains multiple Sort and/or Hash Aggregate operators, the memory is divided between them.
When a query is parallelized, the number of operators is multiplied, which reduces the amount of memory given to each instance of the Sort and Hash Aggregate operators during a query. If you encounter memory issues when running queries with Sort and Hash Aggregate operators, calculate the memory requirements for your queries and the amount of available memory on each node. Based on the information, increase the value of the `planner.memory.max_query_memory_per_node` option using the ALTER SYSTEM|SESSION SET command, as shown: