You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jr...@apache.org on 2017/10/07 00:19:19 UTC
incubator-impala git commit: IMPALA-3200: [DOCS] Document user-facing
aspects of new buffer pool
Repository: incubator-impala
Updated Branches:
refs/heads/master e278ed228 -> 874a11589
IMPALA-3200: [DOCS] Document user-facing aspects of new buffer pool
In particular, the new query options:
BUFFER_POOL_LIMIT
MAX_ROW_SIZE
MIN_SPILLABLE_BUFFER_SIZE
DEFAULT_SPILLABLE_BUFFER_SIZE
Change-Id: I49323f8ffbff3e195058e88762eedbb1fcb1bc0e
Reviewed-on: http://gerrit.cloudera.org:8080/8003
Tested-by: Impala Public Jenkins
Reviewed-by: John Russell <jr...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/874a1158
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/874a1158
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/874a1158
Branch: refs/heads/master
Commit: 874a1158972566de06b4d3b355b56b18c874fd92
Parents: e278ed2
Author: John Russell <jr...@cloudera.com>
Authored: Tue Aug 15 15:59:37 2017 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Sat Oct 7 00:18:52 2017 +0000
----------------------------------------------------------------------
docs/impala.ditamap | 6 +-
docs/impala_keydefs.ditamap | 9 +-
docs/shared/impala_common.xml | 14 +-
docs/topics/impala_buffer_pool_limit.xml | 85 +++++++
.../impala_default_spillable_buffer_size.xml | 97 ++++++++
docs/topics/impala_max_row_size.xml | 231 +++++++++++++++++++
.../topics/impala_min_spillable_buffer_size.xml | 97 ++++++++
docs/topics/impala_scalability.xml | 88 ++++++-
8 files changed, 619 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/874a1158/docs/impala.ditamap
----------------------------------------------------------------------
diff --git a/docs/impala.ditamap b/docs/impala.ditamap
index 80514da..3dd5137 100644
--- a/docs/impala.ditamap
+++ b/docs/impala.ditamap
@@ -175,6 +175,7 @@ under the License.
<topicref href="topics/impala_allow_unsupported_formats.xml"/>
<topicref href="topics/impala_appx_count_distinct.xml"/>
<topicref href="topics/impala_batch_size.xml"/>
+ <topicref rev="2.10.0 IMPALA-3200" href="topics/impala_buffer_pool_limit.xml"/>
<topicref href="topics/impala_compression_codec.xml"/>
<topicref href="topics/impala_debug_action.xml"/>
<topicref rev="2.9.0 IMPALA-5381" href="topics/impala_default_join_distribution_mode.xml"/>
@@ -182,6 +183,7 @@ under the License.
<topicref audience="hidden" href="topics/impala_disable_cached_reads.xml"/>
<topicref href="topics/impala_disable_codegen.xml"/>
<topicref rev="2.9.0" href="topics/impala_decimal_v2.xml"/>
+ <topicref rev="2.10.0 IMPALA-3200" href="topics/impala_default_spillable_buffer_size.xml"/>
<topicref audience="hidden" href="topics/impala_disable_outermost_topn.xml"/>
<topicref rev="2.5.0" href="topics/impala_disable_row_runtime_filtering.xml"/>
<topicref rev="2.5.0" href="topics/impala_disable_streaming_preaggregations.xml"/>
@@ -194,9 +196,11 @@ under the License.
<topicref href="topics/impala_live_summary.xml"/>
<topicref href="topics/impala_max_errors.xml"/>
<topicref href="topics/impala_max_io_buffers.xml"/>
- <topicref href="topics/impala_max_scan_range_length.xml"/>
+ <topicref rev="2.10.0 IMPALA-3200" href="topics/impala_max_row_size.xml"/>
<topicref rev="2.5.0" href="topics/impala_max_num_runtime_filters.xml"/>
+ <topicref href="topics/impala_max_scan_range_length.xml"/>
<topicref href="topics/impala_mem_limit.xml"/>
+ <topicref rev="2.10.0 IMPALA-3200" href="topics/impala_min_spillable_buffer_size.xml"/>
<topicref rev="2.8.0" href="topics/impala_mt_dop.xml"/>
<topicref href="topics/impala_num_nodes.xml"/>
<topicref href="topics/impala_num_scanner_threads.xml"/>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/874a1158/docs/impala_keydefs.ditamap
----------------------------------------------------------------------
diff --git a/docs/impala_keydefs.ditamap b/docs/impala_keydefs.ditamap
index d92fdc5..86112e9 100644
--- a/docs/impala_keydefs.ditamap
+++ b/docs/impala_keydefs.ditamap
@@ -10561,7 +10561,9 @@ under the License.
<keydef keys="impala220"><topicmeta><keywords><keyword>Impala 2.2.0</keyword></keywords></topicmeta></keydef>
<keydef keys="impala2110"><topicmeta><keywords><keyword>Impala 2.1.10</keyword></keywords></topicmeta></keydef>
<keydef keys="impala217"><topicmeta><keywords><keyword>Impala 2.1.7</keyword></keywords></topicmeta></keydef>
- <keydef keys="impala210"><topicmeta><keywords><keyword>Impala 2.1.0</keyword></keywords></topicmeta></keydef>
+<!-- This 3-part lookup key was defined for completeness, but never referenced.
+ Commenting out to avoid a conflict with the short key for Impala 2.10. -->
+<!-- <keydef keys="impala210"><topicmeta><keywords><keyword>Impala 2.1.0</keyword></keywords></topicmeta></keydef> -->
<keydef keys="impala200"><topicmeta><keywords><keyword>Impala 2.0.0</keyword></keywords></topicmeta></keydef>
<keydef keys="impala140"><topicmeta><keywords><keyword>Impala 1.4.0</keyword></keywords></topicmeta></keydef>
<keydef keys="impala132"><topicmeta><keywords><keyword>Impala 1.3.2</keyword></keywords></topicmeta></keydef>
@@ -10755,10 +10757,12 @@ under the License.
<keydef href="topics/impala_allow_unsupported_formats.xml" keys="allow_unsupported_formats"/>
<keydef href="topics/impala_appx_count_distinct.xml" keys="appx_count_distinct"/>
<keydef href="topics/impala_batch_size.xml" keys="batch_size"/>
+ <keydef rev="2.10.0 IMPALA-3200" href="topics/impala_buffer_pool_limit.xml" keys="buffer_pool_limit"/>
<keydef href="topics/impala_compression_codec.xml" keys="compression_codec"/>
<keydef href="topics/impala_debug_action.xml" keys="debug_action"/>
<keydef href="topics/impala_default_join_distribution_mode.xml" keys="default_join_distribution_mode"/>
<keydef href="topics/impala_default_order_by_limit.xml" keys="default_order_by_limit"/>
+ <keydef rev="2.10.0 IMPALA-3200" href="topics/impala_default_spillable_buffer_size.xml" keys="default_spillable_buffer_size"/>
<keydef href="topics/impala_disable_cached_reads.xml" keys="disable_cached_reads"/>
<keydef href="topics/impala_disable_codegen.xml" keys="disable_codegen"/>
<keydef href="topics/impala_disable_outermost_topn.xml" keys="disable_outermost_topn"/>
@@ -10773,9 +10777,11 @@ under the License.
<keydef href="topics/impala_live_summary.xml" keys="live_summary"/>
<keydef href="topics/impala_max_errors.xml" keys="max_errors"/>
<keydef href="topics/impala_max_io_buffers.xml" keys="max_io_buffers"/>
+ <keydef rev="2.10.0 IMPALA-3200" href="topics/impala_max_row_size.xml" keys="max_row_size"/>
<keydef href="topics/impala_max_scan_range_length.xml" keys="max_scan_range_length"/>
<keydef href="topics/impala_max_num_runtime_filters.xml" keys="max_num_runtime_filters"/>
<keydef href="topics/impala_mem_limit.xml" keys="mem_limit"/>
+ <keydef rev="2.10.0 IMPALA-3200" href="topics/impala_min_spillable_buffer_size.xml" keys="min_spillable_buffer_size"/>
<keydef href="topics/impala_mt_dop.xml" keys="mt_dop"/>
<keydef href="topics/impala_num_nodes.xml" keys="num_nodes"/>
<keydef href="topics/impala_num_scanner_threads.xml" keys="num_scanner_threads"/>
@@ -10938,6 +10944,7 @@ under the License.
<keydef href="topics/impala_scalability.xml#scalability_catalog" keys="scalability_catalog"/>
<keydef href="topics/impala_scalability.xml#scalability_coordinator" keys="scalability_coordinator"/>
<keydef href="topics/impala_scalability.xml#statestore_scalability" keys="statestore_scalability"/>
+ <keydef href="topics/impala_scalability.xml#scalability_buffer_pool" keys="scalability_buffer_pool"/>
<keydef audience="hidden" href="topics/impala_scalability.xml#scalability_cluster_size" keys="scalability_cluster_size"/>
<keydef audience="hidden" href="topics/impala_scalability.xml#concurrent_connections" keys="concurrent_connections"/>
<keydef href="topics/impala_scalability.xml#spill_to_disk" keys="spill_to_disk"/>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/874a1158/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index 18a93de..84cc7a2 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -1322,7 +1322,6 @@ drop database temp;
<ph conref="../shared/impala_common.xml#common/cast_int_to_timestamp"/>
</p>
-
<p>
<ph id="cast_int_to_timestamp">Casting an integer or floating-point value <codeph>N</codeph> to
<codeph>TIMESTAMP</codeph> produces a value that is <codeph>N</codeph> seconds past the start of the epoch
@@ -1865,6 +1864,10 @@ explain select s from yy2 where year in (select year from yy where year between
<b>Type:</b> integer
</p>
+ <p id="default_blurb">
+ <b>Default:</b>
+ </p>
+
<p id="default_false">
<b>Default:</b> <codeph>false</codeph>
</p>
@@ -1881,6 +1884,12 @@ explain select s from yy2 where year in (select year from yy where year between
<b>Default:</b> <codeph>true</codeph> (shown as 1 in output of <codeph>SET</codeph> statement)
</p>
+ <p id="units_blurb">
+ <b>Units:</b> A numeric argument represents a size in bytes; you can also use a suffix of <codeph>m</codeph>
+ or <codeph>mb</codeph> for megabytes, or <codeph>g</codeph> or <codeph>gb</codeph> for gigabytes. If you
+ specify a value with unrecognized formats, subsequent queries fail with an error.
+ </p>
+
<p id="odd_return_type_string">
Currently, the return value is always a <codeph>STRING</codeph>. The return type is subject to change in
future releases. Always use <codeph>CAST()</codeph> to convert the result to whichever data type is
@@ -2778,6 +2787,9 @@ flight_num: INT32 SNAPPY DO:83456393 FPO:83488603 SZ:10216514/11474301
each value.
</p>
+ <p rev="2.9.0" id="added_in_2100">
+ <b>Added in:</b> <keyword keyref="impala2100"/>
+ </p>
<p rev="2.9.0" id="added_in_290">
<b>Added in:</b> <keyword keyref="impala290"/>
</p>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/874a1158/docs/topics/impala_buffer_pool_limit.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_buffer_pool_limit.xml b/docs/topics/impala_buffer_pool_limit.xml
new file mode 100644
index 0000000..7fba9b3
--- /dev/null
+++ b/docs/topics/impala_buffer_pool_limit.xml
@@ -0,0 +1,85 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="buffer_pool_limit" rev="2.10.0 IMPALA-3200">
+
+ <title>BUFFER_POOL_LIMIT Query Option</title>
+ <titlealts audience="PDF"><navtitle>BUFFER_POOL_LIMIT</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Impala Query Options"/>
+ <data name="Category" value="Querying"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ <indexterm audience="hidden">BUFFER_POOL_LIMIT query option</indexterm>
+ Defines a limit on the amount of memory that a query can allocate from the
+ internal buffer pool. The value for this limit applies to the memory on each host,
+ not the aggregate memory across the cluster. Typically not changed by users, except
+ during diagnosis of out-of-memory errors during queries.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+<!-- BUFFER_POOL_LIMIT: [0] -->
+
+ <p conref="../shared/impala_common.xml#common/default_blurb"/>
+ <p>
+ The default setting for this option is the lower of 80% of the
+ <codeph>MEM_LIMIT</codeph> setting, or the <codeph>MEM_LIMIT</codeph>
+ setting minus 100 MB.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/added_in_2100"/>
+
+ <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+ <p>
+ If queries encounter out-of-memory errors, consider decreasing the
+ <codeph>BUFFER_POOL_LIMIT</codeph> setting to less than 80% of the
+ <codeph>MEM_LIMIT setting</codeph>.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock><![CDATA[
+-- Set an absolute value.
+set buffer_pool_limit=8GB;
+
+-- Set a relative value based on the MEM_LIMIT setting.
+set buffer_pool_limit=80%;
+]]>
+</codeblock>
+
+ <p conref="../shared/impala_common.xml#common/related_info"/>
+ <p>
+ <xref keyref="default_spillable_buffer_size"/>,
+ <xref keyref="max_row_size"/>,
+ <xref keyref="min_spillable_buffer_size"/>,
+ <xref keyref="scalability_buffer_pool"/>
+ </p>
+
+ </conbody>
+</concept>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/874a1158/docs/topics/impala_default_spillable_buffer_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_default_spillable_buffer_size.xml b/docs/topics/impala_default_spillable_buffer_size.xml
new file mode 100644
index 0000000..b2bb46d
--- /dev/null
+++ b/docs/topics/impala_default_spillable_buffer_size.xml
@@ -0,0 +1,97 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="default_spillable_buffer_size" rev="2.10.0 IMPALA-3200">
+
+ <title>DEFAULT_SPILLABLE_BUFFER_SIZE Query Option</title>
+ <titlealts audience="PDF"><navtitle>DEFAULT_SPILLABLE_BUFFER_SIZE</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Impala Query Options"/>
+ <data name="Category" value="Querying"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ <indexterm audience="hidden">DEFAULT_SPILLABLE_BUFFER_SIZE query option</indexterm>
+ Specifies the default size for a memory buffer used when the
+ spill-to-disk mechanism is activated, for example for queries against
+ a large table with no statistics, or large join operations.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+<!-- DEFAULT_SPILLABLE_BUFFER_SIZE: [2097152] -->
+
+ <p conref="../shared/impala_common.xml#common/default_blurb"/>
+ <p>
+ <codeph>2097152</codeph> (2 MB)
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/units_blurb"/>
+
+ <p conref="../shared/impala_common.xml#common/added_in_2100"/>
+
+ <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+ <p>
+ This query option sets an upper bound on the size of the internal
+ buffer size that can be used during spill-to-disk operations. The
+ actual size of the buffer is chosen by the query planner.
+ </p>
+ <p>
+ If overall query performance is limited by the time needed for spilling,
+ consider increasing the <codeph>DEFAULT_SPILLABLE_BUFFER_SIZE</codeph> setting.
+ Larger buffer sizes result in Impala issuing larger I/O requests to storage
+ devices, which might result in higher throughput, particularly on rotational
+ disks.
+ </p>
+ <p>
+ The tradeoff with a large value for this setting is increased memory usage during
+ spill-to-disk operations. Reducing this value may reduce memory consumption.
+ </p>
+ <p>
+ To determine if the value for this setting is having an effect by capping the
+ spillable buffer size, you can see the buffer size chosen by the query planner for
+ a particular query. <codeph>EXPLAIN</codeph> the query while the setting
+ <codeph>EXPLAIN_LEVEL=2</codeph> is in effect.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock><![CDATA[
+set default_spillable_buffer_size=4MB;
+]]>
+</codeblock>
+
+ <p conref="../shared/impala_common.xml#common/related_info"/>
+ <p>
+ <xref keyref="buffer_pool_limit"/>,
+ <xref keyref="max_row_size"/>,
+ <xref keyref="min_spillable_buffer_size"/>,
+ <xref keyref="scalability_buffer_pool"/>
+ </p>
+
+ </conbody>
+</concept>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/874a1158/docs/topics/impala_max_row_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_max_row_size.xml b/docs/topics/impala_max_row_size.xml
new file mode 100644
index 0000000..f1da99b
--- /dev/null
+++ b/docs/topics/impala_max_row_size.xml
@@ -0,0 +1,231 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="max_row_size" rev="2.10.0 IMPALA-3200">
+
+ <title>MAX_ROW_SIZE Query Option</title>
+ <titlealts audience="PDF"><navtitle>MAX_ROW_SIZE</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Impala Query Options"/>
+ <data name="Category" value="Querying"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ <indexterm audience="hidden">MAX_ROW_SIZE query option</indexterm>
+ Ensures that Impala can process rows of at least the specified size. (Larger
+ rows might be successfully processed, but that is not guaranteed.) Applies when
+ constructing intermediate or final rows in the result set. This setting prevents
+ out-of-control memory use when accessing columns containing huge strings.
+ </p>
+
+<!-- MAX_ROW_SIZE: [524288] -->
+
+ <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+ <p conref="../shared/impala_common.xml#common/default_blurb"/>
+ <p>
+ <codeph>524288</codeph> (512 KB)
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/units_blurb"/>
+
+ <p conref="../shared/impala_common.xml#common/added_in_2100"/>
+
+ <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+ <p>
+ If a query fails because it involves rows with long strings and/or
+ many columns, causing the total row size to exceed <codeph>MAX_ROW_SIZE</codeph>
+ bytes, increase the <codeph>MAX_ROW_SIZE</codeph> setting to accommodate
+ the total bytes stored in the largest row. Examine the error messages for any
+ failed queries to see the size of the row that caused the problem.
+ </p>
+ <p>
+ Impala attempts to handle rows that exceed the <codeph>MAX_ROW_SIZE</codeph>
+ value where practical, so in many cases, queries succeed despite having rows
+ that are larger than this setting.
+ </p>
+ <p>
+ Specifying a value that is substantially higher than actually needed can cause
+ Impala to reserve more memory than is necessary to execute the query.
+ </p>
+ <p>
+ In a Hadoop cluster with highly concurrent workloads and queries that process
+ high volumes of data, traditional SQL tuning advice about minimizing wasted memory
+ is worth remembering. For example, if a table has <codeph>STRING</codeph> columns
+ where a single value might be multiple megabytes, make sure that the
+ <codeph>SELECT</codeph> lists in queries only refer to columns that are actually
+ needed in the result set, instead of using the <codeph>SELECT *</codeph> shorthand.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+ <p>
+ The following examples show the kinds of situations where it is necessary to
+ adjust the <codeph>MAX_ROW_SIZE</codeph> setting. First, we create a table
+ containing some very long values in <codeph>STRING</codeph> columns:
+ </p>
+
+<codeblock><![CDATA[
+create table big_strings (s1 string, s2 string, s3 string) stored as parquet;
+
+-- Turn off compression to more easily reason about data volume by doing SHOW TABLE STATS.
+-- Does not actually affect query success or failure, because MAX_ROW_SIZE applies when
+-- column values are materialized in memory.
+set compression_codec=none;
+set;
+...
+ MAX_ROW_SIZE: [524288]
+...
+
+-- A very small row.
+insert into big_strings values ('one', 'two', 'three');
+-- A row right around the default MAX_ROW_SIZE limit: a 500 KiB string and a 30 KiB string.
+insert into big_strings values (repeat('12345',100000), 'short', repeat('123',10000));
+-- A row that is too big if the query has to materialize both S1 and S3.
+insert into big_strings values (repeat('12345',100000), 'short', repeat('12345',100000));
+]]>
+</codeblock>
+
+ <p>
+ With the default <codeph>MAX_ROW_SIZE</codeph> setting, different queries succeed
+ or fail based on which column values have to be materialized during query processing:
+ </p>
+
+<codeblock><![CDATA[
+-- All the S1 values can be materialized within the 512 KB MAX_ROW_SIZE buffer.
+select count(distinct s1) from big_strings;
++--------------------+
+| count(distinct s1) |
++--------------------+
+| 2 |
++--------------------+
+
+-- A row where even the S1 value is too large to materialize within MAX_ROW_SIZE.
+insert into big_strings values (repeat('12345',1000000), 'short', repeat('12345',1000000));
+
+-- The 5 MiB string is too large to materialize. The message explains the size of the result
+-- set row the query is attempting to materialize.
+select count(distinct(s1)) from big_strings;
+WARNINGS: Row of size 4.77 MB could not be materialized in plan node with id 1.
+ Increase the max_row_size query option (currently 512.00 KB) to process larger rows.
+
+-- If more columns are involved, the result set row being materialized is bigger.
+select count(distinct s1, s2, s3) from big_strings;
+WARNINGS: Row of size 9.54 MB could not be materialized in plan node with id 1.
+ Increase the max_row_size query option (currently 512.00 KB) to process larger rows.
+
+-- Column S2, containing only short strings, can still be examined.
+select count(distinct(s2)) from big_strings;
++----------------------+
+| count(distinct (s2)) |
++----------------------+
+| 2 |
++----------------------+
+
+-- Queries that do not materialize the big column values are OK.
+select count(*) from big_strings;
++----------+
+| count(*) |
++----------+
+| 4 |
++----------+
+]]>
+</codeblock>
+
+ <p>
+ The following examples show how adjusting <codeph>MAX_ROW_SIZE</codeph> upward
+ allows queries involving the long string columns to succeed:
+ </p>
+
+<codeblock><![CDATA[
+-- Boosting MAX_ROW_SIZE moderately allows all S1 values to be materialized.
+set max_row_size=7mb;
+
+select count(distinct s1) from big_strings;
++--------------------+
+| count(distinct s1) |
++--------------------+
+| 3 |
++--------------------+
+
+-- But the combination of S1 + S3 strings is still too large.
+select count(distinct s1, s2, s3) from big_strings;
+WARNINGS: Row of size 9.54 MB could not be materialized in plan node with id 1. Increase the max_row_size query option (currently 7.00 MB) to process larger rows.
+
+-- Boosting MAX_ROW_SIZE to larger than the largest row in the table allows
+-- all queries to complete successfully.
+set max_row_size=12mb;
+
+select count(distinct s1, s2, s3) from big_strings;
++----------------------------+
+| count(distinct s1, s2, s3) |
++----------------------------+
+| 4 |
++----------------------------+
+]]>
+</codeblock>
+
+ <p>
+ The following examples show how to reason about appropriate values for
+ <codeph>MAX_ROW_SIZE</codeph>, based on the characteristics of the
+ columns containing the long values:
+ </p>
+
+<codeblock><![CDATA[
+-- With a large MAX_ROW_SIZE in place, we can examine the columns to
+-- understand the practical lower limit for MAX_ROW_SIZE based on the
+-- table structure and column values.
+select max(length(s1) + length(s2) + length(s3)) / 1e6 as megabytes from big_strings;
++-----------+
+| megabytes |
++-----------+
+| 10.000005 |
++-----------+
+
+-- We can also examine the 'Max Size' for each column after computing stats.
+compute stats big_strings;
+show column stats big_strings;
++--------+--------+------------------+--------+----------+-----------+
+| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+--------+------------------+--------+----------+-----------+
+| s1 | STRING | 2 | -1 | 5000000 | 2500002.5 |
+| s2 | STRING | 2 | -1 | 10 | 7.5 |
+| s3 | STRING | 2 | -1 | 5000000 | 2500005 |
++--------+--------+------------------+--------+----------+-----------+
+]]>
+</codeblock>
+
+ <p conref="../shared/impala_common.xml#common/related_info"/>
+ <p>
+ <xref keyref="buffer_pool_limit"/>,
+ <xref keyref="default_spillable_buffer_size"/>,
+ <xref keyref="min_spillable_buffer_size"/>,
+ <xref keyref="scalability_buffer_pool"/>
+ </p>
+
+ </conbody>
+</concept>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/874a1158/docs/topics/impala_min_spillable_buffer_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_min_spillable_buffer_size.xml b/docs/topics/impala_min_spillable_buffer_size.xml
new file mode 100644
index 0000000..b5dec64
--- /dev/null
+++ b/docs/topics/impala_min_spillable_buffer_size.xml
@@ -0,0 +1,97 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="min_spillable_buffer_size" rev="2.10.0 IMPALA-3200">
+
+ <title>MIN_SPILLABLE_BUFFER_SIZE Query Option</title>
+ <titlealts audience="PDF"><navtitle>MIN_SPILLABLE_BUFFER_SIZE</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Impala Query Options"/>
+ <data name="Category" value="Querying"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ <indexterm audience="hidden">MIN_SPILLABLE_BUFFER_SIZE query option</indexterm>
+ Specifies the minimum size for a memory buffer used when the
+ spill-to-disk mechanism is activated, for example for queries against
+ a large table with no statistics, or large join operations.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+<!-- MIN_SPILLABLE_BUFFER_SIZE: [65536] -->
+
+ <p conref="../shared/impala_common.xml#common/default_blurb"/>
+ <p>
+ <codeph>65536</codeph> (64 KB)
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/units_blurb"/>
+
+ <p conref="../shared/impala_common.xml#common/added_in_2100"/>
+
+ <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+ <p>
+ This query option sets a lower bound on the size of the internal
+ buffer size that can be used during spill-to-disk operations. The
+ actual size of the buffer is chosen by the query planner.
+ </p>
+ <p>
+ If overall query performance is limited by the time needed for spilling,
+ consider increasing the <codeph>MIN_SPILLABLE_BUFFER_SIZE</codeph> setting.
+ Larger buffer sizes result in Impala issuing larger I/O requests to storage
+ devices, which might result in higher throughput, particularly on rotational
+ disks.
+ </p>
+ <p>
+ The tradeoff with a large value for this setting is increased memory usage during
+ spill-to-disk operations. Reducing this value may reduce memory consumption.
+ </p>
+ <p>
+ To determine if the value for this setting is having an effect by capping the
+ spillable buffer size, you can see the buffer size chosen by the query planner for
+ a particular query. <codeph>EXPLAIN</codeph> the query while the setting
+ <codeph>EXPLAIN_LEVEL=2</codeph> is in effect.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock><![CDATA[
+set min_spillable_buffer_size=128KB;
+]]>
+</codeblock>
+
+ <p conref="../shared/impala_common.xml#common/related_info"/>
+ <p>
+ <xref keyref="buffer_pool_limit"/>,
+ <xref keyref="default_spillable_buffer_size"/>,
+ <xref keyref="max_row_size"/>,
+ <xref keyref="scalability_buffer_pool"/>
+ </p>
+
+ </conbody>
+</concept>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/874a1158/docs/topics/impala_scalability.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_scalability.xml b/docs/topics/impala_scalability.xml
index 2533e16..beb450f 100644
--- a/docs/topics/impala_scalability.xml
+++ b/docs/topics/impala_scalability.xml
@@ -417,6 +417,38 @@ Memory Usage: Additional Notes
</conbody>
</concept>
+ <concept id="scalability_buffer_pool" rev="2.10.0 IMPALA-3200">
+ <title>Effect of Buffer Pool on Memory Usage (<keyword keyref="impala210"/> and higher)</title>
+ <conbody>
+ <p>
+ The buffer pool feature, available in <keyword keyref="impala210"/> and higher, changes the
+ way Impala allocates memory during a query. Most of the memory needed is reserved at the
+ beginning of the query, avoiding cases where a query might run for a long time before failing
+ with an out-of-memory error. The actual memory estimates and memory buffers are typically
+ smaller than before, so that more queries can run concurrently or process larger volumes
+ of data than previously.
+ </p>
+ <p>
+ The buffer pool feature includes some query options that you can fine-tune:
+ <xref keyref="buffer_pool_limit"/>,
+ <xref keyref="default_spillable_buffer_size"/>,
+ <xref keyref="max_row_size"/>, and
+ <xref keyref="min_spillable_buffer_size"/>.
+ </p>
+ <p>
+ Most of the effects of the buffer pool are transparent to you as an Impala user.
+ Memory use during spilling is now steadier and more predictable, instead of
+ increasing rapidly as more data is spilled to disk. The main change from a user
+ perspective is the need to increase the <codeph>MAX_ROW_SIZE</codeph> query option
+ setting when querying tables with columns containing long strings, many columns,
+ or other combinations of factors that produce very large rows. If Impala encounters
+ rows that are too large to process with the default query option settings, the query
+ fails with an error message suggesting to increase the <codeph>MAX_ROW_SIZE</codeph>
+ setting.
+ </p>
+ </conbody>
+ </concept>
+
<concept audience="hidden" id="scalability_cluster_size">
<title>Scalability Considerations for Impala Cluster Size and Topology</title>
@@ -456,6 +488,14 @@ Memory Usage: Additional Notes
you should optimize your queries, system parameters, and hardware configuration to make this spilling a rare occurrence.
</p>
+ <note rev="2.10.0 IMPALA-3200">
+ <p>
+ In <keyword keyref="impala210"/> and higher, also see <xref keyref="scalability_buffer_pool"/> for
+ changes to Impala memory allocation that might change the details of which queries spill to disk,
+ and how much memory and disk space is involved in the spilling operation.
+ </p>
+ </note>
+
<p>
<b>What kinds of queries might spill to disk:</b>
</p>
@@ -516,6 +556,33 @@ Memory Usage: Additional Notes
<b>Memory usage for SQL operators:</b>
</p>
+ <p rev="2.10.0 IMPALA-3200">
+ In <keyword keyref="impala210_full"/> and higher, the way SQL operators such as <codeph>GROUP BY</codeph>,
+ <codeph>DISTINCT</codeph>, and joins, transition between using additional memory or activating the
+ spill-to-disk feature is changed. The memory required to spill to disk is reserved up front, and you can
+ examine it in the <codeph>EXPLAIN</codeph> plan when the <codeph>EXPLAIN_LEVEL</codeph> query option is
+ set to 2 or higher.
+ </p>
+
+ <p>
+ The infrastructure of the spilling feature affects the way the affected SQL operators, such as
+ <codeph>GROUP BY</codeph>, <codeph>DISTINCT</codeph>, and joins, use memory.
+ On each host that participates in the query, each such operator in a query requires memory
+ to store rows of data and other data structures. Impala reserves a certain amount of memory
+ up front for each operator that supports spill-to-disk that is sufficient to execute the
+ operator. If an operator accumulates more data than can fit in the reserved memory, it
+ can either reserve more memory to continue processing data in memory or start spilling
+ data to temporary scratch files on disk. Thus, operators with spill-to-disk support
+ can adapt to different memory constraints by using however much memory is available
+ to speed up execution, yet tolerate low memory conditions by spilling data to disk.
+ </p>
+
+ <p>
+ The amount data depends on the portion of the data being handled by that host, and thus
+ the operator may end up consuming different amounts of memory on different hosts.
+ </p>
+
+<!--
<p>
The infrastructure of the spilling feature affects the way the affected SQL operators, such as
<codeph>GROUP BY</codeph>, <codeph>DISTINCT</codeph>, and joins, use memory.
@@ -547,12 +614,15 @@ Memory Usage: Additional Notes
causing its memory requirement to rise suddenly and then flatten out. The larger the cluster, less data is processed
on any particular host, thus reducing the chance of requiring the extra memory allocation.
</p>
+-->
<p>
<b>Added in:</b> This feature was added to the <codeph>ORDER BY</codeph> clause in Impala 1.4.
This feature was extended to cover join queries, aggregation functions, and analytic
functions in Impala 2.0. The size of the memory work area required by
each operator that spills was reduced from 512 megabytes to 256 megabytes in Impala 2.2.
+ <ph rev="2.10.0 IMPALA-3200">The spilling mechanism was reworked to take advantage of the
+ Impala buffer pool feature and be more predictable and stable in <keyword keyref="impala210_full"/>.</ph>
</p>
<p>
@@ -572,8 +642,10 @@ Memory Usage: Additional Notes
<li>
The output of the <codeph>PROFILE</codeph> command in the <cmdname>impala-shell</cmdname>
interpreter. This data shows the memory usage for each host and in total across the cluster. The
- <codeph>BlockMgr.BytesWritten</codeph> counter reports how much data was written to disk during the
- query.
+ <codeph>WriteIoBytes</codeph> counter reports how much data was written to disk for each operator
+ during the query. (In <keyword keyref="impala29_full"/>, the counter was named
+ <codeph>ScratchBytesWritten</codeph>; in <keyword keyref="impala28_full"/> and earlier, it was named
+ <codeph>BytesWritten</codeph>.)
</li>
<li>
@@ -676,10 +748,16 @@ Memory Usage: Additional Notes
<p>
Issue the <codeph>PROFILE</codeph> command to get a detailed breakdown of the memory usage on each node
- during the query. The crucial part of the profile output concerning memory is the <codeph>BlockMgr</codeph>
+ during the query.
+ <!--
+ The crucial part of the profile output concerning memory is the <codeph>BlockMgr</codeph>
portion. For example, this profile shows that the query did not quite exceed the memory limit.
+ -->
</p>
+<!-- Commenting out because now stale due to changes from the buffer pool (IMPALA-3200).
+ To do: Revisit these details later if indicated by user feedback.
+
<codeblock>BlockMgr:
- BlockWritesIssued: 1
- BlockWritesOutstanding: 0
@@ -699,11 +777,11 @@ Memory Usage: Additional Notes
In this case, because the memory limit was already below any recommended value, I increased the volume of
data for the query rather than reducing the memory limit any further.
</p>
+-->
<p>
Set the <codeph>MEM_LIMIT</codeph> query option to a value that is smaller than the peak memory usage
- reported in the profile output. Do not specify a memory limit lower than about 300 MB, because with such a
- low limit, queries could fail to start for other reasons. Now try the memory-intensive query again.
+ reported in the profile output. Now try the memory-intensive query again.
</p>
<p>