You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ar...@apache.org on 2019/03/13 21:35:09 UTC
[impala] branch master updated: IMPALA-7718: [DOCS] Additional info
in the extended EXPLAIN output
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new ca98d66 IMPALA-7718: [DOCS] Additional info in the extended EXPLAIN output
ca98d66 is described below
commit ca98d6649d700517e90359d39c94d3356c6c092e
Author: Alex Rodoni <ar...@cloudera.com>
AuthorDate: Tue Mar 12 13:44:54 2019 -0700
IMPALA-7718: [DOCS] Additional info in the extended EXPLAIN output
Change-Id: I0ad5794d8a9b62cc7d01d023f56e700dc018f24b
Reviewed-on: http://gerrit.cloudera.org:8080/12732
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Paul Rogers <pr...@cloudera.com>
---
docs/topics/impala_explain.xml | 161 +++++++++++++++--------------------
docs/topics/impala_explain_level.xml | 27 +++---
2 files changed, 83 insertions(+), 105 deletions(-)
diff --git a/docs/topics/impala_explain.xml b/docs/topics/impala_explain.xml
index f338636..72d6661 100644
--- a/docs/topics/impala_explain.xml
+++ b/docs/topics/impala_explain.xml
@@ -39,12 +39,11 @@ under the License.
<conbody>
- <p>
- <indexterm audience="hidden">EXPLAIN statement</indexterm>
- Returns the execution plan for a statement, showing the low-level mechanisms that Impala will use to read the
- data, divide the work among nodes in the cluster, and transmit intermediate and final results across the
- network. Use <codeph>explain</codeph> followed by a complete <codeph>SELECT</codeph> query. For example:
- </p>
+ <p> Returns the execution plan for a statement, showing the low-level
+ mechanisms that Impala will use to read the data, divide the work among
+ nodes in the cluster, and transmit intermediate and final results across
+ the network. Use <codeph>explain</codeph> followed by a complete
+ <codeph>SELECT</codeph> query. For example: </p>
<p conref="../shared/impala_common.xml#common/syntax_blurb"/>
@@ -96,6 +95,14 @@ under the License.
example by using a query that affects only certain partitions within a partitioned table, then you might be
able to optimize a query so that it executes in seconds rather than minutes.
</p>
+ <p> The <codeph>EXPLAIN</codeph> output becomes more accurate and informative
+ as statistics are gathered by the <codeph>COMPUTE STATS</codeph>
+ statement. Initially, the information about data size and distribution,
+ such as the number of rows or number of distinct values for each column,
+ is marked "<codeph>unavailable</codeph>". The <codeph>COMPUTE
+ STATS</codeph> statement performs the analysis, so a subsequent
+ <codeph>EXPLAIN</codeph> statement has additional information to use in
+ deciding how to optimize the distributed query. </p>
<p>
For more information and examples to help you interpret <codeph>EXPLAIN</codeph> output, see
@@ -106,24 +113,30 @@ under the License.
<b>Extended EXPLAIN output:</b>
</p>
- <p rev="1.2">
- For performance tuning of complex queries, and capacity planning (such as using the admission control and
- resource management features), you can enable more detailed and informative output for the
- <codeph>EXPLAIN</codeph> statement. In the <cmdname>impala-shell</cmdname> interpreter, issue the command
- <codeph>SET EXPLAIN_LEVEL=<varname>level</varname></codeph>, where <varname>level</varname> is an integer
- from 0 to 3 or corresponding mnemonic values <codeph>minimal</codeph>, <codeph>standard</codeph>,
- <codeph>extended</codeph>, or <codeph>verbose</codeph>.
- </p>
-
- <p rev="1.2">
- When extended <codeph>EXPLAIN</codeph> output is enabled, <codeph>EXPLAIN</codeph> statements print
- information about estimated memory requirements, minimum number of virtual cores, and so on.
- <!--
- that you can use to fine-tune the resource management options explained in <xref href="impala_resource_management.xml#rm_options"/>.
- (The estimated memory requirements are intentionally on the high side, to allow a margin for error,
- to avoid cancelling a query unnecessarily if you set the <codeph>MEM_LIMIT</codeph> option to the estimated memory figure.)
- -->
- </p>
+ <p rev="1.2"> For performance tuning of complex queries, and capacity
+ planning (such as using the admission control and resource management
+ features), you can enable more detailed and informative output for the
+ <codeph>EXPLAIN</codeph> statement. In the
+ <cmdname>impala-shell</cmdname> interpreter, issue the command
+ <codeph>SET EXPLAIN_LEVEL=<varname>level</varname></codeph>, where
+ <varname>level</varname> is <codeph>MINIMAL</codeph>,
+ <codeph>STANDARD</codeph>, <codeph>EXTENDED</codeph>, or
+ <codeph>VERBOSE</codeph>. </p>
+
+ <p rev="1.2"> When extended <codeph>EXPLAIN</codeph> output is enabled,
+ <codeph>EXPLAIN</codeph> statements print information about estimated
+ memory requirements, minimum number of virtual cores, and so on. </p>
+ <p>Starting in <keyword keyref="impala32_full"/>, if the
+ <codeph>EXPLAIN_LEVEL</codeph> option is set to
+ <codeph>EXTENDED</codeph> level or <codeph>VERBOSE</codeph>, the output
+ contains the following additional information.</p>
+ <ul>
+ <li>The analyzed query, in the output header. <p>The analyzed query may
+ have been rewritten to include various optimizations and implicit
+ casts. See the example below.</p></li>
+ <li>The predicates in the plan output includes the same implicit casts and
+ literals printed with a cast to show the type.</li>
+ </ul>
<p>
See <xref href="impala_explain_level.xml#explain_level"/> for details and examples.
@@ -131,24 +144,28 @@ under the License.
<p conref="../shared/impala_common.xml#common/example_blurb"/>
- <p>
- This example shows how the standard <codeph>EXPLAIN</codeph> output moves from the lowest (physical) level to
- the higher (logical) levels. The query begins by scanning a certain amount of data; each node performs an
- aggregation operation (evaluating <codeph>COUNT(*)</codeph>) on some subset of data that is local to that
- node; the intermediate results are transmitted back to the coordinator node (labelled here as the
- <codeph>EXCHANGE</codeph> node); lastly, the intermediate results are summed to display the final result.
- </p>
-
-<codeblock id="explain_plan_simple">[impalad-host:21000] > explain select count(*) from customer_address;
+ <p> This example shows how the standard <codeph>EXPLAIN</codeph> output
+ moves from the lowest (physical) level to the higher (logical) levels. </p>
+ <ol>
+ <li>The query begins by scanning a certain amount of data; each node
+ performs an aggregation operation (evaluating <codeph>COUNT(*)</codeph>)
+ on some subset of data that is local to that node.</li>
+ <li>The intermediate results are transmitted back to the coordinator node
+ (labelled here as the <codeph>EXCHANGE</codeph> node).</li>
+ <li>Lastly, the intermediate results are summed to display the final
+ result. </li>
+ </ol>
+
+<codeblock id="explain_plan_simple">[impalad-host:21000] > EXPLAIN SELECT COUNT(*) FROM customer_address;
+----------------------------------------------------------+
| Explain String |
+----------------------------------------------------------+
-| Estimated Per-Host Requirements: Memory=42.00MB VCores=1 |
+| ... |
| |
-| 03:AGGREGATE [MERGE FINALIZE] |
+| 03:AGGREGATE [FINALIZE] |
| | output: sum(count(*)) |
| | |
-| 02:EXCHANGE [PARTITION=UNPARTITIONED] |
+| 02:EXCHANGE [UNPARTITIONED] |
| | |
| 01:AGGREGATE |
| | output: count(*) |
@@ -157,66 +174,24 @@ under the License.
| partitions=1/1 size=5.25MB |
+----------------------------------------------------------+
</codeblock>
-
- <p>
- These examples show how the extended <codeph>EXPLAIN</codeph> output becomes more accurate and informative as
- statistics are gathered by the <codeph>COMPUTE STATS</codeph> statement. Initially, much of the information
- about data size and distribution is marked <q>unavailable</q>. Impala can determine the raw data size, but
- not the number of rows or number of distinct values for each column without additional analysis. The
- <codeph>COMPUTE STATS</codeph> statement performs this analysis, so a subsequent <codeph>EXPLAIN</codeph>
- statement has additional information to use in deciding how to optimize the distributed query.
- </p>
-
- <!-- To do:
- Re-run these examples with more substantial tables populated with data.
- -->
-
-<codeblock rev="1.2">[localhost:21000] > set explain_level=extended;
-EXPLAIN_LEVEL set to extended
-[localhost:21000] > explain select x from t1;
-[localhost:21000] > explain select x from t1;
+ <p>The following example shows an extended <codeph>EXPLAIN</codeph> output.
+ Note that the analyzed query was rewritten to include:</p>
+ <ul>
+ <li>The 'constant folding' optimization, which simplified the expression
+ in the original query, '<codeph>1000 / 100</codeph>' to
+ '<codeph>10</codeph>'.</li>
+ <li>The implicit casts in the <codeph>WHERE</codeph> clause.</li>
+ </ul>
+ <codeblock>EXPLAIN SELECT * FROM functional_kudu.alltypestiny WHERE bigint_col < 1000 / 100;
+----------------------------------------------------------+
| Explain String |
+----------------------------------------------------------+
-| Estimated Per-Host Requirements: Memory=32.00MB VCores=1 |
-| |
-| 01:EXCHANGE [PARTITION=UNPARTITIONED] |
-| | hosts=1 per-host-mem=unavailable |
-<b>| | tuple-ids=0 row-size=4B cardinality=unavailable |</b>
-| | |
-| 00:SCAN HDFS [default.t2, PARTITION=RANDOM] |
-| partitions=1/1 size=36B |
-<b>| table stats: unavailable |</b>
-<b>| column stats: unavailable |</b>
-| hosts=1 per-host-mem=32.00MB |
-<b>| tuple-ids=0 row-size=4B cardinality=unavailable |</b>
-+----------------------------------------------------------+
-</codeblock>
-
-<codeblock rev="1.2">[localhost:21000] > compute stats t1;
-+-----------------------------------------+
-| summary |
-+-----------------------------------------+
-| Updated 1 partition(s) and 1 column(s). |
-+-----------------------------------------+
-[localhost:21000] > explain select x from t1;
-+----------------------------------------------------------+
-| Explain String |
-+----------------------------------------------------------+
-| Estimated Per-Host Requirements: Memory=64.00MB VCores=1 |
-| |
-| 01:EXCHANGE [PARTITION=UNPARTITIONED] |
-| | hosts=1 per-host-mem=unavailable |
-| | tuple-ids=0 row-size=4B cardinality=0 |
-| | |
-| 00:SCAN HDFS [default.t1, PARTITION=RANDOM] |
-| partitions=1/1 size=36B |
-<b>| table stats: 0 rows total |</b>
-<b>| column stats: all |</b>
-| hosts=1 per-host-mem=64.00MB |
-<b>| tuple-ids=0 row-size=4B cardinality=0 |</b>
-+----------------------------------------------------------+
-</codeblock>
+| ...
+| Analyzed query: SELECT * FROM mytable WHERE CAST(bigint_col AS DOUBLE) < CAST(10 AS DOUBLE)
+| ...
+| 00:SCAN KUDU [functional_kudu.alltypestiny]
+| predicates: CAST(bigint_col AS DOUBLE) < CAST(10 AS DOUBLE)
+...</codeblock>
<p conref="../shared/impala_common.xml#common/security_blurb"/>
<p conref="../shared/impala_common.xml#common/redaction_yes"/>
diff --git a/docs/topics/impala_explain_level.xml b/docs/topics/impala_explain_level.xml
index ecd06b7..e44677c 100644
--- a/docs/topics/impala_explain_level.xml
+++ b/docs/topics/impala_explain_level.xml
@@ -38,14 +38,14 @@ under the License.
<conbody>
- <p>
- <indexterm audience="hidden">EXPLAIN_LEVEL query option</indexterm>
- Controls the amount of detail provided in the output of the <codeph>EXPLAIN</codeph> statement. The basic
- output can help you identify high-level performance issues such as scanning a higher volume of data or more
- partitions than you expect. The higher levels of detail show how intermediate results flow between nodes and
- how different SQL operations such as <codeph>ORDER BY</codeph>, <codeph>GROUP BY</codeph>, joins, and
- <codeph>WHERE</codeph> clauses are implemented within a distributed query.
- </p>
+ <p> Controls the amount of detail provided in the output of the
+ <codeph>EXPLAIN</codeph> statement. The basic output can help you
+ identify high-level performance issues such as scanning a higher volume of
+ data or more partitions than you expect. The higher levels of detail show
+ how intermediate results flow between nodes and how different SQL
+ operations such as <codeph>ORDER BY</codeph>, <codeph>GROUP BY</codeph>,
+ joins, and <codeph>WHERE</codeph> clauses are implemented within a
+ distributed query. </p>
<p>
<b>Type:</b> <codeph>STRING</codeph> or <codeph>INT</codeph>
@@ -76,10 +76,13 @@ under the License.
</li>
<li>
- <codeph>2</codeph> or <codeph>EXTENDED</codeph>: Includes additional detail about how the query planner
- uses statistics in its decision-making process, to understand how a query could be tuned by gathering
- statistics, using query hints, adding or removing predicates, and so on.
- </li>
+ <codeph>2</codeph> or <codeph>EXTENDED</codeph>: Includes additional
+ detail about how the query planner uses statistics in its
+ decision-making process, to understand how a query could be tuned by
+ gathering statistics, using query hints, adding or removing predicates,
+ and so on. In <keyword keyref="impala32_full"/> and higher, the output
+ also includes the analyzed query with the cast information in the output
+ header, and the implicit cast info in the Predicate section.</li>
<li>
<codeph>3</codeph> or <codeph>VERBOSE</codeph>: The maximum level of detail, showing how work is split up