You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ar...@apache.org on 2019/03/13 21:35:09 UTC

[impala] branch master updated: IMPALA-7718: [DOCS] Additional info in the extended EXPLAIN output

This is an automated email from the ASF dual-hosted git repository.

arodoni pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new ca98d66  IMPALA-7718: [DOCS] Additional info in the extended EXPLAIN output
ca98d66 is described below

commit ca98d6649d700517e90359d39c94d3356c6c092e
Author: Alex Rodoni <ar...@cloudera.com>
AuthorDate: Tue Mar 12 13:44:54 2019 -0700

    IMPALA-7718: [DOCS] Additional info in the extended EXPLAIN output
    
    Change-Id: I0ad5794d8a9b62cc7d01d023f56e700dc018f24b
    Reviewed-on: http://gerrit.cloudera.org:8080/12732
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
    Reviewed-by: Paul Rogers <pr...@cloudera.com>
---
 docs/topics/impala_explain.xml       | 161 +++++++++++++++--------------------
 docs/topics/impala_explain_level.xml |  27 +++---
 2 files changed, 83 insertions(+), 105 deletions(-)

diff --git a/docs/topics/impala_explain.xml b/docs/topics/impala_explain.xml
index f338636..72d6661 100644
--- a/docs/topics/impala_explain.xml
+++ b/docs/topics/impala_explain.xml
@@ -39,12 +39,11 @@ under the License.
 
   <conbody>
 
-    <p>
-      <indexterm audience="hidden">EXPLAIN statement</indexterm>
-      Returns the execution plan for a statement, showing the low-level mechanisms that Impala will use to read the
-      data, divide the work among nodes in the cluster, and transmit intermediate and final results across the
-      network. Use <codeph>explain</codeph> followed by a complete <codeph>SELECT</codeph> query. For example:
-    </p>
+    <p> Returns the execution plan for a statement, showing the low-level
+      mechanisms that Impala will use to read the data, divide the work among
+      nodes in the cluster, and transmit intermediate and final results across
+      the network. Use <codeph>explain</codeph> followed by a complete
+        <codeph>SELECT</codeph> query. For example: </p>
 
     <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
 
@@ -96,6 +95,14 @@ under the License.
       example by using a query that affects only certain partitions within a partitioned table, then you might be
       able to optimize a query so that it executes in seconds rather than minutes.
     </p>
+    <p> The <codeph>EXPLAIN</codeph> output becomes more accurate and informative
+      as statistics are gathered by the <codeph>COMPUTE STATS</codeph>
+      statement. Initially, the information about data size and distribution,
+      such as the number of rows or number of distinct values for each column,
+      is marked "<codeph>unavailable</codeph>". The <codeph>COMPUTE
+        STATS</codeph> statement performs the analysis, so a subsequent
+        <codeph>EXPLAIN</codeph> statement has additional information to use in
+      deciding how to optimize the distributed query. </p>
 
     <p>
       For more information and examples to help you interpret <codeph>EXPLAIN</codeph> output, see
@@ -106,24 +113,30 @@ under the License.
       <b>Extended EXPLAIN output:</b>
     </p>
 
-    <p rev="1.2">
-      For performance tuning of complex queries, and capacity planning (such as using the admission control and
-      resource management features), you can enable more detailed and informative output for the
-      <codeph>EXPLAIN</codeph> statement. In the <cmdname>impala-shell</cmdname> interpreter, issue the command
-      <codeph>SET EXPLAIN_LEVEL=<varname>level</varname></codeph>, where <varname>level</varname> is an integer
-      from 0 to 3 or corresponding mnemonic values <codeph>minimal</codeph>, <codeph>standard</codeph>,
-      <codeph>extended</codeph>, or <codeph>verbose</codeph>.
-    </p>
-
-    <p rev="1.2">
-      When extended <codeph>EXPLAIN</codeph> output is enabled, <codeph>EXPLAIN</codeph> statements print
-      information about estimated memory requirements, minimum number of virtual cores, and so on.
-      <!--
-      that you can use to fine-tune the resource management options explained in <xref href="impala_resource_management.xml#rm_options"/>.
-      (The estimated memory requirements are intentionally on the high side, to allow a margin for error,
-      to avoid cancelling a query unnecessarily if you set the <codeph>MEM_LIMIT</codeph> option to the estimated memory figure.)
-      -->
-    </p>
+    <p rev="1.2"> For performance tuning of complex queries, and capacity
+      planning (such as using the admission control and resource management
+      features), you can enable more detailed and informative output for the
+        <codeph>EXPLAIN</codeph> statement. In the
+        <cmdname>impala-shell</cmdname> interpreter, issue the command
+        <codeph>SET EXPLAIN_LEVEL=<varname>level</varname></codeph>, where
+        <varname>level</varname> is <codeph>MINIMAL</codeph>,
+        <codeph>STANDARD</codeph>, <codeph>EXTENDED</codeph>, or
+        <codeph>VERBOSE</codeph>. </p>
+
+    <p rev="1.2"> When extended <codeph>EXPLAIN</codeph> output is enabled,
+        <codeph>EXPLAIN</codeph> statements print information about estimated
+      memory requirements, minimum number of virtual cores, and so on. </p>
+    <p>Starting in <keyword keyref="impala32_full"/>, if the
+        <codeph>EXPLAIN_LEVEL</codeph> option is set to
+        <codeph>EXTENDED</codeph> level or <codeph>VERBOSE</codeph>, the output
+      contains the following additional information.</p>
+    <ul>
+      <li>The analyzed query, in the output header. <p>The analyzed query may
+          have been rewritten to include various optimizations and implicit
+          casts. See the example below.</p></li>
+      <li>The predicates in the plan output includes the same implicit casts and
+        literals printed with a cast to show the type.</li>
+    </ul>
 
     <p>
       See <xref href="impala_explain_level.xml#explain_level"/> for details and examples.
@@ -131,24 +144,28 @@ under the License.
 
     <p conref="../shared/impala_common.xml#common/example_blurb"/>
 
-    <p>
-      This example shows how the standard <codeph>EXPLAIN</codeph> output moves from the lowest (physical) level to
-      the higher (logical) levels. The query begins by scanning a certain amount of data; each node performs an
-      aggregation operation (evaluating <codeph>COUNT(*)</codeph>) on some subset of data that is local to that
-      node; the intermediate results are transmitted back to the coordinator node (labelled here as the
-      <codeph>EXCHANGE</codeph> node); lastly, the intermediate results are summed to display the final result.
-    </p>
-
-<codeblock id="explain_plan_simple">[impalad-host:21000] &gt; explain select count(*) from customer_address;
+    <p> This example shows how the standard <codeph>EXPLAIN</codeph> output
+      moves from the lowest (physical) level to the higher (logical) levels. </p>
+    <ol>
+      <li>The query begins by scanning a certain amount of data; each node
+        performs an aggregation operation (evaluating <codeph>COUNT(*)</codeph>)
+        on some subset of data that is local to that node.</li>
+      <li>The intermediate results are transmitted back to the coordinator node
+        (labelled here as the <codeph>EXCHANGE</codeph> node).</li>
+      <li>Lastly, the intermediate results are summed to display the final
+        result. </li>
+    </ol>
+
+<codeblock id="explain_plan_simple">[impalad-host:21000] &gt; EXPLAIN SELECT COUNT(*) FROM customer_address;
 +----------------------------------------------------------+
 | Explain String                                           |
 +----------------------------------------------------------+
-| Estimated Per-Host Requirements: Memory=42.00MB VCores=1 |
+| ...                                                      |
 |                                                          |
-| 03:AGGREGATE [MERGE FINALIZE]                            |
+| 03:AGGREGATE [FINALIZE]                                  |
 | |  output: sum(count(*))                                 |
 | |                                                        |
-| 02:EXCHANGE [PARTITION=UNPARTITIONED]                    |
+| 02:EXCHANGE [UNPARTITIONED]                              |
 | |                                                        |
 | 01:AGGREGATE                                             |
 | |  output: count(*)                                      |
@@ -157,66 +174,24 @@ under the License.
 |    partitions=1/1 size=5.25MB                            |
 +----------------------------------------------------------+
 </codeblock>
-
-    <p>
-      These examples show how the extended <codeph>EXPLAIN</codeph> output becomes more accurate and informative as
-      statistics are gathered by the <codeph>COMPUTE STATS</codeph> statement. Initially, much of the information
-      about data size and distribution is marked <q>unavailable</q>. Impala can determine the raw data size, but
-      not the number of rows or number of distinct values for each column without additional analysis. The
-      <codeph>COMPUTE STATS</codeph> statement performs this analysis, so a subsequent <codeph>EXPLAIN</codeph>
-      statement has additional information to use in deciding how to optimize the distributed query.
-    </p>
-
-    <!-- To do:
-      Re-run these examples with more substantial tables populated with data.
-    -->
-
-<codeblock rev="1.2">[localhost:21000] &gt; set explain_level=extended;
-EXPLAIN_LEVEL set to extended
-[localhost:21000] &gt; explain select x from t1;
-[localhost:21000] &gt; explain select x from t1;
+    <p>The following example shows an extended <codeph>EXPLAIN</codeph> output.
+      Note that the analyzed query was rewritten to include:</p>
+    <ul>
+      <li>The 'constant folding' optimization, which simplified the expression
+        in the original query, '<codeph>1000 / 100</codeph>' to
+          '<codeph>10</codeph>'.</li>
+      <li>The implicit casts in the <codeph>WHERE</codeph> clause.</li>
+    </ul>
+    <codeblock>EXPLAIN SELECT * FROM functional_kudu.alltypestiny WHERE bigint_col &lt; 1000 / 100;
 +----------------------------------------------------------+
 | Explain String                                           |
 +----------------------------------------------------------+
-| Estimated Per-Host Requirements: Memory=32.00MB VCores=1 |
-|                                                          |
-| 01:EXCHANGE [PARTITION=UNPARTITIONED]                    |
-| |  hosts=1 per-host-mem=unavailable                      |
-<b>| |  tuple-ids=0 row-size=4B cardinality=unavailable       |</b>
-| |                                                        |
-| 00:SCAN HDFS [default.t2, PARTITION=RANDOM]              |
-|    partitions=1/1 size=36B                               |
-<b>|    table stats: unavailable                              |</b>
-<b>|    column stats: unavailable                             |</b>
-|    hosts=1 per-host-mem=32.00MB                          |
-<b>|    tuple-ids=0 row-size=4B cardinality=unavailable       |</b>
-+----------------------------------------------------------+
-</codeblock>
-
-<codeblock rev="1.2">[localhost:21000] &gt; compute stats t1;
-+-----------------------------------------+
-| summary                                 |
-+-----------------------------------------+
-| Updated 1 partition(s) and 1 column(s). |
-+-----------------------------------------+
-[localhost:21000] &gt; explain select x from t1;
-+----------------------------------------------------------+
-| Explain String                                           |
-+----------------------------------------------------------+
-| Estimated Per-Host Requirements: Memory=64.00MB VCores=1 |
-|                                                          |
-| 01:EXCHANGE [PARTITION=UNPARTITIONED]                    |
-| |  hosts=1 per-host-mem=unavailable                      |
-| |  tuple-ids=0 row-size=4B cardinality=0                 |
-| |                                                        |
-| 00:SCAN HDFS [default.t1, PARTITION=RANDOM]              |
-|    partitions=1/1 size=36B                               |
-<b>|    table stats: 0 rows total                             |</b>
-<b>|    column stats: all                                     |</b>
-|    hosts=1 per-host-mem=64.00MB                          |
-<b>|    tuple-ids=0 row-size=4B cardinality=0                 |</b>
-+----------------------------------------------------------+
-</codeblock>
+| ...
+| Analyzed query: SELECT * FROM mytable WHERE CAST(bigint_col AS DOUBLE) &lt; CAST(10 AS DOUBLE)
+| ...
+| 00:SCAN KUDU [functional_kudu.alltypestiny]
+| predicates: CAST(bigint_col AS DOUBLE) &lt; CAST(10 AS DOUBLE)
+...</codeblock>
 
     <p conref="../shared/impala_common.xml#common/security_blurb"/>
     <p conref="../shared/impala_common.xml#common/redaction_yes"/>
diff --git a/docs/topics/impala_explain_level.xml b/docs/topics/impala_explain_level.xml
index ecd06b7..e44677c 100644
--- a/docs/topics/impala_explain_level.xml
+++ b/docs/topics/impala_explain_level.xml
@@ -38,14 +38,14 @@ under the License.
 
   <conbody>
 
-    <p>
-      <indexterm audience="hidden">EXPLAIN_LEVEL query option</indexterm>
-      Controls the amount of detail provided in the output of the <codeph>EXPLAIN</codeph> statement. The basic
-      output can help you identify high-level performance issues such as scanning a higher volume of data or more
-      partitions than you expect. The higher levels of detail show how intermediate results flow between nodes and
-      how different SQL operations such as <codeph>ORDER BY</codeph>, <codeph>GROUP BY</codeph>, joins, and
-      <codeph>WHERE</codeph> clauses are implemented within a distributed query.
-    </p>
+    <p> Controls the amount of detail provided in the output of the
+        <codeph>EXPLAIN</codeph> statement. The basic output can help you
+      identify high-level performance issues such as scanning a higher volume of
+      data or more partitions than you expect. The higher levels of detail show
+      how intermediate results flow between nodes and how different SQL
+      operations such as <codeph>ORDER BY</codeph>, <codeph>GROUP BY</codeph>,
+      joins, and <codeph>WHERE</codeph> clauses are implemented within a
+      distributed query. </p>
 
     <p>
       <b>Type:</b> <codeph>STRING</codeph> or <codeph>INT</codeph>
@@ -76,10 +76,13 @@ under the License.
       </li>
 
       <li>
-        <codeph>2</codeph> or <codeph>EXTENDED</codeph>: Includes additional detail about how the query planner
-        uses statistics in its decision-making process, to understand how a query could be tuned by gathering
-        statistics, using query hints, adding or removing predicates, and so on.
-      </li>
+        <codeph>2</codeph> or <codeph>EXTENDED</codeph>: Includes additional
+        detail about how the query planner uses statistics in its
+        decision-making process, to understand how a query could be tuned by
+        gathering statistics, using query hints, adding or removing predicates,
+        and so on. In <keyword keyref="impala32_full"/> and higher, the output
+        also includes the analyzed query with the cast information in the output
+        header, and the implicit cast info in the Predicate section.</li>
 
       <li>
         <codeph>3</codeph> or <codeph>VERBOSE</codeph>: The maximum level of detail, showing how work is split up