You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jr...@apache.org on 2017/07/12 07:17:48 UTC
[3/6] incubator-impala git commit: Add Impala 2.9 docs from master
branch, with commit hash f1a3d8e14dae4948ce77e2f85e036d83f2d8b246
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ae2f8d03/docs/build/html/topics/impala_float.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_float.html b/docs/build/html/topics/impala_float.html
index 97125af..8af0d7e 100644
--- a/docs/build/html/topics/impala_float.html
+++ b/docs/build/html/topics/impala_float.html
@@ -52,6 +52,21 @@
</p>
<p class="p">
+ <strong class="ph b">Usage notes:</strong>
+ </p>
+
+
+ <p class="p">
+ Impala does not evaluate NaN (not a number) as equal to any other numeric values,
+ including other NaN values. For example, the following statement, which evaluates equality
+ between two NaN values, returns <code class="ph codeph">false</code>:
+ </p>
+
+<pre class="pre codeblock"><code>
+SELECT CAST('nan' AS FLOAT)=CAST('nan' AS FLOAT);
+</code></pre>
+
+ <p class="p">
<strong class="ph b">Examples:</strong>
</p>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ae2f8d03/docs/build/html/topics/impala_group_concat.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_group_concat.html b/docs/build/html/topics/impala_group_concat.html
index dcc0439..dfa32ac 100644
--- a/docs/build/html/topics/impala_group_concat.html
+++ b/docs/build/html/topics/impala_group_concat.html
@@ -19,9 +19,7 @@
<strong class="ph b">Syntax:</strong>
</p>
-
-
-<pre class="pre codeblock"><code>GROUP_CONCAT([ALL] <var class="keyword varname">expression</var> [, <var class="keyword varname">separator</var>])</code></pre>
+<pre class="pre codeblock"><code>GROUP_CONCAT([ALL<span class="ph"> | DISTINCT</span>] <var class="keyword varname">expression</var> [, <var class="keyword varname">separator</var>])</code></pre>
<p class="p">
<strong class="ph b">Usage notes:</strong> <code class="ph codeph">concat()</code> and <code class="ph codeph">concat_ws()</code> are appropriate for
@@ -40,14 +38,6 @@
</p>
<p class="p">
- <strong class="ph b">Restrictions:</strong>
- </p>
-
- <p class="p">
- You cannot apply the <code class="ph codeph">DISTINCT</code> operator to the argument of this function.
- </p>
-
- <p class="p">
This function cannot be used in an analytic context. That is, the <code class="ph codeph">OVER()</code> clause is not allowed at all with this function.
</p>
@@ -85,6 +75,20 @@
</code></pre>
<p class="p">
+ Specify the <code class="ph codeph">DISTINCT</code> keyword to eliminate duplicate values from
+ the concatenated result:
+ </p>
+
+<pre class="pre codeblock"><code>
+[localhost:21000] > select group_concat(distinct s) from t1;
++--------------------------+
+| group_concat(distinct s) |
++--------------------------+
+| three, two, one |
++--------------------------+
+</code></pre>
+
+ <p class="p">
The optional separator lets you format the result in flexible ways. The separator can be an arbitrary string
expression, not just a single character.
</p>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ae2f8d03/docs/build/html/topics/impala_hints.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_hints.html b/docs/build/html/topics/impala_hints.html
index 3b0c81d..f2b48f0 100644
--- a/docs/build/html/topics/impala_hints.html
+++ b/docs/build/html/topics/impala_hints.html
@@ -35,26 +35,39 @@
<strong class="ph b">Syntax:</strong>
</p>
- <p class="p">
- You can represent the hints as keywords surrounded by <code class="ph codeph">[]</code> square brackets; include the
- brackets in the text of the SQL statement.
- </p>
+ <div class="p">
+ You can also represent the hints as keywords surrounded by <code class="ph codeph">[]</code>
+ square brackets; include the brackets in the text of the SQL statement.
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ The square bracket style of hint is now deprecated and might be removed in
+ a future release. For that reason, any newly added hints are not available
+ with the square bracket syntax.
+ </div>
+ </div>
<pre class="pre codeblock"><code>SELECT STRAIGHT_JOIN <var class="keyword varname">select_list</var> FROM
<var class="keyword varname">join_left_hand_table</var>
- JOIN [{BROADCAST|SHUFFLE}]
+ JOIN [{ /* +BROADCAST */ | /* +SHUFFLE */ }]
<var class="keyword varname">join_right_hand_table</var>
<var class="keyword varname">remainder_of_query</var>;
INSERT <var class="keyword varname">insert_clauses</var>
- [{SHUFFLE|NOSHUFFLE}]
+ [{ /* +SHUFFLE */ | /* +NOSHUFFLE */ }]
+ [<span class="ph">/* +CLUSTERED */</span>]
SELECT <var class="keyword varname">remainder_of_query</var>;
</code></pre>
<p class="p">
In <span class="keyword">Impala 2.0</span> and higher, you can also specify the hints inside comments that use
either the <code class="ph codeph">/* */</code> or <code class="ph codeph">--</code> notation. Specify a <code class="ph codeph">+</code> symbol
- immediately before the hint name.
+ immediately before the hint name. Recently added hints are only available using the <code class="ph codeph">/* */</code>
+ and <code class="ph codeph">--</code> notation.
+ For clarity, the <code class="ph codeph">/* */</code> and <code class="ph codeph">--</code> styles
+ are used in the syntax and examples throughout this section.
+ With the <code class="ph codeph">/* */</code> or <code class="ph codeph">--</code> notation for
+ hints, specify a <code class="ph codeph">+</code> symbol immediately before the first hint name.
+ Multiple hints can be specified separated by commas, for example
+ <code class="ph codeph">/* +clustered,shuffle */</code>
</p>
<pre class="pre codeblock"><code>SELECT STRAIGHT_JOIN <var class="keyword varname">select_list</var> FROM
@@ -76,6 +89,20 @@ INSERT <var class="keyword varname">insert_clauses</var>
INSERT <var class="keyword varname">insert_clauses</var>
-- +SHUFFLE|NOSHUFFLE
SELECT <var class="keyword varname">remainder_of_query</var>;
+
+<span class="ph">SELECT <var class="keyword varname">select_list</var> FROM
+<var class="keyword varname">table_ref</var>
+ /* +{SCHEDULE_CACHE_LOCAL | SCHEDULE_DISK_LOCAL | SCHEDULE_REMOTE}
+ [,RANDOM_REPLICA] */
+<var class="keyword varname">remainder_of_query</var>;</span>
+
+<span class="ph">INSERT <var class="keyword varname">insert_clauses</var>
+ -- +CLUSTERED
+ SELECT <var class="keyword varname">remainder_of_query</var>;
+
+INSERT <var class="keyword varname">insert_clauses</var>
+ /* +CLUSTERED */
+ SELECT <var class="keyword varname">remainder_of_query</var>;</span>
</code></pre>
<p class="p">
@@ -85,7 +112,7 @@ INSERT <var class="keyword varname">insert_clauses</var>
<p class="p">
With both forms of hint syntax, include the <code class="ph codeph">STRAIGHT_JOIN</code>
keyword immediately after the <code class="ph codeph">SELECT</code> keyword to prevent Impala from
- reordering the tables in a way that makes the hint ineffective.
+ reordering the tables in a way that makes the join-related hints ineffective.
</p>
<p class="p">
@@ -107,14 +134,14 @@ INSERT <var class="keyword varname">insert_clauses</var>
</p>
<p class="p">
- The <code class="ph codeph">[BROADCAST]</code> and <code class="ph codeph">[SHUFFLE]</code> hints control the execution strategy for join
+ The <code class="ph codeph">/* +BROADCAST */</code> and <code class="ph codeph">/* +SHUFFLE */</code> hints control the execution strategy for join
queries. Specify one of the following constructs immediately after the <code class="ph codeph">JOIN</code> keyword in a
query:
</p>
<ul class="ul">
<li class="li">
- <code class="ph codeph">[SHUFFLE]</code> - Makes that join operation use the <span class="q">"partitioned"</span> technique, which divides
+ <code class="ph codeph">/* +SHUFFLE */</code> - Makes that join operation use the <span class="q">"partitioned"</span> technique, which divides
up corresponding rows from both tables using a hashing algorithm, sending subsets of the rows to other
nodes for processing. (The keyword <code class="ph codeph">SHUFFLE</code> is used to indicate a <span class="q">"partitioned join"</span>,
because that type of join is not related to <span class="q">"partitioned tables"</span>.) Since the alternative
@@ -124,7 +151,7 @@ INSERT <var class="keyword varname">insert_clauses</var>
</li>
<li class="li">
- <code class="ph codeph">[BROADCAST]</code> - Makes that join operation use the <span class="q">"broadcast"</span> technique that sends the
+ <code class="ph codeph">/* +BROADCAST */</code> - Makes that join operation use the <span class="q">"broadcast"</span> technique that sends the
entire contents of the right-hand table to all nodes involved in processing the join. This is the default
mode of operation when table and index statistics are unavailable, so you would typically only need it if
stale metadata caused Impala to mistakenly choose a partitioned join operation. Typically, broadcast joins
@@ -142,24 +169,21 @@ INSERT <var class="keyword varname">insert_clauses</var>
the <code class="ph codeph">INSERT</code> statement to fine-tune the overall performance of the operation and its
resource usage:
<ul class="ul">
- <li class="li">
- These hints are available in Impala 1.2.2 and higher.
- </li>
<li class="li">
- You would only use these hints if an <code class="ph codeph">INSERT</code> into a partitioned Parquet table was
+ You would only use hints if an <code class="ph codeph">INSERT</code> into a partitioned Parquet table was
failing due to capacity limits, or if such an <code class="ph codeph">INSERT</code> was succeeding but with
less-than-optimal performance.
</li>
<li class="li">
- To use these hints, put the hint keyword <code class="ph codeph">[SHUFFLE]</code> or <code class="ph codeph">[NOSHUFFLE]</code>
+ To use a hint to influence the join order, put the hint keyword <code class="ph codeph">/* +SHUFFLE */</code> or <code class="ph codeph">/* +NOSHUFFLE */</code>
(including the square brackets) after the <code class="ph codeph">PARTITION</code> clause, immediately before the
<code class="ph codeph">SELECT</code> keyword.
</li>
<li class="li">
- <code class="ph codeph">[SHUFFLE]</code> selects an execution plan that minimizes the number of files being written
+ <code class="ph codeph">/* +SHUFFLE */</code> selects an execution plan that reduces the number of files being written
simultaneously to HDFS, and the number of memory buffers holding data for individual partitions. Thus
it reduces overall resource usage for the <code class="ph codeph">INSERT</code> operation, allowing some
<code class="ph codeph">INSERT</code> operations to succeed that otherwise would fail. It does involve some data
@@ -168,31 +192,74 @@ INSERT <var class="keyword varname">insert_clauses</var>
</li>
<li class="li">
- <code class="ph codeph">[NOSHUFFLE]</code> selects an execution plan that might be faster overall, but might also
+ <code class="ph codeph">/* +NOSHUFFLE */</code> selects an execution plan that might be faster overall, but might also
produce a larger number of small data files or exceed capacity limits, causing the
- <code class="ph codeph">INSERT</code> operation to fail. Use <code class="ph codeph">[SHUFFLE]</code> in cases where an
+ <code class="ph codeph">INSERT</code> operation to fail. Use <code class="ph codeph">/* +SHUFFLE */</code> in cases where an
<code class="ph codeph">INSERT</code> statement fails or runs inefficiently due to all nodes attempting to construct
data for all partitions.
</li>
<li class="li">
- Impala automatically uses the <code class="ph codeph">[SHUFFLE]</code> method if any partition key column in the
+ Impala automatically uses the <code class="ph codeph">/* +SHUFFLE */</code> method if any partition key column in the
source table, mentioned in the <code class="ph codeph">INSERT ... SELECT</code> query, does not have column
- statistics. In this case, only the <code class="ph codeph">[NOSHUFFLE]</code> hint would have any effect.
+ statistics. In this case, only the <code class="ph codeph">/* +NOSHUFFLE */</code> hint would have any effect.
</li>
<li class="li">
If column statistics are available for all partition key columns in the source table mentioned in the
- <code class="ph codeph">INSERT ... SELECT</code> query, Impala chooses whether to use the <code class="ph codeph">[SHUFFLE]</code>
- or <code class="ph codeph">[NOSHUFFLE]</code> technique based on the estimated number of distinct values in those
+ <code class="ph codeph">INSERT ... SELECT</code> query, Impala chooses whether to use the <code class="ph codeph">/* +SHUFFLE */</code>
+ or <code class="ph codeph">/* +NOSHUFFLE */</code> technique based on the estimated number of distinct values in those
columns and the number of nodes involved in the <code class="ph codeph">INSERT</code> operation. In this case, you
- might need the <code class="ph codeph">[SHUFFLE]</code> or the <code class="ph codeph">[NOSHUFFLE]</code> hint to override the
+ might need the <code class="ph codeph">/* +SHUFFLE */</code> or the <code class="ph codeph">/* +NOSHUFFLE */</code> hint to override the
execution plan selected by Impala.
</li>
+
+ <li class="li">
+ In <span class="keyword">Impala 2.8</span> or higher, you can make the
+ <code class="ph codeph">INSERT</code> operation organize (<span class="q">"cluster"</span>)
+ the data for each partition to avoid buffering data for multiple partitions
+ and reduce the risk of an out-of-memory condition. Specify the hint as
+ <code class="ph codeph">/* +CLUSTERED */</code>. This technique is primarily
+ useful for inserts into Parquet tables, where the large block
+ size requires substantial memory to buffer data for multiple
+ output files at once.
+ </li>
+
</ul>
</div>
<p class="p">
+ <strong class="ph b">Hints for scheduling of HDFS blocks:</strong>
+ </p>
+
+ <p class="p">
+ The hints <code class="ph codeph">/* +SCHEDULE_CACHE_LOCAL */</code>,
+ <code class="ph codeph">/* +SCHEDULE_DISK_LOCAL */</code>, and
+ <code class="ph codeph">/* +SCHEDULE_REMOTE */</code> have the same effect
+ as specifying the <code class="ph codeph">REPLICA_PREFERENCE</code> query
+ option with the respective option settings of <code class="ph codeph">CACHE_LOCAL</code>,
+ <code class="ph codeph">DISK_LOCAL</code>, or <code class="ph codeph">REMOTE</code>.
+ The hint <code class="ph codeph">/* +RANDOM_REPLICA */</code> is the same as
+ enabling the <code class="ph codeph">SCHEDULE_RANDOM_REPLICA</code> query option.
+ </p>
+
+ <p class="p">
+ You can use these hints in combination by separating them with commas,
+ for example, <code class="ph codeph">/* +SCHEDULE_CACHE_LOCAL,RANDOM_REPLICA */</code>.
+ See <a class="xref" href="impala_replica_preference.html">REPLICA_PREFERENCE Query Option (Impala 2.7 or higher only)</a> and
+ <a class="xref" href="impala_schedule_random_replica.html">SCHEDULE_RANDOM_REPLICA Query Option (Impala 2.5 or higher only)</a> for information about how
+ these settings influence the way Impala processes HDFS data blocks.
+ </p>
+
+ <p class="p">
+ Specifying the replica preference as a query hint always overrides the
+ query option setting. Specifying either the <code class="ph codeph">SCHEDULE_RANDOM_REPLICA</code>
+ query option or the corresponding <code class="ph codeph">RANDOM_REPLICA</code> query hint
+ enables the random tie-breaking behavior when processing data blocks
+ during the query.
+ </p>
+
+ <p class="p">
<strong class="ph b">Suggestions versus directives:</strong>
</p>
@@ -228,7 +295,7 @@ INSERT <var class="keyword varname">insert_clauses</var>
</p>
<p class="p">
- In particular, the <code class="ph codeph">[BROADCAST]</code> and <code class="ph codeph">[SHUFFLE]</code> hints are expected to be
+ In particular, the <code class="ph codeph">/* +BROADCAST */</code> and <code class="ph codeph">/* +SHUFFLE */</code> hints are expected to be
needed much less frequently in Impala 1.2.2 and higher, because the join order optimization feature in
combination with the <code class="ph codeph">COMPUTE STATS</code> statement now automatically choose join order and join
mechanism without the need to rewrite the query and add hints. See
@@ -263,21 +330,21 @@ INSERT <var class="keyword varname">insert_clauses</var>
<p class="p">
For example, this query joins a large customer table with a small lookup table of less than 100 rows. The
right-hand table can be broadcast efficiently to all nodes involved in the join. Thus, you would use the
- <code class="ph codeph">[broadcast]</code> hint to force a broadcast join strategy:
+ <code class="ph codeph">/* +broadcast */</code> hint to force a broadcast join strategy:
</p>
<pre class="pre codeblock"><code>select straight_join customer.address, state_lookup.state_name
- from customer join <strong class="ph b">[broadcast]</strong> state_lookup
+ from customer join <strong class="ph b">/* +broadcast */</strong> state_lookup
on customer.state_id = state_lookup.state_id;</code></pre>
<p class="p">
This query joins two large tables of unpredictable size. You might benchmark the query with both kinds of
hints and find that it is more efficient to transmit portions of each table to other nodes for processing.
- Thus, you would use the <code class="ph codeph">[shuffle]</code> hint to force a partitioned join strategy:
+ Thus, you would use the <code class="ph codeph">/* +shuffle */</code> hint to force a partitioned join strategy:
</p>
<pre class="pre codeblock"><code>select straight_join weather.wind_velocity, geospatial.altitude
- from weather join <strong class="ph b">[shuffle]</strong> geospatial
+ from weather join <strong class="ph b">/* +shuffle */</strong> geospatial
on weather.lat = geospatial.lat and weather.long = geospatial.long;</code></pre>
<p class="p">
@@ -289,7 +356,7 @@ INSERT <var class="keyword varname">insert_clauses</var>
</p>
<pre class="pre codeblock"><code>select straight_join t1.name, t2.id, t3.price
- from t1 join <strong class="ph b">[shuffle]</strong> t2 join <strong class="ph b">[broadcast]</strong> t3
+ from t1 join <strong class="ph b">/* +shuffle */</strong> t2 join <strong class="ph b">/* +broadcast */</strong> t3
on t1.id = t2.id and t2.id = t3.id;</code></pre>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ae2f8d03/docs/build/html/topics/impala_incompatible_changes.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_incompatible_changes.html b/docs/build/html/topics/impala_incompatible_changes.html
index 642a334..9623ba1 100644
--- a/docs/build/html/topics/impala_incompatible_changes.html
+++ b/docs/build/html/topics/impala_incompatible_changes.html
@@ -23,9 +23,27 @@
<p class="p toc inpage"></p>
</div>
- <nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_release_notes.html">Impala Release Notes</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="incompatible_changes__incompatible_changes_28x">
+ <nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_release_notes.html">Impala Release Notes</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="incompatible_changes__incompatible_changes_29x">
- <h2 class="title topictitle2" id="ariaid-title2">Incompatible Changes Introduced in Impala 2.8.x</h2>
+ <h2 class="title topictitle2" id="ariaid-title2">Incompatible Changes Introduced in Impala 2.9.x</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ For the full list of issues closed in this release, including any that introduce
+ behavior changes or incompatibilities, see the
+ <a class="xref" href="https://impala.incubator.apache.org/docs/changelog-2.9.html" target="_blank">changelog for <span class="keyword">Impala 2.9</span></a>.
+ </p>
+
+
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="incompatible_changes__incompatible_changes_28x">
+
+ <h2 class="title topictitle2" id="ariaid-title3">Incompatible Changes Introduced in Impala 2.8.x</h2>
<div class="body conbody">
<ul class="ul">
@@ -99,14 +117,26 @@
The replication factor for Kudu tables must be an odd number.
</p>
</li>
+ <li class="li">
+ <p class="p">
+ A UDF compiled into an LLVM IR bitcode module (<code class="ph codeph">.bc</code>) might
+ encounter a runtime error when native code generation is turned off by
+ setting the query option <code class="ph codeph">DISABLE_CODEGEN=1</code>.
+ This issue also applies when running a built-in or native UDF with
+ more than 20 arguments.
+ See <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-4432" target="_blank">IMPALA-4432</a> for details.
+ As a workaround, either turn native code generation back on with the query option
+ <code class="ph codeph">DISABLE_CODEGEN=0</code>, or use the regular UDF compilation path
+ that does not produce an IR module.
+ </p>
+ </li>
</ul>
</div>
-
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="incompatible_changes__incompatible_changes_27x">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="incompatible_changes__incompatible_changes_27x">
- <h2 class="title topictitle2" id="ariaid-title3">Incompatible Changes Introduced in Impala 2.7.x</h2>
+ <h2 class="title topictitle2" id="ariaid-title4">Incompatible Changes Introduced in Impala 2.7.x</h2>
<div class="body conbody">
<ul class="ul">
@@ -128,9 +158,9 @@
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="incompatible_changes__incompatible_changes_26x">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title5" id="incompatible_changes__incompatible_changes_26x">
- <h2 class="title topictitle2" id="ariaid-title4">Incompatible Changes Introduced in Impala 2.6.x</h2>
+ <h2 class="title topictitle2" id="ariaid-title5">Incompatible Changes Introduced in Impala 2.6.x</h2>
<div class="body conbody">
<ul class="ul">
@@ -277,9 +307,9 @@
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title5" id="incompatible_changes__incompatible_changes_25x">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title6" id="incompatible_changes__incompatible_changes_25x">
- <h2 class="title topictitle2" id="ariaid-title5">Incompatible Changes Introduced in Impala 2.5.x</h2>
+ <h2 class="title topictitle2" id="ariaid-title6">Incompatible Changes Introduced in Impala 2.5.x</h2>
<div class="body conbody">
<ul class="ul">
@@ -365,9 +395,9 @@
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title6" id="incompatible_changes__incompatible_changes_24x">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title7" id="incompatible_changes__incompatible_changes_24x">
- <h2 class="title topictitle2" id="ariaid-title6">Incompatible Changes Introduced in Impala 2.4.x</h2>
+ <h2 class="title topictitle2" id="ariaid-title7">Incompatible Changes Introduced in Impala 2.4.x</h2>
<div class="body conbody">
<p class="p">
@@ -382,9 +412,9 @@
- <article class="topic concept nested1" aria-labelledby="ariaid-title7" id="incompatible_changes__incompatible_changes_23x">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title8" id="incompatible_changes__incompatible_changes_23x">
- <h2 class="title topictitle2" id="ariaid-title7">Incompatible Changes Introduced in Impala 2.3.x</h2>
+ <h2 class="title topictitle2" id="ariaid-title8">Incompatible Changes Introduced in Impala 2.3.x</h2>
<div class="body conbody">
@@ -452,9 +482,9 @@
- <article class="topic concept nested1" aria-labelledby="ariaid-title8" id="incompatible_changes__incompatible_changes_22x">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title9" id="incompatible_changes__incompatible_changes_22x">
- <h2 class="title topictitle2" id="ariaid-title8">Incompatible Changes Introduced in Impala 2.2.x</h2>
+ <h2 class="title topictitle2" id="ariaid-title9">Incompatible Changes Introduced in Impala 2.2.x</h2>
<div class="body conbody">
@@ -497,9 +527,9 @@
- <article class="topic concept nested1" aria-labelledby="ariaid-title9" id="incompatible_changes__incompatible_changes_21x">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title10" id="incompatible_changes__incompatible_changes_21x">
- <h2 class="title topictitle2" id="ariaid-title9">Incompatible Changes Introduced in Impala 2.1.x</h2>
+ <h2 class="title topictitle2" id="ariaid-title10">Incompatible Changes Introduced in Impala 2.1.x</h2>
<div class="body conbody">
@@ -542,9 +572,9 @@
- <article class="topic concept nested1" aria-labelledby="ariaid-title10" id="incompatible_changes__incompatible_changes_205">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title11" id="incompatible_changes__incompatible_changes_205">
- <h2 class="title topictitle2" id="ariaid-title10">Incompatible Changes Introduced in Impala 2.0.5</h2>
+ <h2 class="title topictitle2" id="ariaid-title11">Incompatible Changes Introduced in Impala 2.0.5</h2>
<div class="body conbody">
@@ -555,9 +585,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title11" id="incompatible_changes__incompatible_changes_204">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title12" id="incompatible_changes__incompatible_changes_204">
- <h2 class="title topictitle2" id="ariaid-title11">Incompatible Changes Introduced in Impala 2.0.4</h2>
+ <h2 class="title topictitle2" id="ariaid-title12">Incompatible Changes Introduced in Impala 2.0.4</h2>
<div class="body conbody">
@@ -568,18 +598,18 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title12" id="incompatible_changes__incompatible_changes_203">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title13" id="incompatible_changes__incompatible_changes_203">
- <h2 class="title topictitle2" id="ariaid-title12">Incompatible Changes Introduced in Impala 2.0.3</h2>
+ <h2 class="title topictitle2" id="ariaid-title13">Incompatible Changes Introduced in Impala 2.0.3</h2>
<div class="body conbody">
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title13" id="incompatible_changes__incompatible_changes_202">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title14" id="incompatible_changes__incompatible_changes_202">
- <h2 class="title topictitle2" id="ariaid-title13">Incompatible Changes Introduced in Impala 2.0.2</h2>
+ <h2 class="title topictitle2" id="ariaid-title14">Incompatible Changes Introduced in Impala 2.0.2</h2>
<div class="body conbody">
@@ -590,9 +620,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title14" id="incompatible_changes__incompatible_changes_201">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title15" id="incompatible_changes__incompatible_changes_201">
- <h2 class="title topictitle2" id="ariaid-title14">Incompatible Changes Introduced in Impala 2.0.1</h2>
+ <h2 class="title topictitle2" id="ariaid-title15">Incompatible Changes Introduced in Impala 2.0.1</h2>
<div class="body conbody">
@@ -629,9 +659,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title15" id="incompatible_changes__incompatible_changes_200">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title16" id="incompatible_changes__incompatible_changes_200">
- <h2 class="title topictitle2" id="ariaid-title15">Incompatible Changes Introduced in Impala 2.0.0</h2>
+ <h2 class="title topictitle2" id="ariaid-title16">Incompatible Changes Introduced in Impala 2.0.0</h2>
<div class="body conbody">
@@ -825,8 +855,8 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title16" id="incompatible_changes__incompatible_changes_144">
- <h2 class="title topictitle2" id="ariaid-title16">Incompatible Changes Introduced in Impala 1.4.4</h2>
+ <article class="topic concept nested1" aria-labelledby="ariaid-title17" id="incompatible_changes__incompatible_changes_144">
+ <h2 class="title topictitle2" id="ariaid-title17">Incompatible Changes Introduced in Impala 1.4.4</h2>
<div class="body conbody">
<p class="p">
No incompatible changes.
@@ -835,9 +865,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title17" id="incompatible_changes__incompatible_changes_143">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title18" id="incompatible_changes__incompatible_changes_143">
- <h2 class="title topictitle2" id="ariaid-title17">Incompatible Changes Introduced in Impala 1.4.3</h2>
+ <h2 class="title topictitle2" id="ariaid-title18">Incompatible Changes Introduced in Impala 1.4.3</h2>
<div class="body conbody">
@@ -849,9 +879,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title18" id="incompatible_changes__incompatible_changes_142">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title19" id="incompatible_changes__incompatible_changes_142">
- <h2 class="title topictitle2" id="ariaid-title18">Incompatible Changes Introduced in Impala 1.4.2</h2>
+ <h2 class="title topictitle2" id="ariaid-title19">Incompatible Changes Introduced in Impala 1.4.2</h2>
<div class="body conbody">
@@ -862,9 +892,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title19" id="incompatible_changes__incompatible_changes_141">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title20" id="incompatible_changes__incompatible_changes_141">
- <h2 class="title topictitle2" id="ariaid-title19">Incompatible Changes Introduced in Impala 1.4.1</h2>
+ <h2 class="title topictitle2" id="ariaid-title20">Incompatible Changes Introduced in Impala 1.4.1</h2>
<div class="body conbody">
@@ -874,9 +904,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title20" id="incompatible_changes__incompatible_changes_140">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title21" id="incompatible_changes__incompatible_changes_140">
- <h2 class="title topictitle2" id="ariaid-title20">Incompatible Changes Introduced in Impala 1.4.0</h2>
+ <h2 class="title topictitle2" id="ariaid-title21">Incompatible Changes Introduced in Impala 1.4.0</h2>
<div class="body conbody">
@@ -974,9 +1004,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title21" id="incompatible_changes__incompatible_changes_133">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title22" id="incompatible_changes__incompatible_changes_133">
- <h2 class="title topictitle2" id="ariaid-title21">Incompatible Changes Introduced in Impala 1.3.3</h2>
+ <h2 class="title topictitle2" id="ariaid-title22">Incompatible Changes Introduced in Impala 1.3.3</h2>
<div class="body conbody">
@@ -988,9 +1018,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title22" id="incompatible_changes__incompatible_changes_132">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title23" id="incompatible_changes__incompatible_changes_132">
- <h2 class="title topictitle2" id="ariaid-title22">Incompatible Changes Introduced in Impala 1.3.2</h2>
+ <h2 class="title topictitle2" id="ariaid-title23">Incompatible Changes Introduced in Impala 1.3.2</h2>
<div class="body conbody">
@@ -1001,9 +1031,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title23" id="incompatible_changes__incompatible_changes_131">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title24" id="incompatible_changes__incompatible_changes_131">
- <h2 class="title topictitle2" id="ariaid-title23">Incompatible Changes Introduced in Impala 1.3.1</h2>
+ <h2 class="title topictitle2" id="ariaid-title24">Incompatible Changes Introduced in Impala 1.3.1</h2>
<div class="body conbody">
@@ -1031,9 +1061,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title24" id="incompatible_changes__incompatible_changes_130">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title25" id="incompatible_changes__incompatible_changes_130">
- <h2 class="title topictitle2" id="ariaid-title24">Incompatible Changes Introduced in Impala 1.3.0</h2>
+ <h2 class="title topictitle2" id="ariaid-title25">Incompatible Changes Introduced in Impala 1.3.0</h2>
<div class="body conbody">
@@ -1118,9 +1148,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title25" id="incompatible_changes__incompatible_changes_124">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title26" id="incompatible_changes__incompatible_changes_124">
- <h2 class="title topictitle2" id="ariaid-title25">Incompatible Changes Introduced in Impala 1.2.4</h2>
+ <h2 class="title topictitle2" id="ariaid-title26">Incompatible Changes Introduced in Impala 1.2.4</h2>
<div class="body conbody">
@@ -1140,9 +1170,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title26" id="incompatible_changes__incompatible_changes_123">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title27" id="incompatible_changes__incompatible_changes_123">
- <h2 class="title topictitle2" id="ariaid-title26">Incompatible Changes Introduced in Impala 1.2.3</h2>
+ <h2 class="title topictitle2" id="ariaid-title27">Incompatible Changes Introduced in Impala 1.2.3</h2>
<div class="body conbody">
@@ -1154,9 +1184,9 @@
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title27" id="incompatible_changes__incompatible_changes_122">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title28" id="incompatible_changes__incompatible_changes_122">
- <h2 class="title topictitle2" id="ariaid-title27">Incompatible Changes Introduced in Impala 1.2.2</h2>
+ <h2 class="title topictitle2" id="ariaid-title28">Incompatible Changes Introduced in Impala 1.2.2</h2>
<div class="body conbody">
@@ -1222,9 +1252,9 @@ select * from `cross`;</code></pre>
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title28" id="incompatible_changes__incompatible_changes_121">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title29" id="incompatible_changes__incompatible_changes_121">
- <h2 class="title topictitle2" id="ariaid-title28">Incompatible Changes Introduced in Impala 1.2.1</h2>
+ <h2 class="title topictitle2" id="ariaid-title29">Incompatible Changes Introduced in Impala 1.2.1</h2>
<div class="body conbody">
@@ -1291,9 +1321,9 @@ select * from `cross`;</code></pre>
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title29" id="incompatible_changes__incompatible_changes_120">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title30" id="incompatible_changes__incompatible_changes_120">
- <h2 class="title topictitle2" id="ariaid-title29">Incompatible Changes Introduced in Impala 1.2.0 (Beta)</h2>
+ <h2 class="title topictitle2" id="ariaid-title30">Incompatible Changes Introduced in Impala 1.2.0 (Beta)</h2>
<div class="body conbody">
@@ -1347,9 +1377,9 @@ select * from `cross`;</code></pre>
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title30" id="incompatible_changes__incompatible_changes_111">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title31" id="incompatible_changes__incompatible_changes_111">
- <h2 class="title topictitle2" id="ariaid-title30">Incompatible Changes Introduced in Impala 1.1.1</h2>
+ <h2 class="title topictitle2" id="ariaid-title31">Incompatible Changes Introduced in Impala 1.1.1</h2>
<div class="body conbody">
@@ -1397,9 +1427,9 @@ ALTER TABLE <var class="keyword varname">table_name</var> SET FILEFORMAT
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title31" id="incompatible_changes__incompatible_changes_11">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title32" id="incompatible_changes__incompatible_changes_11">
- <h2 class="title topictitle2" id="ariaid-title31">Incompatible Change Introduced in Impala 1.1</h2>
+ <h2 class="title topictitle2" id="ariaid-title32">Incompatible Change Introduced in Impala 1.1</h2>
<div class="body conbody">
@@ -1424,9 +1454,9 @@ ALTER TABLE <var class="keyword varname">table_name</var> SET FILEFORMAT
</div>
</article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title32" id="incompatible_changes__incompatible_changes_10">
+ <article class="topic concept nested1" aria-labelledby="ariaid-title33" id="incompatible_changes__incompatible_changes_10">
- <h2 class="title topictitle2" id="ariaid-title32">Incompatible Changes Introduced in Impala 1.0</h2>
+ <h2 class="title topictitle2" id="ariaid-title33">Incompatible Changes Introduced in Impala 1.0</h2>
<div class="body conbody">
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ae2f8d03/docs/build/html/topics/impala_insert.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_insert.html b/docs/build/html/topics/impala_insert.html
index 557ab70..90a3e25 100644
--- a/docs/build/html/topics/impala_insert.html
+++ b/docs/build/html/topics/impala_insert.html
@@ -29,9 +29,25 @@ INSERT { INTO | OVERWRITE } [TABLE] <var class="keyword varname">table_name</var
partition_clause ::= <var class="keyword varname">col_name</var> [= <var class="keyword varname">constant</var>] [, <var class="keyword varname">col_name</var> [= <var class="keyword varname">constant</var>] ...]
-hint_clause ::= [SHUFFLE] | [NOSHUFFLE] (Note: the square brackets are part of the syntax.)
+hint_clause ::=
+ <var class="keyword varname">hint_with_dashes</var> |
+ <var class="keyword varname">hint_with_cstyle_delimiters</var> |
+ <var class="keyword varname">hint_with_brackets</var>
+
+hint_with_dashes ::= -- +SHUFFLE | -- +NOSHUFFLE <span class="ph">-- +CLUSTERED</span>
+
+hint_with_cstyle_comments ::= /* +SHUFFLE */ | /* +NOSHUFFLE */ <span class="ph">| /* +CLUSTERED */</span>
+
+hint_with_brackets ::= [SHUFFLE] | [NOSHUFFLE]
+ (With this hint format, the square brackets are part of the syntax.)
</code></pre>
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ The square bracket style of hint is now deprecated and might be removed in
+ a future release. For that reason, any newly added hints are not available
+ with the square bracket syntax.
+ </div>
+
<p class="p">
<strong class="ph b">Appending or replacing (INTO and OVERWRITE clauses):</strong>
</p>
@@ -739,6 +755,22 @@ Inserted 2 rows in 0.16s
<p class="p">See <a class="xref" href="impala_s3.html#s3">Using Impala with the Amazon S3 Filesystem</a> for details about reading and writing S3 data with Impala.</p>
<p class="p">
+ <strong class="ph b">ADLS considerations:</strong>
+ </p>
+ <p class="p">
+ In <span class="keyword">Impala 2.9</span> and higher, the Impala DML statements (<code class="ph codeph">INSERT</code>, <code class="ph codeph">LOAD DATA</code>,
+ and <code class="ph codeph">CREATE TABLE AS SELECT</code>) can write data into a table or partition that resides in the
+ Azure Data Lake Store (ADLS).
+ The syntax of the DML statements is the same as for any other tables, because the ADLS location for tables and
+ partitions is specified by an <code class="ph codeph">adl://</code> prefix in the
+ <code class="ph codeph">LOCATION</code> attribute of
+ <code class="ph codeph">CREATE TABLE</code> or <code class="ph codeph">ALTER TABLE</code> statements.
+ If you bring data into ADLS using the normal ADLS transfer mechanisms instead of Impala DML statements,
+ issue a <code class="ph codeph">REFRESH</code> statement for the table before using Impala to query the ADLS data.
+ </p>
+ <p class="p">See <a class="xref" href="impala_adls.html#adls">Using Impala with the Azure Data Lake Store (ADLS)</a> for details about reading and writing ADLS data with Impala.</p>
+
+ <p class="p">
<strong class="ph b">Security considerations:</strong>
</p>
<p class="p">
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ae2f8d03/docs/build/html/topics/impala_kerberos.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_kerberos.html b/docs/build/html/topics/impala_kerberos.html
index 858ff87..7c7a0b6 100644
--- a/docs/build/html/topics/impala_kerberos.html
+++ b/docs/build/html/topics/impala_kerberos.html
@@ -339,4 +339,39 @@ $ chown impala:impala impala-http.keytab</code></pre>
</div>
</article>
+ <article class="topic concept nested1" aria-labelledby="ariaid-title11" id="kerberos__kerberos_overhead_memory_usage">
+ <h2 class="title topictitle2" id="ariaid-title11">Kerberos-Related Memory Overhead for Large Clusters</h2>
+ <div class="body conbody">
+ <div class="p">
+ On a kerberized cluster with high memory utilization, <span class="keyword cmdname">kinit</span> commands executed after
+ every <code class="ph codeph">'kerberos_reinit_interval'</code> may cause out-of-memory errors, because executing
+ the command involves a fork of the Impala process. The error looks similar to the following:
+<pre class="pre codeblock"><code>
+Failed to obtain Kerberos ticket for principal: <varname>principal_details</varname>
+Failed to execute shell cmd: 'kinit -k -t <varname>keytab_details</varname>',
+error was: Error(12): Cannot allocate memory
+
+</code></pre>
+ </div>
+ <div class="p">
+ The following command changes the <code class="ph codeph">vm.overcommit_memory</code>
+ setting immediately on a running host. However, this setting is reset
+ when the host is restarted.
+<pre class="pre codeblock"><code>
+echo 1 > /proc/sys/vm/overcommit_memory
+
+</code></pre>
+ </div><div class="p">
+ To change the setting in a persistent way, add the following line to the
+ <span class="ph filepath">/etc/sysctl.conf</span> file:
+<pre class="pre codeblock"><code>
+vm.overcommit_memory=1
+
+</code></pre>
+ </div><p class="p">
+ Then run <code class="ph codeph">sysctl -p</code>. No reboot is needed.
+ </p>
+ </div>
+ </article>
+
</article></main></body></html>
\ No newline at end of file