You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by mi...@apache.org on 2018/12/06 23:14:39 UTC
[13/51] [partial] impala git commit: [DOCS] Impala 3.1 Docs to be
published
http://git-wip-us.apache.org/repos/asf/impala/blob/b4ad38a9/docs/build/html/topics/impala_parquet_file_size.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_parquet_file_size.html b/docs/build/html/topics/impala_parquet_file_size.html
index 4262437..7e410a6 100644
--- a/docs/build/html/topics/impala_parquet_file_size.html
+++ b/docs/build/html/topics/impala_parquet_file_size.html
@@ -1,8 +1,28 @@
+<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html
- SYSTEM "about:legacy-compat">
-<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_query_options.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="parquet_file_size"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>PARQUET_FILE_SIZE Query Option</title></head><body id="parquet_file_size"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+ PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+
+<meta name="copyright" content="(C) Copyright 2018" />
+<meta name="DC.rights.owner" content="(C) Copyright 2018" />
+<meta name="DC.Type" content="concept" />
+<meta name="DC.Title" content="PARQUET_FILE_SIZE Query Option" />
+<meta name="DC.Relation" scheme="URI" content="../topics/impala_query_options.html" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="DC.Format" content="XHTML" />
+<meta name="DC.Identifier" content="parquet_file_size" />
+<link rel="stylesheet" type="text/css" href="../commonltr.css" />
+<title>PARQUET_FILE_SIZE Query Option</title>
+</head>
+<body id="parquet_file_size">
+
<h1 class="title topictitle1" id="ariaid-title1">PARQUET_FILE_SIZE Query Option</h1>
+
@@ -13,15 +33,18 @@
Specifies the maximum size of each Parquet data file produced by Impala <code class="ph codeph">INSERT</code> statements.
</p>
+
<p class="p">
<strong class="ph b">Syntax:</strong>
</p>
+
<p class="p">
Specify the size in bytes, or with a trailing <code class="ph codeph">m</code> or <code class="ph codeph">g</code> character to indicate
megabytes or gigabytes. For example:
</p>
+
<pre class="pre codeblock"><code>-- 128 megabytes.
set PARQUET_FILE_SIZE=134217728
INSERT OVERWRITE parquet_table SELECT * FROM text_table;
@@ -39,6 +62,7 @@ INSERT OVERWRITE parquet_table SELECT * FROM text_table;
<strong class="ph b">Usage notes:</strong>
</p>
+
<p class="p">
With tables that are small or finely partitioned, the default Parquet block size (formerly 1 GB, now 256 MB
in Impala 2.0 and later) could be much larger than needed for each data file. For <code class="ph codeph">INSERT</code>
@@ -48,22 +72,27 @@ INSERT OVERWRITE parquet_table SELECT * FROM text_table;
</p>
+
<p class="p">
<strong class="ph b">Type:</strong> numeric, with optional unit specifier
</p>
- <div class="note important note_important"><span class="note__title importanttitle">Important:</span>
+
+ <div class="note important"><span class="importanttitle">Important:</span>
<p class="p">
Currently, the maximum value for this setting is 1 gigabyte (<code class="ph codeph">1g</code>).
Setting a value higher than 1 gigabyte could result in errors during
an <code class="ph codeph">INSERT</code> operation.
</p>
+
</div>
+
<p class="p">
<strong class="ph b">Default:</strong> 0 (produces files with a target size of 256 MB; files might be larger for very wide tables)
</p>
+
<p class="p">
Because ADLS does not expose the block sizes of data files the way HDFS does,
any Impala <code class="ph codeph">INSERT</code> or <code class="ph codeph">CREATE TABLE AS SELECT</code> statements
@@ -72,9 +101,11 @@ INSERT OVERWRITE parquet_table SELECT * FROM text_table;
for tables that use other file formats.)
</p>
+
<p class="p">
<strong class="ph b">Isilon considerations:</strong>
</p>
+
<div class="p">
Because the EMC Isilon storage devices use a global value for the block size
rather than a configurable value for each file, the <code class="ph codeph">PARQUET_FILE_SIZE</code>
@@ -86,10 +117,12 @@ INSERT OVERWRITE parquet_table SELECT * FROM text_table;
<pre class="pre codeblock"><code>isi hdfs settings modify --default-block-size=256MB</code></pre>
</div>
+
<p class="p">
<strong class="ph b">Related information:</strong>
</p>
+
<p class="p">
For information about the Parquet file format, and how the number and size of data files affects query
performance, see <a class="xref" href="impala_parquet.html#parquet">Using the Parquet File Format with Impala Tables</a>.
@@ -97,5 +130,12 @@ INSERT OVERWRITE parquet_table SELECT * FROM text_table;
+
</div>
-<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_query_options.html">Query Options for the SET Statement</a></div></div></nav></article></main></body></html>
\ No newline at end of file
+
+<div class="related-links">
+<div class="familylinks">
+<div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_query_options.html">Query Options for the SET Statement</a></div>
+</div>
+</div></body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/impala/blob/b4ad38a9/docs/build/html/topics/impala_parquet_read_statistics.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_parquet_read_statistics.html b/docs/build/html/topics/impala_parquet_read_statistics.html
new file mode 100644
index 0000000..947051a
--- /dev/null
+++ b/docs/build/html/topics/impala_parquet_read_statistics.html
@@ -0,0 +1,130 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html
+ PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+
+<meta name="copyright" content="(C) Copyright 2018" />
+<meta name="DC.rights.owner" content="(C) Copyright 2018" />
+<meta name="DC.Type" content="concept" />
+<meta name="DC.Title" content="PARQUET_READ_STATISTICS Query Option (Impala 2.9 or higher only)" />
+<meta name="DC.Relation" scheme="URI" content="../topics/impala_query_options.html" />
+<meta name="prodname" content="Impala" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="DC.Format" content="XHTML" />
+<meta name="DC.Identifier" content="parquet_read_statistics" />
+<link rel="stylesheet" type="text/css" href="../commonltr.css" />
+<title>PARQUET_READ_STATISTICS Query Option (Impala 2.9 or higher only)</title>
+</head>
+<body id="parquet_read_statistics">
+
+
+ <h1 class="title topictitle1" id="ariaid-title1">PARQUET_READ_STATISTICS Query Option (<span class="keyword">Impala 2.9</span> or higher only)</h1>
+
+
+
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ The <code class="ph codeph">PARQUET_READ_STATISTICS</code> query option controls whether to read
+ statistics from Parquet files and use them during query processing.
+ </p>
+
+
+ <p class="p">
+ Parquet stores min/max stats which can be used to skip reading row groups if they don't
+ qualify a certain predicate. When this query option is set to <code class="ph codeph">true</code>,
+ Impala reads the Parquet statistics and skips reading row groups that do not match the
+ conditions in the <code class="ph codeph">WHERE</code> clause.
+ </p>
+
+
+ <p class="p">
+ Impala supports filtering based on Parquet statistics:
+ </p>
+
+
+ <ul class="ul">
+ <li class="li">
+ Of the numerical types for the old version of the statistics: Boolean, Integer, Float
+ </li>
+
+
+ <li class="li">
+ Of the types for the new version of the statistics (starting in IMPALA 2.8): Boolean,
+ Integer, Float, Decimal, String, Timestamp
+ </li>
+
+
+ <li class="li">
+ For simple predicates of the forms: <code class="ph codeph"><slot> <op> <constant></code> or
+ <code class="ph codeph"><constant> <op> <slot></code>, where <code class="ph codeph"><op></code> is LT,
+ LE, GE, GT, and EQ
+ </li>
+
+ </ul>
+
+
+ <p class="p">
+ The <code class="ph codeph">PARQUET_READ_STATISTICS</code> option provides a workaround when dealing
+ with files that have corrupt Parquet statistics and unknown errors.
+ </p>
+
+
+ <p class="p">
+ In the query runtime profile output for each Impalad instance, the
+ <code class="ph codeph">NumStatsFilteredRowGroups</code> field in the SCAN node section shows the number
+ of row groups that were skipped based on Parquet statistics.
+ </p>
+
+
+ <div class="p">
+ The supported values for the query option are:
+ <ul class="ul">
+ <li class="li">
+ <code class="ph codeph">true</code> (<code class="ph codeph">1</code>): Read statistics from Parquet files and use
+ them in query processing.
+ </li>
+
+
+ <li class="li">
+ <code class="ph codeph">false</code> (<code class="ph codeph">0</code>): Do not use Parquet read statistics.
+ </li>
+
+
+ <li class="li">
+ Any other values are treated as <code class="ph codeph">false</code>.
+ </li>
+
+ </ul>
+
+ </div>
+
+
+ <p class="p">
+ <strong class="ph b">Type:</strong> Boolean
+ </p>
+
+
+ <p class="p">
+ <strong class="ph b">Default:</strong> <code class="ph codeph">true</code>
+ </p>
+
+
+ <p class="p">
+ <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.9.0</span>
+ </p>
+
+
+ </div>
+
+
+<div class="related-links">
+<div class="familylinks">
+<div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_query_options.html">Query Options for the SET Statement</a></div>
+</div>
+</div></body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/impala/blob/b4ad38a9/docs/build/html/topics/impala_partitioning.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_partitioning.html b/docs/build/html/topics/impala_partitioning.html
index 65b40f2..55a1a7f 100644
--- a/docs/build/html/topics/impala_partitioning.html
+++ b/docs/build/html/topics/impala_partitioning.html
@@ -1,9 +1,54 @@
+<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html
- SYSTEM "about:legacy-compat">
-<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version"
content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="partitioning"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Partitioning for Impala Tables</title></head><body id="partitioning"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+ PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+
+<meta name="copyright" content="(C) Copyright 2018" />
+<meta name="DC.rights.owner" content="(C) Copyright 2018" />
+<meta name="DC.Type" content="concept" />
+<meta name="DC.Title" content="Partitioning for Impala Tables" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="DC.Format" content="XHTML" />
+<meta name="DC.Identifier" content="partitioning" />
+<link rel="stylesheet" type="text/css" href="../commonltr.css" />
+<title>Partitioning for Impala Tables</title>
+</head>
+<body id="partitioning">
+
<h1 class="title topictitle1" id="ariaid-title1">Partitioning for Impala Tables</h1>
+
@@ -21,45 +66,55 @@
data to read and test.
</p>
+
<p class="p toc inpage"></p>
+
<p class="p">
See <a class="xref" href="impala_tutorial.html#tut_external_partition_data">Attaching an External Partitioned Table to an HDFS Directory Structure</a> for an example that illustrates the syntax for creating partitioned
tables, the underlying directory structure in HDFS, and how to attach a partitioned Impala external table to data files stored
elsewhere in HDFS.
</p>
+
<p class="p">
Parquet is a popular format for partitioned Impala tables because it is well suited to handle huge data volumes. See
<a class="xref" href="impala_parquet.html#parquet_performance">Query Performance for Impala Parquet Tables</a> for performance considerations for partitioned Parquet tables.
</p>
+
<p class="p">
See <a class="xref" href="impala_literals.html#null">NULL</a> for details about how <code class="ph codeph">NULL</code> values are represented in partitioned tables.
</p>
+
<p class="p">
See <a class="xref" href="impala_s3.html#s3">Using Impala with the Amazon S3 Filesystem</a> for details about setting up tables where some or all partitions reside on the Amazon Simple
Storage Service (S3).
</p>
+
</div>
- <article class="topic concept nested1" aria-labelledby="ariaid-title2" id="partitioning__partitioning_choosing">
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title2" id="partitioning_choosing">
<h2 class="title topictitle2" id="ariaid-title2">When to Use Partitioned Tables</h2>
+
<div class="body conbody">
<p class="p">
Partitioning is typically appropriate for:
</p>
+
<ul class="ul">
<li class="li">
Tables that are very large, where reading the entire data set takes an impractical amount of time.
</li>
+
<li class="li">
Tables that are always or almost always queried with conditions on the partitioning columns. In our example of a table partitioned
by year, <code class="ph codeph">SELECT COUNT(*) FROM school_records WHERE year = 1985</code> is efficient, only examining a small fraction of
@@ -68,6 +123,7 @@
based on last name, student ID, and so on without testing the year.
</li>
+
<li class="li">
Columns that have reasonable cardinality (number of different values). If a column only has a small number of values, for example
<code class="ph codeph">Male</code> or <code class="ph codeph">Female</code>, you do not gain much efficiency by eliminating only about 50% of the data to
@@ -78,27 +134,34 @@
hour and minute.)
</li>
+
<li class="li">
Data that already passes through an extract, transform, and load (ETL) pipeline. The values of the partitioning columns are
stripped from the original data files and represented by directory names, so loading data into a partitioned table involves some
sort of transformation or preprocessing.
</li>
+
</ul>
+
</div>
- </article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="partitioning__partition_sql">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title3" id="partition_sql">
<h2 class="title topictitle2" id="ariaid-title3">SQL Statements for Partitioned Tables</h2>
+
<div class="body conbody">
<p class="p">
In terms of Impala SQL syntax, partitioning affects these statements:
</p>
+
<ul class="ul">
<li class="li">
<code class="ph codeph"><a class="xref" href="impala_create_table.html#create_table">CREATE TABLE</a></code>: you specify a <code class="ph codeph">PARTITIONED
@@ -106,38 +169,48 @@
included in the main list of columns for the table.
</li>
+
<li class="li">
In <span class="keyword">Impala 2.5</span> and higher, you can also use the <code class="ph codeph">PARTITIONED BY</code> clause in a <code class="ph codeph">CREATE TABLE AS
SELECT</code> statement. This syntax lets you use a single statement to create a partitioned table, copy data into it, and
create new partitions based on the values in the inserted data.
</li>
+
<li class="li">
- <code class="ph codeph"><a class="xref" href="impala_alter_table.html#alter_table">ALTER TABLE</a></code>: you can add or drop partitions, to work with
- different portions of a huge data set. You can designate the HDFS directory that holds the data files for a specific partition.
- With data partitioned by date values, you might <span class="q">"age out"</span> data that is no longer relevant.
- <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <code class="ph codeph"><a class="xref" href="impala_alter_table.html#alter_table">ALTER
+ TABLE</a></code>: you can add or drop partitions, to work
+ with different portions of a huge data set. You can designate the HDFS
+ directory that holds the data files for a specific partition. With
+ data partitioned by date values, you might <span class="q">"age out"</span> data that is
+ no longer relevant. <div class="note note"><span class="notetitle">Note:</span>
If you are creating a partition for the first time and specifying its location, for maximum efficiency, use
a single <code class="ph codeph">ALTER TABLE</code> statement including both the <code class="ph codeph">ADD PARTITION</code> and
<code class="ph codeph">LOCATION</code> clauses, rather than separate statements with <code class="ph codeph">ADD PARTITION</code> and
<code class="ph codeph">SET LOCATION</code> clauses.
</div>
+
</li>
+
<li class="li">
- <code class="ph codeph"><a class="xref" href="impala_insert.html#insert">INSERT</a></code>: When you insert data into a partitioned table, you identify
- the partitioning columns. One or more values from each inserted row are not stored in data files, but instead determine the
- directory where that row value is stored. You can also specify which partition to load a set of data into, with <code class="ph codeph">INSERT
- OVERWRITE</code> statements; you can replace the contents of a specific partition but you cannot append data to a specific
- partition.
- <p class="p">
+ <code class="ph codeph"><a class="xref" href="impala_insert.html#insert">INSERT</a></code>:
+ When you insert data into a partitioned table, you identify the
+ partitioning columns. One or more values from each inserted row are
+ not stored in data files, but instead determine the directory where
+ that row value is stored. You can also specify which partition to load
+ a set of data into, with <code class="ph codeph">INSERT OVERWRITE</code> statements;
+ you can replace the contents of a specific partition but you cannot
+ append data to a specific partition. <p class="p">
By default, if an <code class="ph codeph">INSERT</code> statement creates any new subdirectories underneath a partitioned
table, those subdirectories are assigned default HDFS permissions for the <code class="ph codeph">impala</code> user. To
make each subdirectory have the same permissions as its parent directory in HDFS, specify the
<code class="ph codeph">--insert_inherit_permissions</code> startup option for the <span class="keyword cmdname">impalad</span> daemon.
</p>
+
</li>
+
<li class="li">
Although the syntax of the <code class="ph codeph"><a class="xref" href="impala_select.html#select">SELECT</a></code> statement is the same whether or
not the table is partitioned, the way queries interact with partitioned tables can have a dramatic impact on performance and
@@ -145,20 +218,26 @@
<a class="xref" href="impala_partitioning.html#partition_pruning">Partition Pruning for Queries</a> for details.
</li>
+
<li class="li">
In Impala 1.4 and later, there is a <code class="ph codeph">SHOW PARTITIONS</code> statement that displays information about each partition in a
table. See <a class="xref" href="impala_show.html#show">SHOW Statement</a> for details.
</li>
+
</ul>
+
</div>
- </article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="partitioning__partition_static_dynamic">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title4" id="partition_static_dynamic">
<h2 class="title topictitle2" id="ariaid-title4">Static and Dynamic Partitioning Clauses</h2>
+
<div class="body conbody">
<p class="p">
@@ -167,6 +246,7 @@
only one partition, or with an <code class="ph codeph">INSERT</code> statement that inserts all values into the same partition:
</p>
+
<pre class="pre codeblock"><code>insert into t1 <strong class="ph b">partition(x=10, y='a')</strong> select c1 from some_other_table;
</code></pre>
@@ -175,6 +255,7 @@
which partition to insert. This technique is called <dfn class="term">dynamic partitioning</dfn>:
</p>
+
<pre class="pre codeblock"><code>insert into t1 <strong class="ph b">partition(x, y='b')</strong> select c1, c2 from some_other_table;
-- Create new partition if necessary based on variable year, month, and day; insert a single value.
insert into weather <strong class="ph b">partition (year, month, day)</strong> select 'cloudy',2014,4,21;
@@ -188,14 +269,18 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
specified value.
</p>
+
</div>
- </article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title5" id="partitioning__partition_refresh">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title5" id="partition_refresh">
<h2 class="title topictitle2" id="ariaid-title5">Refreshing a Single Partition</h2>
+
<div class="body conbody">
<p class="p">
@@ -205,6 +290,7 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
<code class="ph codeph">REFRESH</code> operation for a full partitioned table can take significant time.
</p>
+
<p class="p">
In <span class="keyword">Impala 2.7</span> and higher, you can include a <code class="ph codeph">PARTITION (<var class="keyword varname">partition_spec</var>)</code> clause in the
<code class="ph codeph">REFRESH</code> statement so that only a single partition is refreshed. For example, <code class="ph codeph">REFRESH big_table PARTITION
@@ -212,14 +298,18 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
<a class="xref" href="impala_refresh.html#refresh">REFRESH Statement</a> for more details and examples of <code class="ph codeph">REFRESH</code> syntax and usage.
</p>
+
</div>
- </article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title6" id="partitioning__partition_permissions">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title6" id="partition_permissions">
<h2 class="title topictitle2" id="ariaid-title6">Permissions for Partition Subdirectories</h2>
+
<div class="body conbody">
<p class="p">
@@ -229,14 +319,18 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
<code class="ph codeph">--insert_inherit_permissions</code> startup option for the <span class="keyword cmdname">impalad</span> daemon.
</p>
+
</div>
- </article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title7" id="partitioning__partition_pruning">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title7" id="partition_pruning">
<h2 class="title topictitle2" id="ariaid-title7">Partition Pruning for Queries</h2>
+
<div class="body conbody">
<p class="p">
@@ -245,6 +339,7 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
resources and are thus proportionally faster and more scalable.
</p>
+
<p class="p">
For example, if a table is partitioned by columns <code class="ph codeph">YEAR</code>, <code class="ph codeph">MONTH</code>, and <code class="ph codeph">DAY</code>, then
<code class="ph codeph">WHERE</code> clauses such as <code class="ph codeph">WHERE year = 2013</code>, <code class="ph codeph">WHERE year < 2010</code>, or <code class="ph codeph">WHERE
@@ -253,14 +348,18 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
portion of one year.
</p>
+
<p class="p toc inpage"></p>
+
</div>
- <article class="topic concept nested2" aria-labelledby="ariaid-title8" id="partition_pruning__partition_pruning_checking">
+
+ <div class="topic concept nested2" aria-labelledby="ariaid-title8" id="partition_pruning_checking">
<h3 class="title topictitle3" id="ariaid-title8">Checking if Partition Pruning Happens for a Query</h3>
+
<div class="body conbody">
<p class="p">
@@ -270,6 +369,7 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
pruning.
</p>
+
<pre class="pre codeblock"><code>[localhost:21000] > insert into census partition (year=2010) values ('Smith'),('Jones');
[localhost:21000] > insert into census partition (year=2011) values ('Smith'),('Jones'),('Doe');
[localhost:21000] > insert into census partition (year=2012) values ('Smith'),('Doe');
@@ -306,14 +406,18 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
<code class="ph codeph">PROFILE</code> command; it includes this same summary report near the start of the profile output.
</p>
+
</div>
- </article>
- <article class="topic concept nested2" aria-labelledby="ariaid-title9" id="partition_pruning__partition_pruning_sql">
+ </div>
+
+
+ <div class="topic concept nested2" aria-labelledby="ariaid-title9" id="partition_pruning_sql">
<h3 class="title topictitle3" id="ariaid-title9">What SQL Constructs Work with Partition Pruning</h3>
+
<div class="body conbody">
<p class="p">
@@ -326,6 +430,7 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
again only reads 1 out of 3 partitions.
</p>
+
<pre class="pre codeblock"><code>[localhost:21000] > drop table census;
[localhost:21000] > create table census (name string, census_year int) partitioned by (year int);
[localhost:21000] > insert into census partition (year=2010) values ('Smith',2010),('Jones',2010);
@@ -367,6 +472,7 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
<code class="ph codeph">CREATE VIEW</code> statement were used for partition pruning.
</p>
+
<p class="p">
In queries involving both analytic functions and partitioned tables, partition pruning only occurs for columns named in the <code class="ph codeph">PARTITION BY</code>
clause of the analytic function call. For example, if an analytic function query has a clause such as <code class="ph codeph">WHERE year=2016</code>,
@@ -375,14 +481,18 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
</p>
+
</div>
- </article>
- <article class="topic concept nested2" aria-labelledby="ariaid-title10" id="partition_pruning__dynamic_partition_pruning">
+ </div>
+
+
+ <div class="topic concept nested2" aria-labelledby="ariaid-title10" id="dynamic_partition_pruning">
<h3 class="title topictitle3" id="ariaid-title10">Dynamic Partition Pruning</h3>
+
<div class="body conbody">
<p class="p">
@@ -392,11 +502,13 @@ insert into weather <strong class="ph b">partition (year=2014, month=04, day)</s
the query, and Impala prunes unnecessary partitions in ways that were impractical to predict in advance.
</p>
+
<p class="p">
For example, if partition key columns are compared to literal values in a <code class="ph codeph">WHERE</code> clause, Impala can perform static
partition pruning during the planning phase to only read the relevant partitions:
</p>
+
<pre class="pre codeblock"><code>
-- The query only needs to read 3 partitions whose key values are known ahead of time.
-- That's static partition pruning.
@@ -404,55 +516,80 @@ SELECT COUNT(*) FROM sales_table WHERE year IN (2005, 2010, 2015);
</code></pre>
<p class="p">
- Dynamic partition pruning involves using information only available at run time, such as the result of a subquery:
+ Dynamic partition pruning involves using information only available
+ at run time, such as the result of a subquery. The following example
+ shows a simple dynamic partition pruning.
</p>
+
<pre class="pre codeblock"><code>
-create table yy (s string) partitioned by (year int) stored as parquet;
-insert into yy partition (year) values ('1999', 1999), ('2000', 2000),
- ('2001', 2001), ('2010',2010);
-compute stats yy;
-
-create table yy2 (s string) partitioned by (year int) stored as parquet;
-insert into yy2 partition (year) values ('1999', 1999), ('2000', 2000),
- ('2001', 2001);
-compute stats yy2;
-
--- The query reads an unknown number of partitions, whose key values are only
--- known at run time. The 'runtime filters' lines show how the information about
--- the partitions is calculated in query fragment 02, and then used in query
--- fragment 00 to decide which partitions to skip.
-explain select s from yy2 where year in (select year from yy where year between 2000 and 2005);
-+----------------------------------------------------------+
-| Explain String |
-+----------------------------------------------------------+
-| Estimated Per-Host Requirements: Memory=16.00MB VCores=2 |
-| |
-| 04:EXCHANGE [UNPARTITIONED] |
-| | |
-| 02:HASH JOIN [LEFT SEMI JOIN, BROADCAST] |
-| | hash predicates: year = year |
-| | <strong class="ph b">runtime filters: RF000 <- year</strong> |
-| | |
-| |--03:EXCHANGE [BROADCAST] |
-| | | |
-| | 01:SCAN HDFS [dpp.yy] |
-| | partitions=2/4 files=2 size=468B |
-| | |
-| 00:SCAN HDFS [dpp.yy2] |
-| partitions=2/3 files=2 size=468B |
-| <strong class="ph b">runtime filters: RF000 -> year</strong> |
-+----------------------------------------------------------+
+CREATE TABLE yy (s STRING) PARTITIONED BY (year INT);
+INSERT INTO yy PARTITION (year) VALUES ('1999', 1999), ('2000', 2000),
+ ('2001', 2001), ('2010', 2010), ('2018', 2018);
+COMPUTE STATS yy;
+
+CREATE TABLE yy2 (s STRING, year INT);
+INSERT INTO yy2 VALUES ('1999', 1999), ('2000', 2000), ('2001', 2001);
+COMPUTE STATS yy2;
+
+-- The following query reads an unknown number of partitions, whose key values
+-- are only known at run time. The <strong class="ph b">runtime filters</strong> line shows the
+-- information used in query fragment 02 to decide which partitions to skip.
+
+EXPLAIN SELECT s FROM yy WHERE year IN (SELECT year FROM yy2);
++--------------------------------------------------------------------------+
+| PLAN-ROOT SINK |
+| | |
+| 04:EXCHANGE [UNPARTITIONED] |
+| | |
+| 02:HASH JOIN [LEFT SEMI JOIN, BROADCAST] |
+| | hash predicates: year = year |
+| | <strong class="ph b">runtime filters: RF000 <- year</strong> |
+| | |
+| |--03:EXCHANGE [BROADCAST] |
+| | | |
+| | 01:SCAN HDFS [default.yy2] |
+| | partitions=1/1 files=1 size=620B |
+| | |
+| 00:SCAN HDFS [default.yy] |
+| <strong class="ph b">partitions=5/5</strong> files=5 size=1.71KB |
+| runtime filters: RF000 -> year |
++--------------------------------------------------------------------------+
+
+SELECT s FROM yy WHERE year IN (SELECT year FROM yy2); -- Returns 3 rows from yy
+PROFILE;
</code></pre>
+ <p class="p">
+ In the above example, Impala evaluates the subquery, sends the
+ subquery results to all Impala nodes participating in the query, and
+ then each <span class="keyword cmdname">impalad</span> daemon uses the dynamic partition
+ pruning optimization to read only the partitions with the relevant key
+ values.
+ </p>
+
+
+ <p class="p">
+ The output query plan from the <code class="ph codeph">EXPLAIN</code> statement
+ shows that runtime filters are enabled. The plan also shows that it
+ expects to read all 5 partitions of the <code class="ph codeph">yy</code> table,
+ indicating that static partition pruning will not happen.
+ </p>
<p class="p">
- In this case, Impala evaluates the subquery, sends the subquery results to all Impala nodes participating in the query, and then
- each <span class="keyword cmdname">impalad</span> daemon uses the dynamic partition pruning optimization to read only the partitions with the
- relevant key values.
+ The Filter summary in the <code class="ph codeph">PROFILE</code> output shows that
+ the scan node filtered out based on a runtime filter of dynamic
+ partition pruning.
</p>
+
+<pre class="pre codeblock"><code>Filter 0 (1.00 MB):
+ - Files processed: 3
+ - <strong class="ph b">Files rejected: 1 (1)</strong>
+ - Files total: 3 (3)
+</code></pre>
+
<p class="p">
Dynamic partition pruning is especially effective for queries involving joins of several large partitioned tables. Evaluating the
<code class="ph codeph">ON</code> clauses of the join predicates might normally require reading data from all partitions of certain tables. If
@@ -461,27 +598,34 @@ explain select s from yy2 where year in (select year from yy where year between
and the amount of intermediate data stored and transmitted across the network during the query.
</p>
+
<p class="p">
When the spill-to-disk feature is activated for a join node within a query, Impala does not
produce any runtime filters for that join operation on that host. Other join nodes within
the query are not affected.
</p>
+
<p class="p">
Dynamic partition pruning is part of the runtime filtering feature, which applies to other kinds of queries in addition to queries
against partitioned tables. See <a class="xref" href="impala_runtime_filtering.html#runtime_filtering">Runtime Filtering for Impala Queries (Impala 2.5 or higher only)</a> for full details about this feature.
</p>
+
</div>
- </article>
- </article>
+ </div>
+
- <article class="topic concept nested1" aria-labelledby="ariaid-title11" id="partitioning__partition_key_columns">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title11" id="partition_key_columns">
<h2 class="title topictitle2" id="ariaid-title11">Partition Key Columns</h2>
+
<div class="body conbody">
<p class="p">
@@ -490,21 +634,26 @@ explain select s from yy2 where year in (select year from yy where year between
geographic region when the data is associated with some place.
</p>
+
<ul class="ul">
<li class="li">
<p class="p">
For time-based data, split out the separate parts into their own columns, because Impala cannot partition based on a
<code class="ph codeph">TIMESTAMP</code> column.
</p>
+
</li>
+
<li class="li">
<p class="p">
The data type of the partition columns does not have a significant effect on the storage required, because the values from those
columns are not stored in the data files, rather they are represented as strings inside HDFS directory names.
</p>
+
</li>
+
<li class="li">
<p class="p">
In <span class="keyword">Impala 2.5</span> and higher, you can enable the <code class="ph codeph">OPTIMIZE_PARTITION_KEY_SCANS</code> query option to speed up
@@ -512,15 +661,19 @@ explain select s from yy2 where year in (select year from yy where year between
default because the query behavior is slightly different if the table contains partition directories without actual data inside.
See <a class="xref" href="impala_optimize_partition_key_scans.html#optimize_partition_key_scans">OPTIMIZE_PARTITION_KEY_SCANS Query Option (Impala 2.5 or higher only)</a> for details.
</p>
+
</li>
+
<li class="li">
<p class="p">
Partitioned tables can contain complex type columns.
All the partition key columns must be scalar types.
</p>
+
</li>
+
<li class="li">
<p class="p">
Remember that when Impala queries data stored in HDFS, it is most efficient to use multi-megabyte files to take advantage of the
@@ -531,9 +684,12 @@ explain select s from yy2 where year in (select year from yy where year between
have data with a geographic component, you might partition based on postal code if you have many megabytes of data for each
postal code, but if not, you might partition by some larger region such as city, state, or country. state
</p>
+
</li>
+
</ul>
+
<p class="p">
If you frequently run aggregate functions such as <code class="ph codeph">MIN()</code>, <code class="ph codeph">MAX()</code>, and
<code class="ph codeph">COUNT(DISTINCT)</code> on partition key columns, consider enabling the <code class="ph codeph">OPTIMIZE_PARTITION_KEY_SCANS</code>
@@ -543,14 +699,18 @@ explain select s from yy2 where year in (select year from yy where year between
evaluated when this query option is enabled.
</p>
+
</div>
- </article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title12" id="partitioning__mixed_format_partitions">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title12" id="mixed_format_partitions">
<h2 class="title topictitle2" id="ariaid-title12">Setting Different File Formats for Partitions</h2>
+
<div class="body conbody">
<p class="p">
@@ -561,10 +721,12 @@ explain select s from yy2 where year in (select year from yy where year between
that use different file formats reside in separate partitions.
</p>
+
<p class="p">
For example, here is how you might switch from text to Parquet data as you receive data for different years:
</p>
+
<pre class="pre codeblock"><code>[localhost:21000] > create table census (name string) partitioned by (year smallint);
[localhost:21000] > alter table census add partition (year=2012); -- Text format;
@@ -581,6 +743,7 @@ explain select s from yy2 where year in (select year from yy where year between
produces small files that are inefficient for real-world queries.
</p>
+
<p class="p">
For other file types that Impala cannot create natively, you can switch into Hive and issue the <code class="ph codeph">ALTER TABLE ... SET
FILEFORMAT</code> statements and <code class="ph codeph">INSERT</code> or <code class="ph codeph">LOAD DATA</code> statements there. After switching back to
@@ -588,14 +751,18 @@ explain select s from yy2 where year in (select year from yy where year between
data added through Hive.
</p>
+
</div>
- </article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title13" id="partitioning__partition_management">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title13" id="partition_management">
<h2 class="title topictitle2" id="ariaid-title13">Managing Partitions</h2>
+
<div class="body conbody">
<p class="p">
@@ -605,13 +772,15 @@ explain select s from yy2 where year in (select year from yy where year between
formats.
</p>
- <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+
+ <div class="note note"><span class="notetitle">Note:</span>
If you are creating a partition for the first time and specifying its location, for maximum efficiency, use
a single <code class="ph codeph">ALTER TABLE</code> statement including both the <code class="ph codeph">ADD PARTITION</code> and
<code class="ph codeph">LOCATION</code> clauses, rather than separate statements with <code class="ph codeph">ADD PARTITION</code> and
<code class="ph codeph">SET LOCATION</code> clauses.
</div>
+
<p class="p">
What happens to the data files when a partition is dropped depends on whether the partitioned table is designated as internal or
external. For an internal (managed) table, the data files are deleted. For example, if data in the partitioned table is a copy of
@@ -622,14 +791,18 @@ explain select s from yy2 where year in (select year from yy where year between
again. See <a class="xref" href="impala_tables.html#tables">Overview of Impala Tables</a> for details and examples.
</p>
+
</div>
- </article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title14" id="partitioning__partition_kudu">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title14" id="partition_kudu">
<h2 class="title topictitle2" id="ariaid-title14">Using Partitioning with Kudu Tables</h2>
+
<div class="body conbody">
@@ -640,18 +813,23 @@ explain select s from yy2 where year in (select year from yy where year between
columns.
</p>
+
<p class="p">
See <a class="xref" href="impala_kudu.html#kudu_partitioning">Partitioning for Kudu Tables</a> for
details and examples of the partitioning techniques
for Kudu tables.
</p>
+
</div>
- </article>
- <article class="topic concept nested1" aria-labelledby="ariaid-title15" id="partitioning__partition_stats">
+ </div>
+
+
+ <div class="topic concept nested1" aria-labelledby="ariaid-title15" id="partition_stats">
<h2 class="title topictitle2" id="ariaid-title15">Keeping Statistics Up to Date for Partitioned Tables</h2>
+
<div class="body conbody">
<p class="p">
@@ -660,7 +838,8 @@ explain select s from yy2 where year in (select year from yy where year between
on a per-partition basis such that stats can be incrementally updated when new partitions are added.
</p>
- <div class="note important note_important"><span class="note__title importanttitle">Important:</span>
+
+ <div class="note important"><span class="importanttitle">Important:</span>
<p class="p">
For a particular table, use either <code class="ph codeph">COMPUTE STATS</code> or
<code class="ph codeph">COMPUTE INCREMENTAL STATS</code>, but never combine the two or
@@ -669,6 +848,7 @@ explain select s from yy2 where year in (select year from yy where year between
vice versa, drop all statistics by running <code class="ph codeph">DROP STATS</code> before
making the switch.
</p>
+
<p class="p">
When you run <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> on a table for the first time,
the statistics are computed again from scratch regardless of whether the table already
@@ -676,15 +856,20 @@ explain select s from yy2 where year in (select year from yy where year between
for scanning the entire table when running <code class="ph codeph">COMPUTE INCREMENTAL STATS</code>
for the first time on a given table.
</p>
- <p class="p">
- For a table with a huge number of partitions and many columns, the approximately 400 bytes
- of metadata per column per partition can add up to significant memory overhead, as it must
- be cached on the <span class="keyword cmdname">catalogd</span> host and on every <span class="keyword cmdname">impalad</span> host
- that is eligible to be a coordinator. If this metadata for all tables combined exceeds 2 GB,
- you might experience service downtime.
- </p>
+
+ <p class="p"> In Impala 3.0 and lower, approximately
+ 400 bytes of metadata per column per partition are needed for caching.
+ Tables with a big number of partitions and many columns can add up to a
+ significant memory overhead as the metadata must be cached on the
+ <span class="keyword cmdname">catalogd</span> host and on every
+ <span class="keyword cmdname">impalad</span> host that is eligible to be a coordinator.
+ If this metadata for all tables exceeds 2 GB, you might experience
+ service downtime. In Impala 3.1 and higher, the issue was alleviated
+ with an improved handling of incremental stats.</p>
+
</div>
+
<p class="p">
The <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> variation computes statistics only for partitions that were
added or changed since the last <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> statement, rather than the entire
@@ -693,6 +878,7 @@ explain select s from yy2 where year in (select year from yy where year between
<a class="xref" href="impala_perf_stats.html#perf_stats_incremental">impala_perf_stats.html#perf_stats_incremental</a> for full usage details.
</p>
+
<pre class="pre codeblock"><code>-- Initially the table has no incremental stats, as indicated
-- 'false' under Incremental stats.
show table stats item_partitioned;
@@ -796,6 +982,9 @@ show table stats item_partitioned;
</code></pre>
</div>
- </article>
-</article></main></body></html>
\ No newline at end of file
+ </div>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/impala/blob/b4ad38a9/docs/build/html/topics/impala_perf_benchmarking.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_perf_benchmarking.html b/docs/build/html/topics/impala_perf_benchmarking.html
index 081e026..8e8fb6e 100644
--- a/docs/build/html/topics/impala_perf_benchmarking.html
+++ b/docs/build/html/topics/impala_perf_benchmarking.html
@@ -1,8 +1,28 @@
+<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html
- SYSTEM "about:legacy-compat">
-<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_performance.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="perf_benchmarks"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Benchmarking Impala Queries</title></head><body id="perf_benchmarks"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+ PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+
+<meta name="copyright" content="(C) Copyright 2018" />
+<meta name="DC.rights.owner" content="(C) Copyright 2018" />
+<meta name="DC.Type" content="concept" />
+<meta name="DC.Title" content="Benchmarking Impala Queries" />
+<meta name="DC.Relation" scheme="URI" content="../topics/impala_performance.html" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="DC.Format" content="XHTML" />
+<meta name="DC.Identifier" content="perf_benchmarks" />
+<link rel="stylesheet" type="text/css" href="../commonltr.css" />
+<title>Benchmarking Impala Queries</title>
+</head>
+<body id="perf_benchmarks">
+
<h1 class="title topictitle1" id="ariaid-title1">Benchmarking Impala Queries</h1>
+
@@ -16,6 +36,7 @@
beyond the capacity of a single server.
</p>
+
<p class="p">
When you run queries returning large numbers of rows, the CPU time to pretty-print the output can be
substantial, giving an inaccurate measurement of the actual query time. Consider using the
@@ -23,5 +44,12 @@
optionally the <code class="ph codeph">-o</code> option to store query results in a file rather than printing to the
screen. See <a class="xref" href="impala_shell_options.html#shell_options">impala-shell Configuration Options</a> for details.
</p>
+
</div>
-<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_performance.html">Tuning Impala for Performance</a></div></div></nav></article></main></body></html>
\ No newline at end of file
+
+<div class="related-links">
+<div class="familylinks">
+<div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_performance.html">Tuning Impala for Performance</a></div>
+</div>
+</div></body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/impala/blob/b4ad38a9/docs/build/html/topics/impala_perf_cookbook.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_perf_cookbook.html b/docs/build/html/topics/impala_perf_cookbook.html
index 428dbaa..d6ea994 100644
--- a/docs/build/html/topics/impala_perf_cookbook.html
+++ b/docs/build/html/topics/impala_perf_cookbook.html
@@ -1,8 +1,28 @@
+<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html
- SYSTEM "about:legacy-compat">
-<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_performance.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.12x"><meta name="version" content="Impala 2.12x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="perf_cookbook"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Impala Performance Guidelines and Best Practices</title></head><body id="perf_cookbook"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+ PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+
+<meta name="copyright" content="(C) Copyright 2018" />
+<meta name="DC.rights.owner" content="(C) Copyright 2018" />
+<meta name="DC.Type" content="concept" />
+<meta name="DC.Title" content="Impala Performance Guidelines and Best Practices" />
+<meta name="DC.Relation" scheme="URI" content="../topics/impala_performance.html" />
+<meta name="prodname" content="Impala" />
+<meta name="prodname" content="Impala" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="version" content="Impala 3.0.x" />
+<meta name="DC.Format" content="XHTML" />
+<meta name="DC.Identifier" content="perf_cookbook" />
+<link rel="stylesheet" type="text/css" href="../commonltr.css" />
+<title>Impala Performance Guidelines and Best Practices</title>
+</head>
+<body id="perf_cookbook">
+
<h1 class="title topictitle1" id="ariaid-title1">Impala Performance Guidelines and Best Practices</h1>
+
@@ -15,9 +35,11 @@
emphasize which performance techniques typically provide the highest return on investment
</p>
+
<p class="p toc inpage"></p>
- <section class="section" id="perf_cookbook__perf_cookbook_file_format"><h2 class="title sectiontitle">Choose the appropriate file format for the data.</h2>
+
+ <div class="section" id="perf_cookbook__perf_cookbook_file_format"><h2 class="title sectiontitle">Choose the appropriate file format for the data</h2>
@@ -29,16 +51,19 @@
Parquet file format.
</p>
- <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+
+ <div class="note note"><span class="notetitle">Note:</span>
For smaller volumes of data, a few gigabytes or less for each table or partition, you might not see
significant performance differences between file formats. At small data volumes, reduced I/O from an
efficient compressed file format can be counterbalanced by reduced opportunity for parallel execution. When
planning for a production deployment or conducting benchmarks, always use realistic data volumes to get a
true picture of performance and scalability.
</div>
- </section>
- <section class="section" id="perf_cookbook__perf_cookbook_small_files"><h2 class="title sectiontitle">Avoid data ingestion processes that produce many small files.</h2>
+ </div>
+
+
+ <div class="section" id="perf_cookbook__perf_cookbook_small_files"><h2 class="title sectiontitle">Avoid data ingestion processes that produce many small files</h2>
@@ -50,6 +75,7 @@
that and skip the conversion step inside Impala.
</p>
+
<p class="p">
Always use <code class="ph codeph">INSERT ... SELECT</code> to copy significant volumes of data from table to table
within Impala. Avoid <code class="ph codeph">INSERT ... VALUES</code> for any substantial volume of data or
@@ -57,6 +83,7 @@
<a class="xref" href="impala_insert.html#insert">INSERT Statement</a> for examples of the <code class="ph codeph">INSERT ... SELECT</code> syntax.
</p>
+
<p class="p">
For example, if you have thousands of partitions in a Parquet table, each with less than
<span class="ph">256 MB</span> of data, consider partitioning in a less granular way, such as by
@@ -65,9 +92,11 @@
SELECT</code> to copy all the data to a different table; the data will be reorganized into a smaller
number of larger files by this process.
</p>
- </section>
- <section class="section" id="perf_cookbook__perf_cookbook_partitioning"><h2 class="title sectiontitle">Choose partitioning granularity based on actual data volume.</h2>
+ </div>
+
+
+ <div class="section" id="perf_cookbook__perf_cookbook_partitioning"><h2 class="title sectiontitle">Choose partitioning granularity based on actual data volume</h2>
@@ -78,6 +107,7 @@
data, potentially yielding a huge savings in disk I/O.
</p>
+
<p class="p">
When deciding which column(s) to use for partitioning, choose the right level of granularity. For example,
should you partition by year, month, and day, or only by year and month? Choose a partitioning strategy
@@ -85,11 +115,13 @@
HDFS bulk I/O and Impala distributed queries.
</p>
+
<p class="p">
Over-partitioning can also cause query planning to take longer than necessary, as Impala prunes the
unnecessary partitions. Ideally, keep the number of partitions in the table under 30 thousand.
</p>
+
<p class="p">
When preparing data files to go in a partition directory, create several large files rather than many small
ones. If you receive data in the form of many small files and have no control over the input format,
@@ -98,6 +130,7 @@
cluster).
</p>
+
<p class="p">
If you need to reduce the overall number of partitions and increase the amount of data in each partition,
first look for partition key columns that are rarely referenced or are referenced in non-critical queries
@@ -106,6 +139,7 @@
and day.
</p>
+
<p class="p">
If you need to reduce the granularity even more, consider creating <span class="q">"buckets"</span>, computed values
corresponding to different sets of partition key values. For example, you can use the
@@ -114,13 +148,16 @@
<a class="xref" href="impala_datetime_functions.html#datetime_functions">Impala Date and Time Functions</a> for details.
</p>
+
<p class="p">
See <a class="xref" href="impala_partitioning.html#partitioning">Partitioning for Impala Tables</a> for full details and performance considerations for
partitioning.
</p>
- </section>
- <section class="section" id="perf_cookbook__perf_cookbook_partition_keys"><h2 class="title sectiontitle">Use smallest appropriate integer types for partition key columns.</h2>
+ </div>
+
+
+ <div class="section" id="perf_cookbook__perf_cookbook_partition_keys"><h2 class="title sectiontitle">Use smallest appropriate integer types for partition key columns</h2>
@@ -134,9 +171,11 @@
<code class="ph codeph">TIMESTAMP</code> value, and <code class="ph codeph">CAST()</code> the return value to the appropriate integer
type.
</p>
- </section>
- <section class="section" id="perf_cookbook__perf_cookbook_parquet_block_size"><h2 class="title sectiontitle">Choose an appropriate Parquet block size.</h2>
+ </div>
+
+
+ <div class="section" id="perf_cookbook__perf_cookbook_parquet_block_size"><h2 class="title sectiontitle">Choose an appropriate Parquet block size</h2>
@@ -146,12 +185,14 @@
conservative estimates about compression, resulting in files that were smaller than 1 GB.)
</p>
+
<p class="p">
Each Parquet file written by Impala is a single block, allowing the whole file to be processed as a unit by a single host.
As you copy Parquet files into HDFS or between HDFS filesystems, use <code class="ph codeph">hdfs dfs -pb</code> to preserve the original
block size.
</p>
+
<p class="p">
If there is only one or a few data block in your Parquet table, or in a partition that is the only one
accessed by a query, then you might experience a slowdown for a different reason: not enough data to take
@@ -165,9 +206,11 @@
gigabytes.)</span> Run benchmarks with different file sizes to find the right balance point for your
particular data volume.
</p>
- </section>
- <section class="section" id="perf_cookbook__perf_cookbook_stats"><h2 class="title sectiontitle">Gather statistics for all tables used in performance-critical or high-volume join queries.</h2>
+ </div>
+
+
+ <div class="section" id="perf_cookbook__perf_cookbook_stats"><h2 class="title sectiontitle">Gather statistics for all tables used in performance-critical or high-volume join queries</h2>
@@ -175,9 +218,11 @@
Gather the statistics with the <code class="ph codeph">COMPUTE STATS</code> statement. See
<a class="xref" href="impala_perf_joins.html#perf_joins">Performance Considerations for Join Queries</a> for details.
</p>
- </section>
- <section class="section" id="perf_cookbook__perf_cookbook_network"><h2 class="title sectiontitle">Minimize the overhead of transmitting results back to the client.</h2>
+ </div>
+
+
+ <div class="section" id="perf_cookbook__perf_cookbook_network"><h2 class="title sectiontitle">Minimize the overhead of transmitting results back to the client</h2>
@@ -185,6 +230,7 @@
Use techniques such as:
</p>
+
<ul class="ul">
<li class="li">
Aggregation. If you need to know how many rows match a condition, the total values of matching values
@@ -194,11 +240,13 @@
unaggregated result set could be huge, requiring substantial time to transmit across the network.
</li>
+
<li class="li">
Filtering. Use all applicable tests in the <code class="ph codeph">WHERE</code> clause of a query to eliminate rows
that are not relevant, rather than producing a big result set and filtering it using application logic.
</li>
+
<li class="li">
<code class="ph codeph">LIMIT</code> clause. If you only need to see a few sample values from a result set, or the top
or bottom values from a query using <code class="ph codeph">ORDER BY</code>, include the <code class="ph codeph">LIMIT</code> clause
@@ -206,6 +254,7 @@
the rows away.
</li>
+
<li class="li">
Avoid overhead from pretty-printing the result set and displaying it on the screen. When you retrieve the
results through <span class="keyword cmdname">impala-shell</span>, use <span class="keyword cmdname">impala-shell</span> options such as
@@ -215,10 +264,13 @@
<a class="xref" href="impala_shell_options.html#shell_options">impala-shell Configuration Options</a> for details about the
<span class="keyword cmdname">impala-shell</span> command-line options.
</li>
+
</ul>
- </section>
- <section class="section" id="perf_cookbook__perf_cookbook_explain"><h2 class="title sectiontitle">Verify that your queries are planned in an efficient logical manner.</h2>
+ </div>
+
+
+ <div class="section" id="perf_cookbook__perf_cookbook_explain"><h2 class="title sectiontitle">Verify that your queries are planned in an efficient logical manner</h2>
@@ -227,9 +279,11 @@
<a class="xref" href="impala_explain.html#explain">EXPLAIN Statement</a> and <a class="xref" href="impala_explain_plan.html#perf_explain">Using the EXPLAIN Plan for Performance Tuning</a> for
details.
</p>
- </section>
- <section class="section" id="perf_cookbook__perf_cookbook_profile"><h2 class="title sectiontitle">Verify performance characteristics of queries.</h2>
+ </div>
+
+
+ <div class="section" id="perf_cookbook__perf_cookbook_profile"><h2 class="title sectiontitle">Verify performance characteristics of queries</h2>
@@ -238,9 +292,11 @@
within expected ranges by examining the query profile for a query after running it. See
<a class="xref" href="impala_explain_plan.html#perf_profile">Using the Query Profile for Performance Tuning</a> for details.
</p>
- </section>
- <section class="section" id="perf_cookbook__perf_cookbook_os"><h2 class="title sectiontitle">Use appropriate operating system settings.</h2>
+ </div>
+
+
+ <div class="section" id="perf_cookbook__perf_cookbook_os"><h2 class="title sectiontitle">Use appropriate operating system settings</h2>
@@ -250,7 +306,95 @@
that changing the <code class="ph codeph">vm.swappiness</code> Linux kernel setting to a non-zero value improves
overall performance.
</p>
- </section>
+
+ </div>
+
+ <div class="section" id="perf_cookbook__perf_cookbook_hotspot"><h2 class="title sectiontitle">Hotspot analysis</h2>
+
+
+ <p class="p">
+ In the context of Impala, a hotspot is defined as “an Impala daemon
+ that for a single query or a workload is spending a far greater amount
+ of time processing data relative to its neighbours”.
+ </p>
+
+
+ <p class="p">
+ Before discussing the options to tackle this issue some background is
+ first required to understand how this problem can occur.
+ </p>
+
+
+ <p class="p">
+ By default, the scheduling of scan based plan fragments is
+ deterministic. This means that for multiple queries needing to read the
+ same block of data, the same node will be picked to host the scan. The
+ default scheduling logic does not take into account node workload from
+ prior queries. The complexity of materializing a tuple depends on a few
+ factors, namely: decoding and decompression. If the tuples are densely
+ packed into data pages due to good encoding/compression ratios, there
+ will be more work required when reconstructing the data. Each
+ compression codec offers different performance tradeoffs and should be
+ considered before writing the data. Due to the deterministic nature of
+ the scheduler, single nodes can become bottlenecks for highly concurrent
+ queries that use the same tables.
+ </p>
+
+
+ <p class="p">
+ If, for example, a Parquet based dataset is tiny, e.g. a small
+ dimension table, such that it fits into a single HDFS block (Impala by
+ default will create 256 MB blocks when Parquet is used, each containing
+ a single row group) then there are a number of options that can be
+ considered to resolve the potential scheduling hotspots when querying
+ this data:
+ </p>
+
+
+ <ul class="ul">
+ <li class="li">
+ In <span class="keyword">Impala 2.5</span> and higher, the scheduler’s
+ deterministic behaviour can be changed using the following query
+ options: <code class="ph codeph">REPLICA_PREFERENCE</code> and
+ <code class="ph codeph">RANDOM_REPLICA</code>. For a detailed description of each
+ of these modes see <span class="keyword">IMPALA-2696</span>.
+ </li>
+
+
+ <li class="li">
+ HDFS caching can be used to cache block replicas. This will cause
+ the Impala scheduler to randomly pick (from <span class="keyword">Impala 2.2</span> and higher) a node that is hosting a cached block replica for the
+ scan. Note, although HDFS caching has benefits, it serves only to help
+ with the reading of raw block data and not cached tuple data, but with
+ the right number of cached replicas (by default, HDFS only caches one
+ replica), even load distribution can be achieved for smaller
+ datasets.
+ </li>
+
+
+ <li class="li">
+ Do not compress the table data. The uncompressed table data spans more
+ nodes and eliminates skew caused by compression.
+ </li>
+
+
+ <li class="li">
+ Reduce the Parquet file size via the
+ <code class="ph codeph">PARQUET_FILE_SIZE</code> query option when writing the
+ table data. Using this approach the data will span more nodes. However
+ it’s not recommended to drop the size below 32 MB.
+ </li>
+
+ </ul>
+
+ </div>
+
</div>
-<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_performance.html">Tuning Impala for Performance</a></div></div></nav></article></main></body></html>
\ No newline at end of file
+
+<div class="related-links">
+<div class="familylinks">
+<div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_performance.html">Tuning Impala for Performance</a></div>
+</div>
+</div></body>
+</html>
\ No newline at end of file