You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jb...@apache.org on 2017/04/12 18:25:30 UTC
[26/51] [partial] incubator-impala git commit: IMPALA-4181 [DOCS]
Publish rendered Impala documentation to ASF site
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_kudu.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_kudu.html b/docs/build/html/topics/impala_kudu.html
new file mode 100644
index 0000000..78f6534
--- /dev/null
+++ b/docs/build/html/topics/impala_kudu.html
@@ -0,0 +1,1329 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="impala_kudu"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Using Impala to Query Kudu Tables</title></head><body id="impala_kudu"><main role="main"><article role="article" aria-labelledby="impala_kudu__kudu">
+
+ <h1 class="title topictitle1" id="impala_kudu__kudu">Using Impala to Query Kudu Tables</h1>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+
+ You can use Impala to query tables stored by Apache Kudu. This capability
+ allows convenient access to a storage system that is tuned for different kinds of
+ workloads than the default with Impala.
+ </p>
+
+ <p class="p">
+ By default, Impala tables are stored on HDFS using data files with various file formats.
+ HDFS files are ideal for bulk loads (append operations) and queries using full-table scans,
+ but do not support in-place updates or deletes. Kudu is an alternative storage engine used
+ by Impala which can do both in-place updates (for mixed read/write workloads) and fast scans
+ (for data-warehouse/analytic operations). Using Kudu tables with Impala can simplify the
+ ETL pipeline by avoiding extra steps to segregate and reorganize newly arrived data.
+ </p>
+
+ <p class="p">
+ Certain Impala SQL statements and clauses, such as <code class="ph codeph">DELETE</code>,
+ <code class="ph codeph">UPDATE</code>, <code class="ph codeph">UPSERT</code>, and <code class="ph codeph">PRIMARY KEY</code> work
+ only with Kudu tables. Other statements and clauses, such as <code class="ph codeph">LOAD DATA</code>,
+ <code class="ph codeph">TRUNCATE TABLE</code>, and <code class="ph codeph">INSERT OVERWRITE</code>, are not applicable
+ to Kudu tables.
+ </p>
+
+ <p class="p toc inpage"></p>
+
+ </div>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title2" id="impala_kudu__kudu_benefits">
+
+ <h2 class="title topictitle2" id="ariaid-title2">Benefits of Using Kudu Tables with Impala</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ The combination of Kudu and Impala works best for tables where scan performance is
+ important, but data arrives continuously, in small batches, or needs to be updated
+ without being completely replaced. HDFS-backed tables can require substantial overhead
+ to replace or reorganize data files as new data arrives. Impala can perform efficient
+ lookups and scans within Kudu tables, and Impala can also perform update or
+ delete operations efficiently. You can also use the Kudu Java, C++, and Python APIs to
+ do ingestion or transformation operations outside of Impala, and Impala can query the
+ current data at any time.
+ </p>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="impala_kudu__kudu_config">
+
+ <h2 class="title topictitle2" id="ariaid-title3">Configuring Impala for Use with Kudu</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ The <code class="ph codeph">-kudu_master_hosts</code> configuration property must be set correctly
+ for the <span class="keyword cmdname">impalad</span> daemon, for <code class="ph codeph">CREATE TABLE ... STORED AS
+ KUDU</code> statements to connect to the appropriate Kudu server. Typically, the
+ required value for this setting is <code class="ph codeph"><var class="keyword varname">kudu_host</var>:7051</code>.
+ In a high-availability Kudu deployment, specify the names of multiple Kudu hosts separated by commas.
+ </p>
+
+ <p class="p">
+ If the <code class="ph codeph">-kudu_master_hosts</code> configuration property is not set, you can
+ still associate the appropriate value for each table by specifying a
+ <code class="ph codeph">TBLPROPERTIES('kudu.master_addresses')</code> clause in the <code class="ph codeph">CREATE TABLE</code> statement or
+ changing the <code class="ph codeph">TBLPROPERTIES('kudu.master_addresses')</code> value with an <code class="ph codeph">ALTER TABLE</code>
+ statement.
+ </p>
+
+ </div>
+
+ <article class="topic concept nested2" aria-labelledby="ariaid-title4" id="kudu_config__kudu_topology">
+
+ <h3 class="title topictitle3" id="ariaid-title4">Cluster Topology for Kudu Tables</h3>
+
+ <div class="body conbody">
+
+ <p class="p">
+ With HDFS-backed tables, you are typically concerned with the number of DataNodes in
+ the cluster, how many and how large HDFS data files are read during a query, and
+ therefore the amount of work performed by each DataNode and the network communication
+ to combine intermediate results and produce the final result set.
+ </p>
+
+ <p class="p">
+ With Kudu tables, the topology considerations are different, because:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ The underlying storage is managed and organized by Kudu, not represented as HDFS
+ data files.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Kudu handles some of the underlying mechanics of partitioning the data. You can specify
+ the partitioning scheme with combinations of hash and range partitioning, so that you can
+ decide how much effort to expend to manage the partitions as new data arrives. For example,
+ you can construct partitions that apply to date ranges rather than a separate partition for each
+ day or each hour.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Data is physically divided based on units of storage called <dfn class="term">tablets</dfn>. Tablets are
+ stored by <dfn class="term">tablet servers</dfn>. Each tablet server can store multiple tablets,
+ and each tablet is replicated across multiple tablet servers, managed automatically by Kudu.
+ Where practical, colocate the tablet servers on the same hosts as the DataNodes, although that is not required.
+ </p>
+ </li>
+ </ul>
+
+ <p class="p">
+ One consideration for the cluster topology is that the number of replicas for a Kudu table
+ must be odd.
+ </p>
+
+ </div>
+
+ </article>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title5" id="impala_kudu__kudu_ddl">
+
+ <h2 class="title topictitle2" id="ariaid-title5">Impala DDL Enhancements for Kudu Tables (CREATE TABLE and ALTER TABLE)</h2>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ You can use the Impala <code class="ph codeph">CREATE TABLE</code> and <code class="ph codeph">ALTER TABLE</code>
+ statements to create and fine-tune the characteristics of Kudu tables. Because Kudu
+ tables have features and properties that do not apply to other kinds of Impala tables,
+ familiarize yourself with Kudu-related concepts and syntax first.
+ For the general syntax of the <code class="ph codeph">CREATE TABLE</code>
+ statement for Kudu tables, see <a class="xref" href="impala_create_table.html">CREATE TABLE Statement</a>.
+ </p>
+
+ <p class="p toc inpage"></p>
+
+ </div>
+
+ <article class="topic concept nested2" aria-labelledby="ariaid-title6" id="kudu_ddl__kudu_primary_key">
+
+ <h3 class="title topictitle3" id="ariaid-title6">Primary Key Columns for Kudu Tables</h3>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Kudu tables introduce the notion of primary keys to Impala for the first time. The
+ primary key is made up of one or more columns, whose values are combined and used as a
+ lookup key during queries. The tuple represented by these columns must be unique and cannot contain any
+ <code class="ph codeph">NULL</code> values, and can never be updated once inserted. For a
+ Kudu table, all the partition key columns must come from the set of
+ primary key columns.
+ </p>
+
+ <p class="p">
+ The primary key has both physical and logical aspects:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ On the physical side, it is used to map the data values to particular tablets for fast retrieval.
+ Because the tuples formed by the primary key values are unique, the primary key columns are typically
+ highly selective.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ On the logical side, the uniqueness constraint allows you to avoid duplicate data in a table.
+ For example, if an <code class="ph codeph">INSERT</code> operation fails partway through, only some of the
+ new rows might be present in the table. You can re-run the same <code class="ph codeph">INSERT</code>, and
+ only the missing rows will be added. Or if data in the table is stale, you can run an
+ <code class="ph codeph">UPSERT</code> statement that brings the data up to date, without the possibility
+ of creating duplicate copies of existing rows.
+ </p>
+ </li>
+ </ul>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ Impala only allows <code class="ph codeph">PRIMARY KEY</code> clauses and <code class="ph codeph">NOT NULL</code>
+ constraints on columns for Kudu tables. These constraints are enforced on the Kudu side.
+ </p>
+ </div>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested2" aria-labelledby="ariaid-title7" id="kudu_ddl__kudu_column_attributes">
+
+ <h3 class="title topictitle3" id="ariaid-title7">Kudu-Specific Column Attributes for CREATE TABLE</h3>
+
+ <div class="body conbody">
+
+ <p class="p">
+ For the general syntax of the <code class="ph codeph">CREATE TABLE</code>
+ statement for Kudu tables, see <a class="xref" href="impala_create_table.html">CREATE TABLE Statement</a>.
+ The following sections provide more detail for some of the
+ Kudu-specific keywords you can use in column definitions.
+ </p>
+
+ <p class="p">
+ The column list in a <code class="ph codeph">CREATE TABLE</code> statement can include the following
+ attributes, which only apply to Kudu tables:
+ </p>
+
+<pre class="pre codeblock"><code>
+ PRIMARY KEY
+| [NOT] NULL
+| ENCODING <var class="keyword varname">codec</var>
+| COMPRESSION <var class="keyword varname">algorithm</var>
+| DEFAULT <var class="keyword varname">constant_expression</var>
+| BLOCK_SIZE <var class="keyword varname">number</var>
+</code></pre>
+
+ <p class="p toc inpage">
+ See the following sections for details about each column attribute.
+ </p>
+
+ </div>
+
+ <article class="topic concept nested3" aria-labelledby="ariaid-title8" id="kudu_column_attributes__kudu_primary_key_attribute">
+
+ <h4 class="title topictitle4" id="ariaid-title8">PRIMARY KEY Attribute</h4>
+
+ <div class="body conbody">
+
+ <p class="p">
+ The primary key for a Kudu table is a column, or set of columns, that uniquely
+ identifies every row. The primary key value also is used as the natural sort order
+ for the values from the table. The primary key value for each row is based on the
+ combination of values for the columns.
+ </p>
+
+ <p class="p">
+ Because all of the primary key columns must have non-null values, specifying a column
+ in the <code class="ph codeph">PRIMARY KEY</code> clause implicitly adds the <code class="ph codeph">NOT
+ NULL</code> attribute to that column.
+ </p>
+
+ <p class="p">
+ The primary key columns must be the first ones specified in the <code class="ph codeph">CREATE
+ TABLE</code> statement. For a single-column primary key, you can include a
+ <code class="ph codeph">PRIMARY KEY</code> attribute inline with the column definition. For a
+ multi-column primary key, you include a <code class="ph codeph">PRIMARY KEY (<var class="keyword varname">c1</var>,
+ <var class="keyword varname">c2</var>, ...)</code> clause as a separate entry at the end of the
+ column list.
+ </p>
+
+ <p class="p">
+ You can specify the <code class="ph codeph">PRIMARY KEY</code> attribute either inline in a single
+ column definition, or as a separate clause at the end of the column list:
+ </p>
+
+<pre class="pre codeblock"><code>
+CREATE TABLE pk_inline
+(
+ col1 BIGINT PRIMARY KEY,
+ col2 STRING,
+ col3 BOOLEAN
+) PARTITION BY HASH(col1) PARTITIONS 2 STORED AS KUDU;
+
+CREATE TABLE pk_at_end
+(
+ col1 BIGINT,
+ col2 STRING,
+ col3 BOOLEAN,
+ PRIMARY KEY (col1)
+) PARTITION BY HASH(col1) PARTITIONS 2 STORED AS KUDU;
+</code></pre>
+
+ <p class="p">
+ When the primary key is a single column, these two forms are equivalent. If the
+ primary key consists of more than one column, you must specify the primary key using
+ a separate entry in the column list:
+ </p>
+
+<pre class="pre codeblock"><code>
+CREATE TABLE pk_multiple_columns
+(
+ col1 BIGINT,
+ col2 STRING,
+ col3 BOOLEAN,
+ <strong class="ph b">PRIMARY KEY (col1, col2)</strong>
+) PARTITION BY HASH(col2) PARTITIONS 2 STORED AS KUDU;
+</code></pre>
+
+ <p class="p">
+ The <code class="ph codeph">SHOW CREATE TABLE</code> statement always represents the
+ <code class="ph codeph">PRIMARY KEY</code> specification as a separate item in the column list:
+ </p>
+
+<pre class="pre codeblock"><code>
+CREATE TABLE inline_pk_rewritten (id BIGINT <strong class="ph b">PRIMARY KEY</strong>, s STRING)
+ PARTITION BY HASH(id) PARTITIONS 2 STORED AS KUDU;
+
+SHOW CREATE TABLE inline_pk_rewritten;
++------------------------------------------------------------------------------+
+| result |
++------------------------------------------------------------------------------+
+| CREATE TABLE user.inline_pk_rewritten ( |
+| id BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, |
+| s STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, |
+| <strong class="ph b">PRIMARY KEY (id)</strong> |
+| ) |
+| PARTITION BY HASH (id) PARTITIONS 2 |
+| STORED AS KUDU |
+| TBLPROPERTIES ('kudu.master_addresses'='host.example.com') |
++------------------------------------------------------------------------------+
+</code></pre>
+
+ <p class="p">
+ The notion of primary key only applies to Kudu tables. Every Kudu table requires a
+ primary key. The primary key consists of one or more columns. You must specify any
+ primary key columns first in the column list.
+ </p>
+
+ <p class="p">
+ The contents of the primary key columns cannot be changed by an
+ <code class="ph codeph">UPDATE</code> or <code class="ph codeph">UPSERT</code> statement. Including too many
+ columns in the primary key (more than 5 or 6) can also reduce the performance of
+ write operations. Therefore, pick the most selective and most frequently
+ tested non-null columns for the primary key specification.
+ If a column must always have a value, but that value
+ might change later, leave it out of the primary key and use a <code class="ph codeph">NOT
+ NULL</code> clause for that column instead. If an existing row has an
+ incorrect or outdated key column value, delete the old row and insert an entirely
+ new row with the correct primary key.
+ </p>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested3" aria-labelledby="ariaid-title9" id="kudu_column_attributes__kudu_not_null_attribute">
+
+ <h4 class="title topictitle4" id="ariaid-title9">NULL | NOT NULL Attribute</h4>
+
+ <div class="body conbody">
+
+ <p class="p">
+ For Kudu tables, you can specify which columns can contain nulls or not. This
+ constraint offers an extra level of consistency enforcement for Kudu tables. If an
+ application requires a field to always be specified, include a <code class="ph codeph">NOT
+ NULL</code> clause in the corresponding column definition, and Kudu prevents rows
+ from being inserted with a <code class="ph codeph">NULL</code> in that column.
+ </p>
+
+ <p class="p">
+ For example, a table containing geographic information might require the latitude
+ and longitude coordinates to always be specified. Other attributes might be allowed
+ to be <code class="ph codeph">NULL</code>. For example, a location might not have a designated
+ place name, its altitude might be unimportant, and its population might be initially
+ unknown, to be filled in later.
+ </p>
+
+ <p class="p">
+ Because all of the primary key columns must have non-null values, specifying a column
+ in the <code class="ph codeph">PRIMARY KEY</code> clause implicitly adds the <code class="ph codeph">NOT
+ NULL</code> attribute to that column.
+ </p>
+
+ <p class="p">
+ For non-Kudu tables, Impala allows any column to contain <code class="ph codeph">NULL</code>
+ values, because it is not practical to enforce a <span class="q">"not null"</span> constraint on HDFS
+ data files that could be prepared using external tools and ETL processes.
+ </p>
+
+<pre class="pre codeblock"><code>
+CREATE TABLE required_columns
+(
+ id BIGINT PRIMARY KEY,
+ latitude DOUBLE NOT NULL,
+ longitude DOUBLE NOT NULL,
+ place_name STRING,
+ altitude DOUBLE,
+ population BIGINT
+) PARTITION BY HASH(id) PARTITIONS 2 STORED AS KUDU;
+</code></pre>
+
+ <p class="p">
+ During performance optimization, Kudu can use the knowledge that nulls are not
+ allowed to skip certain checks on each input row, speeding up queries and join
+ operations. Therefore, specify <code class="ph codeph">NOT NULL</code> constraints when
+ appropriate.
+ </p>
+
+ <p class="p">
+ The <code class="ph codeph">NULL</code> clause is the default condition for all columns that are not
+ part of the primary key. You can omit it, or specify it to clarify that you have made a
+ conscious design decision to allow nulls in a column.
+ </p>
+
+ <p class="p">
+ Because primary key columns cannot contain any <code class="ph codeph">NULL</code> values, the
+ <code class="ph codeph">NOT NULL</code> clause is not required for the primary key columns,
+ but you might still specify it to make your code self-describing.
+ </p>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested3" aria-labelledby="ariaid-title10" id="kudu_column_attributes__kudu_default_attribute">
+
+ <h4 class="title topictitle4" id="ariaid-title10">DEFAULT Attribute</h4>
+
+ <div class="body conbody">
+
+ <p class="p">
+ You can specify a default value for columns in Kudu tables. The default value can be
+ any constant expression, for example, a combination of literal values, arithmetic
+ and string operations. It cannot contain references to columns or non-deterministic
+ function calls.
+ </p>
+
+ <p class="p">
+ The following example shows different kinds of expressions for the
+ <code class="ph codeph">DEFAULT</code> clause. The requirement to use a constant value means that
+ you can fill in a placeholder value such as <code class="ph codeph">NULL</code>, empty string,
+ 0, -1, <code class="ph codeph">'N/A'</code> and so on, but you cannot reference functions or
+ column names. Therefore, you cannot use <code class="ph codeph">DEFAULT</code> to do things such as
+ automatically making an uppercase copy of a string value, storing Boolean values based
+ on tests of other columns, or add or subtract one from another column representing a sequence number.
+ </p>
+
+<pre class="pre codeblock"><code>
+CREATE TABLE default_vals
+(
+ id BIGINT PRIMARY KEY,
+ name STRING NOT NULL DEFAULT 'unknown',
+ address STRING DEFAULT upper('no fixed address'),
+ age INT DEFAULT -1,
+ earthling BOOLEAN DEFAULT TRUE,
+ planet_of_origin STRING DEFAULT 'Earth',
+ optional_col STRING DEFAULT NULL
+) PARTITION BY HASH(id) PARTITIONS 2 STORED AS KUDU;
+</code></pre>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ When designing an entirely new schema, prefer to use <code class="ph codeph">NULL</code> as the
+ placeholder for any unknown or missing values, because that is the universal convention
+ among database systems. Null values can be stored efficiently, and easily checked with the
+ <code class="ph codeph">IS NULL</code> or <code class="ph codeph">IS NOT NULL</code> operators. The <code class="ph codeph">DEFAULT</code>
+ attribute is appropriate when ingesting data that already has an established convention for
+ representing unknown or missing values, or where the vast majority of rows have some common
+ non-null value.
+ </p>
+ </div>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested3" aria-labelledby="ariaid-title11" id="kudu_column_attributes__kudu_encoding_attribute">
+
+ <h4 class="title topictitle4" id="ariaid-title11">ENCODING Attribute</h4>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Each column in a Kudu table can optionally use an encoding, a low-overhead form of
+ compression that reduces the size on disk, then requires additional CPU cycles to
+ reconstruct the original values during queries. Typically, highly compressible data
+ benefits from the reduced I/O to read the data back from disk. By default, each
+ column uses the <span class="q">"plain"</span> encoding where the data is stored unchanged.
+ </p>
+
+ <div class="p">
+ The encoding keywords that Impala recognizes are:
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph">AUTO_ENCODING</code>: use the default encoding based on the column
+ type; currently always the same as <code class="ph codeph">PLAIN_ENCODING</code>, but subject to
+ change in the future.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph">PLAIN_ENCODING</code>: leave the value in its original binary format.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph">RLE</code>: compress repeated values (when sorted in primary key
+ order) by including a count.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph">DICT_ENCODING</code>: when the number of different string values is
+ low, replace the original string with a numeric ID.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph">BIT_SHUFFLE</code>: rearrange the bits of the values to efficiently
+ compress sequences of values that are identical or vary only slightly based
+ on primary key order. The resulting encoded data is also compressed with LZ4.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph">PREFIX_ENCODING</code>: compress common prefixes in string values; mainly for use internally within Kudu.
+ </p>
+ </li>
+ </ul>
+ </div>
+
+
+
+ <p class="p">
+ The following example shows the Impala keywords representing the encoding types.
+ (The Impala keywords match the symbolic names used within Kudu.)
+ For usage guidelines on the different kinds of encoding, see
+ <a class="xref" href="https://kudu.apache.org/docs/schema_design.html" target="_blank">the Kudu documentation</a>.
+ The <code class="ph codeph">DESCRIBE</code> output shows how the encoding is reported after
+ the table is created, and that omitting the encoding (in this case, for the
+ <code class="ph codeph">ID</code> column) is the same as specifying <code class="ph codeph">DEFAULT_ENCODING</code>.
+ </p>
+
+<pre class="pre codeblock"><code>
+CREATE TABLE various_encodings
+(
+ id BIGINT PRIMARY KEY,
+ c1 BIGINT ENCODING PLAIN_ENCODING,
+ c2 BIGINT ENCODING AUTO_ENCODING,
+ c3 TINYINT ENCODING BIT_SHUFFLE,
+ c4 DOUBLE ENCODING BIT_SHUFFLE,
+ c5 BOOLEAN ENCODING RLE,
+ c6 STRING ENCODING DICT_ENCODING,
+ c7 STRING ENCODING PREFIX_ENCODING
+) PARTITION BY HASH(id) PARTITIONS 2 STORED AS KUDU;
+
+-- Some columns are omitted from the output for readability.
+describe various_encodings;
++------+---------+-------------+----------+-----------------+
+| name | type | primary_key | nullable | encoding |
++------+---------+-------------+----------+-----------------+
+| id | bigint | true | false | AUTO_ENCODING |
+| c1 | bigint | false | true | PLAIN_ENCODING |
+| c2 | bigint | false | true | AUTO_ENCODING |
+| c3 | tinyint | false | true | BIT_SHUFFLE |
+| c4 | double | false | true | BIT_SHUFFLE |
+| c5 | boolean | false | true | RLE |
+| c6 | string | false | true | DICT_ENCODING |
+| c7 | string | false | true | PREFIX_ENCODING |
++------+---------+-------------+----------+-----------------+
+</code></pre>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested3" aria-labelledby="ariaid-title12" id="kudu_column_attributes__kudu_compression_attribute">
+
+ <h4 class="title topictitle4" id="ariaid-title12">COMPRESSION Attribute</h4>
+
+ <div class="body conbody">
+
+ <p class="p">
+ You can specify a compression algorithm to use for each column in a Kudu table. This
+ attribute imposes more CPU overhead when retrieving the values than the
+ <code class="ph codeph">ENCODING</code> attribute does. Therefore, use it primarily for columns with
+ long strings that do not benefit much from the less-expensive <code class="ph codeph">ENCODING</code>
+ attribute.
+ </p>
+
+ <p class="p">
+ The choices for <code class="ph codeph">COMPRESSION</code> are <code class="ph codeph">LZ4</code>,
+ <code class="ph codeph">SNAPPY</code>, and <code class="ph codeph">ZLIB</code>.
+ </p>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ Columns that use the <code class="ph codeph">BITSHUFFLE</code> encoding are already compressed
+ using <code class="ph codeph">LZ4</code>, and so typically do not need any additional
+ <code class="ph codeph">COMPRESSION</code> attribute.
+ </p>
+ </div>
+
+ <p class="p">
+ The following example shows design considerations for several
+ <code class="ph codeph">STRING</code> columns with different distribution characteristics, leading
+ to choices for both the <code class="ph codeph">ENCODING</code> and <code class="ph codeph">COMPRESSION</code>
+ attributes. The <code class="ph codeph">country</code> values come from a specific set of strings,
+ therefore this column is a good candidate for dictionary encoding. The
+ <code class="ph codeph">post_id</code> column contains an ascending sequence of integers, where
+ several leading bits are likely to be all zeroes, therefore this column is a good
+ candidate for bitshuffle encoding. The <code class="ph codeph">body</code>
+ column and the corresponding columns for translated versions tend to be long unique
+ strings that are not practical to use with any of the encoding schemes, therefore
+ they employ the <code class="ph codeph">COMPRESSION</code> attribute instead. The ideal compression
+ codec in each case would require some experimentation to determine how much space
+ savings it provided and how much CPU overhead it added, based on real-world data.
+ </p>
+
+<pre class="pre codeblock"><code>
+CREATE TABLE blog_posts
+(
+ user_id STRING ENCODING DICT_ENCODING,
+ post_id BIGINT ENCODING BIT_SHUFFLE,
+ subject STRING ENCODING PLAIN_ENCODING,
+ body STRING COMPRESSION LZ4,
+ spanish_translation STRING COMPRESSION SNAPPY,
+ esperanto_translation STRING COMPRESSION ZLIB,
+ PRIMARY KEY (user_id, post_id)
+) PARTITION BY HASH(user_id, post_id) PARTITIONS 2 STORED AS KUDU;
+</code></pre>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested3" aria-labelledby="ariaid-title13" id="kudu_column_attributes__kudu_block_size_attribute">
+
+ <h4 class="title topictitle4" id="ariaid-title13">BLOCK_SIZE Attribute</h4>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Although Kudu does not use HDFS files internally, and thus is not affected by
+ the HDFS block size, it does have an underlying unit of I/O called the
+ <dfn class="term">block size</dfn>. The <code class="ph codeph">BLOCK_SIZE</code> attribute lets you set the
+ block size for any column.
+ </p>
+
+ <p class="p">
+ The block size attribute is a relatively advanced feature. Refer to
+ <a class="xref" href="https://kudu.apache.org/docs/index.html" target="_blank">the Kudu documentation</a>
+ for usage details.
+ </p>
+
+
+
+ </div>
+
+ </article>
+
+ </article>
+
+ <article class="topic concept nested2" aria-labelledby="ariaid-title14" id="kudu_ddl__kudu_partitioning">
+
+ <h3 class="title topictitle3" id="ariaid-title14">Partitioning for Kudu Tables</h3>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Kudu tables use special mechanisms to distribute data among the underlying
+ tablet servers. Although we refer to such tables as partitioned tables, they are
+ distinguished from traditional Impala partitioned tables by use of different clauses
+ on the <code class="ph codeph">CREATE TABLE</code> statement. Kudu tables use
+ <code class="ph codeph">PARTITION BY</code>, <code class="ph codeph">HASH</code>, <code class="ph codeph">RANGE</code>, and
+ range specification clauses rather than the <code class="ph codeph">PARTITIONED BY</code> clause
+ for HDFS-backed tables, which specifies only a column name and creates a new partition for each
+ different value.
+ </p>
+
+ <p class="p">
+ For background information and architectural details about the Kudu partitioning
+ mechanism, see
+ <a class="xref" href="https://kudu.apache.org/kudu.pdf" target="_blank">the Kudu white paper, section 3.2</a>.
+ </p>
+
+
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ The Impala DDL syntax for Kudu tables is different than in early Kudu versions,
+ which used an experimental fork of the Impala code. For example, the
+ <code class="ph codeph">DISTRIBUTE BY</code> clause is now <code class="ph codeph">PARTITION BY</code>, the
+ <code class="ph codeph">INTO <var class="keyword varname">n</var> BUCKETS</code> clause is now
+ <code class="ph codeph">PARTITIONS <var class="keyword varname">n</var></code> and the range partitioning syntax
+ is reworked to replace the <code class="ph codeph">SPLIT ROWS</code> clause with more expressive
+ syntax involving comparison operators.
+ </p>
+ </div>
+
+ <p class="p toc inpage"></p>
+
+ </div>
+
+ <article class="topic concept nested3" aria-labelledby="ariaid-title15" id="kudu_partitioning__kudu_hash_partitioning">
+ <h4 class="title topictitle4" id="ariaid-title15">Hash Partitioning</h4>
+ <div class="body conbody">
+
+ <p class="p">
+ Hash partitioning is the simplest type of partitioning for Kudu tables.
+ For hash-partitioned Kudu tables, inserted rows are divided up between a fixed number
+ of <span class="q">"buckets"</span> by applying a hash function to the values of the columns specified
+ in the <code class="ph codeph">HASH</code> clause.
+ Hashing ensures that rows with similar values are evenly distributed, instead of
+ clumping together all in the same bucket. Spreading new rows across the buckets this
+ way lets insertion operations work in parallel across multiple tablet servers.
+ Separating the hashed values can impose additional overhead on queries, where
+ queries with range-based predicates might have to read multiple tablets to retrieve
+ all the relevant values.
+ </p>
+
+<pre class="pre codeblock"><code>
+-- 1M rows with 50 hash partitions = approximately 20,000 rows per partition.
+-- The values in each partition are not sequential, but rather based on a hash function.
+-- Rows 1, 99999, and 123456 might be in the same partition.
+CREATE TABLE million_rows (id string primary key, s string)
+ PARTITION BY HASH(id) PARTITIONS 50
+ STORED AS KUDU;
+
+-- Because the ID values are unique, we expect the rows to be roughly
+-- evenly distributed between the buckets in the destination table.
+INSERT INTO million_rows SELECT * FROM billion_rows ORDER BY id LIMIT 1e6;
+</code></pre>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ The largest number of buckets that you can create with a <code class="ph codeph">PARTITIONS</code>
+ clause varies depending on the number of tablet servers in the cluster, while the smallest is 2.
+ For simplicity, some of the simple <code class="ph codeph">CREATE TABLE</code> statements throughout this section
+ use <code class="ph codeph">PARTITIONS 2</code> to illustrate the minimum requirements for a Kudu table.
+ For large tables, prefer to use roughly 10 partitions per server in the cluster.
+ </p>
+ </div>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested3" aria-labelledby="ariaid-title16" id="kudu_partitioning__kudu_range_partitioning">
+ <h4 class="title topictitle4" id="ariaid-title16">Range Partitioning</h4>
+ <div class="body conbody">
+
+ <p class="p">
+ Range partitioning lets you specify partitioning precisely, based on single values or ranges
+ of values within one or more columns. You add one or more <code class="ph codeph">RANGE</code> clauses to the
+ <code class="ph codeph">CREATE TABLE</code> statement, following the <code class="ph codeph">PARTITION BY</code>
+ clause.
+ </p>
+
+ <p class="p">
+ Range-partitioned Kudu tables use one or more range clauses, which include a
+ combination of constant expressions, <code class="ph codeph">VALUE</code> or <code class="ph codeph">VALUES</code>
+ keywords, and comparison operators. (This syntax replaces the <code class="ph codeph">SPLIT
+ ROWS</code> clause used with early Kudu versions.)
+ For the full syntax, see <a class="xref" href="impala_create_table.html">CREATE TABLE Statement</a>.
+ </p>
+
+<pre class="pre codeblock"><code>
+-- 50 buckets, all for IDs beginning with a lowercase letter.
+-- Having only a single range enforces the allowed range of values
+-- but does not add any extra parallelism.
+create table million_rows_one_range (id string primary key, s string)
+ partition by hash(id) partitions 50,
+ range (partition 'a' <= values < '{')
+ stored as kudu;
+
+-- 50 buckets for IDs beginning with a lowercase letter
+-- plus 50 buckets for IDs beginning with an uppercase letter.
+-- Total number of buckets = number in the PARTITIONS clause x number of ranges.
+-- We are still enforcing constraints on the primary key values
+-- allowed in the table, and the 2 ranges provide better parallelism
+-- as rows are inserted or the table is scanned.
+create table million_rows_two_ranges (id string primary key, s string)
+ partition by hash(id) partitions 50,
+ range (partition 'a' <= values < '{', partition 'A' <= values < '[')
+ stored as kudu;
+
+-- Same as previous table, with an extra range covering the single key value '00000'.
+create table million_rows_three_ranges (id string primary key, s string)
+ partition by hash(id) partitions 50,
+ range (partition 'a' <= values < '{', partition 'A' <= values < '[', partition value = '00000')
+ stored as kudu;
+
+-- The range partitioning can be displayed with a SHOW command in impala-shell.
+show range partitions million_rows_three_ranges;
++---------------------+
+| RANGE (id) |
++---------------------+
+| VALUE = "00000" |
+| "A" <= VALUES < "[" |
+| "a" <= VALUES < "{" |
++---------------------+
+
+</code></pre>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ When defining ranges, be careful to avoid <span class="q">"fencepost errors"</span> where values at the
+ extreme ends might be included or omitted by accident. For example, in the tables defined
+ in the preceding code listings, the range <code class="ph codeph">"a" <= VALUES < "{"</code> ensures that
+ any values starting with <code class="ph codeph">z</code>, such as <code class="ph codeph">za</code> or <code class="ph codeph">zzz</code>
+ or <code class="ph codeph">zzz-ZZZ</code>, are all included, by using a less-than operator for the smallest
+ value after all the values starting with <code class="ph codeph">z</code>.
+ </p>
+ </div>
+
+ <p class="p">
+ For range-partitioned Kudu tables, an appropriate range must exist before a data value can be created in the table.
+ Any <code class="ph codeph">INSERT</code>, <code class="ph codeph">UPDATE</code>, or <code class="ph codeph">UPSERT</code> statements fail if they try to
+ create column values that fall outside the specified ranges. The error checking for ranges is performed on the
+ Kudu side; Impala passes the specified range information to Kudu, and passes back any error or warning if the
+ ranges are not valid. (A nonsensical range specification causes an error for a DDL statement, but only a warning
+ for a DML statement.)
+ </p>
+
+ <p class="p">
+ Ranges can be non-contiguous:
+ </p>
+
+<pre class="pre codeblock"><code>
+partition by range (year) (partition 1885 <= values <= 1889, partition 1893 <= values <= 1897)
+
+partition by range (letter_grade) (partition value = 'A', partition value = 'B',
+ partition value = 'C', partition value = 'D', partition value = 'F')
+
+</code></pre>
+
+ <p class="p">
+ The <code class="ph codeph">ALTER TABLE</code> statement with the <code class="ph codeph">ADD PARTITION</code> or
+ <code class="ph codeph">DROP PARTITION</code> clauses can be used to add or remove ranges from an
+ existing Kudu table.
+ </p>
+
+<pre class="pre codeblock"><code>
+ALTER TABLE foo ADD PARTITION 30 <= VALUES < 50;
+ALTER TABLE foo DROP PARTITION 1 <= VALUES < 5;
+
+</code></pre>
+
+ <p class="p">
+ When a range is added, the new range must not overlap with any of the previous ranges;
+ that is, it can only fill in gaps within the previous ranges.
+ </p>
+
+<pre class="pre codeblock"><code>
+alter table test_scores add range partition value = 'E';
+
+alter table year_ranges add range partition 1890 <= values < 1893;
+
+</code></pre>
+
+ <p class="p">
+ When a range is removed, all the associated rows in the table are deleted. (This
+ is true whether the table is internal or external.)
+ </p>
+
+<pre class="pre codeblock"><code>
+alter table test_scores drop range partition value = 'E';
+
+alter table year_ranges drop range partition 1890 <= values < 1893;
+
+</code></pre>
+
+ <p class="p">
+ Kudu tables can also use a combination of hash and range partitioning.
+ </p>
+
+<pre class="pre codeblock"><code>
+partition by hash (school) partitions 10,
+ range (letter_grade) (partition value = 'A', partition value = 'B',
+ partition value = 'C', partition value = 'D', partition value = 'F')
+
+</code></pre>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested3" aria-labelledby="ariaid-title17" id="kudu_partitioning__kudu_partitioning_misc">
+ <h4 class="title topictitle4" id="ariaid-title17">Working with Partitioning in Kudu Tables</h4>
+ <div class="body conbody">
+
+ <p class="p">
+ To see the current partitioning scheme for a Kudu table, you can use the <code class="ph codeph">SHOW
+ CREATE TABLE</code> statement or the <code class="ph codeph">SHOW PARTITIONS</code> statement. The
+ <code class="ph codeph">CREATE TABLE</code> syntax displayed by this statement includes all the
+ hash, range, or both clauses that reflect the original table structure plus any
+ subsequent <code class="ph codeph">ALTER TABLE</code> statements that changed the table structure.
+ </p>
+
+ <p class="p">
+ To see the underlying buckets and partitions for a Kudu table, use the
+ <code class="ph codeph">SHOW TABLE STATS</code> or <code class="ph codeph">SHOW PARTITIONS</code> statement.
+ </p>
+
+ </div>
+ </article>
+
+ </article>
+
+ <article class="topic concept nested2" aria-labelledby="ariaid-title18" id="kudu_ddl__kudu_timestamps">
+
+ <h3 class="title topictitle3" id="ariaid-title18">Handling Date, Time, or Timestamp Data with Kudu</h3>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Because currently a Kudu table cannot have a column with the Impala
+ <code class="ph codeph">TIMESTAMP</code> type, expect to store date/time information as the number
+ of seconds, milliseconds, or microseconds since the Unix epoch date of January 1,
+ 1970. Specify the column as <code class="ph codeph">BIGINT</code> in the Impala <code class="ph codeph">CREATE
+ TABLE</code> statement, corresponding to an 8-byte integer (an
+ <code class="ph codeph">int64</code>) in the underlying Kudu table). Then use Impala date/time
+ conversion functions as necessary to produce a numeric, <code class="ph codeph">TIMESTAMP</code>,
+ or <code class="ph codeph">STRING</code> value depending on the context.
+ </p>
+
+ <p class="p">
+ For example, the <code class="ph codeph">unix_timestamp()</code> function returns an integer result
+ representing the number of seconds past the epoch. The <code class="ph codeph">now()</code> function
+ produces a <code class="ph codeph">TIMESTAMP</code> representing the current date and time, which can
+ be passed as an argument to <code class="ph codeph">unix_timestamp()</code>. And string literals
+ representing dates and date/times can be cast to <code class="ph codeph">TIMESTAMP</code>, and from there
+ converted to numeric values. The following examples show how you might store a date/time
+ column as <code class="ph codeph">BIGINT</code> in a Kudu table, but still use string literals and
+ <code class="ph codeph">TIMESTAMP</code> values for convenience.
+ </p>
+
+<pre class="pre codeblock"><code>
+-- now() returns a TIMESTAMP and shows the format for string literals you can cast to TIMESTAMP.
+select now();
++-------------------------------+
+| now() |
++-------------------------------+
+| 2017-01-25 23:50:10.132385000 |
++-------------------------------+
+
+-- unix_timestamp() accepts either a TIMESTAMP or an equivalent string literal.
+select unix_timestamp(now());
++------------------+
+| unix_timestamp() |
++------------------+
+| 1485386670 |
++------------------+
+
+select unix_timestamp('2017-01-01');
++------------------------------+
+| unix_timestamp('2017-01-01') |
++------------------------------+
+| 1483228800 |
++------------------------------+
+
+-- Make a table representing a date/time value as BIGINT.
+-- Construct 1 range partition and 20 associated hash partitions for each year.
+-- Use date/time conversion functions to express the ranges as human-readable dates.
+create table time_series(id bigint, when_exactly bigint, event string, primary key (id, when_exactly))
+ partition by hash (id) partitions 20,
+ range (when_exactly)
+ (
+ partition unix_timestamp('2015-01-01') <= values < unix_timestamp('2016-01-01'),
+ partition unix_timestamp('2016-01-01') <= values < unix_timestamp('2017-01-01'),
+ partition unix_timestamp('2017-01-01') <= values < unix_timestamp('2018-01-01')
+ )
+ stored as kudu;
+
+-- On insert, we can transform a human-readable date/time into a numeric value.
+insert into time_series values (12345, unix_timestamp('2017-01-25 23:24:56'), 'Working on doc examples');
+
+-- On retrieval, we can examine the numeric date/time value or turn it back into a string for readability.
+select id, when_exactly, from_unixtime(when_exactly) as 'human-readable date/time', event
+ from time_series order by when_exactly limit 100;
++-------+--------------+--------------------------+-------------------------+
+| id | when_exactly | human-readable date/time | event |
++-------+--------------+--------------------------+-------------------------+
+| 12345 | 1485386696 | 2017-01-25 23:24:56 | Working on doc examples |
++-------+--------------+--------------------------+-------------------------+
+
+</code></pre>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ If you do high-precision arithmetic involving numeric date/time values,
+ when dividing millisecond values by 1000, or microsecond values by 1 million, always
+ cast the integer numerator to a <code class="ph codeph">DECIMAL</code> with sufficient precision
+ and scale to avoid any rounding or loss of precision.
+ </p>
+ </div>
+
+<pre class="pre codeblock"><code>
+-- 1 million and 1 microseconds = 1.000001 seconds.
+select microseconds,
+ cast (microseconds as decimal(20,7)) / 1e6 as fractional_seconds
+ from table_with_microsecond_column;
++--------------+----------------------+
+| microseconds | fractional_seconds |
++--------------+----------------------+
+| 1000001 | 1.000001000000000000 |
++--------------+----------------------+
+
+</code></pre>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested2" aria-labelledby="ariaid-title19" id="kudu_ddl__kudu_metadata">
+
+ <h3 class="title topictitle3" id="ariaid-title19">How Impala Handles Kudu Metadata</h3>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Much of the metadata for Kudu tables is handled by the underlying
+ storage layer. Kudu tables have less reliance on the metastore
+ database, and require less metadata caching on the Impala side.
+ For example, information about partitions in Kudu tables is managed
+ by Kudu, and Impala does not cache any block locality metadata
+ for Kudu tables.
+ </p>
+ <p class="p">
+ The <code class="ph codeph">REFRESH</code> and <code class="ph codeph">INVALIDATE METADATA</code>
+ statements are needed less frequently for Kudu tables than for
+ HDFS-backed tables. Neither statement is needed when data is
+ added to, removed, or updated in a Kudu table, even if the changes
+ are made directly to Kudu through a client program using the Kudu API.
+ Run <code class="ph codeph">REFRESH <var class="keyword varname">table_name</var></code> or
+ <code class="ph codeph">INVALIDATE METADATA <var class="keyword varname">table_name</var></code>
+ for a Kudu table only after making a change to the Kudu table schema,
+ such as adding or dropping a column, by a mechanism other than
+ Impala.
+ </p>
+
+ <p class="p">
+ Because Kudu manages the metadata for its own tables separately from the metastore
+ database, there is a table name stored in the metastore database for Impala to use,
+ and a table name on the Kudu side, and these names can be modified independently
+ through <code class="ph codeph">ALTER TABLE</code> statements.
+ </p>
+
+ <p class="p">
+ To avoid potential name conflicts, the prefix <code class="ph codeph">impala::</code>
+ and the Impala database name are encoded into the underlying Kudu
+ table name:
+ </p>
+
+<pre class="pre codeblock"><code>
+create database some_database;
+use some_database;
+
+create table table_name_demo (x int primary key, y int)
+ partition by hash (x) partitions 2 stored as kudu;
+
+describe formatted table_name_demo;
+...
+kudu.table_name | impala::some_database.table_name_demo
+
+</code></pre>
+
+ <p class="p">
+ See <a class="xref" href="impala_tables.html">Overview of Impala Tables</a> for examples of how to change the name of
+ the Impala table in the metastore database, the name of the underlying Kudu
+ table, or both.
+ </p>
+
+ </div>
+
+ </article>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title20" id="impala_kudu__kudu_etl">
+
+ <h2 class="title topictitle2" id="ariaid-title20">Loading Data into Kudu Tables</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Kudu tables are well-suited to use cases where data arrives continuously, in small or
+ moderate volumes. To bring data into Kudu tables, use the Impala <code class="ph codeph">INSERT</code>
+ and <code class="ph codeph">UPSERT</code> statements. The <code class="ph codeph">LOAD DATA</code> statement does
+ not apply to Kudu tables.
+ </p>
+
+ <p class="p">
+ Because Kudu manages its own storage layer that is optimized for smaller block sizes than
+ HDFS, and performs its own housekeeping to keep data evenly distributed, it is not
+ subject to the <span class="q">"many small files"</span> issue and does not need explicit reorganization
+ and compaction as the data grows over time. The partitions within a Kudu table can be
+ specified to cover a variety of possible data distributions, instead of hardcoding a new
+ partition for each new day, hour, and so on, which can lead to inefficient,
+ hard-to-scale, and hard-to-manage partition schemes with HDFS tables.
+ </p>
+
+ <p class="p">
+ Your strategy for performing ETL or bulk updates on Kudu tables should take into account
+ the limitations on consistency for DML operations.
+ </p>
+
+ <p class="p">
+ Make <code class="ph codeph">INSERT</code>, <code class="ph codeph">UPDATE</code>, and <code class="ph codeph">UPSERT</code>
+ operations <dfn class="term">idempotent</dfn>: that is, able to be applied multiple times and still
+ produce an identical result.
+ </p>
+
+ <p class="p">
+ If a bulk operation is in danger of exceeding capacity limits due to timeouts or high
+ memory usage, split it into a series of smaller operations.
+ </p>
+
+ <p class="p">
+ Avoid running concurrent ETL operations where the end results depend on precise
+ ordering. In particular, do not rely on an <code class="ph codeph">INSERT ... SELECT</code> statement
+ that selects from the same table into which it is inserting, unless you include extra
+ conditions in the <code class="ph codeph">WHERE</code> clause to avoid reading the newly inserted rows
+ within the same statement.
+ </p>
+
+ <p class="p">
+ Because relationships between tables cannot be enforced by Impala and Kudu, and cannot
+ be committed or rolled back together, do not expect transactional semantics for
+ multi-table operations.
+ </p>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title21" id="impala_kudu__kudu_dml">
+
+ <h2 class="title topictitle2" id="ariaid-title21">Impala DML Support for Kudu Tables (INSERT, UPDATE, DELETE, UPSERT)</h2>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ Impala supports certain DML statements for Kudu tables only. The <code class="ph codeph">UPDATE</code>
+ and <code class="ph codeph">DELETE</code> statements let you modify data within Kudu tables without
+ rewriting substantial amounts of table data. The <code class="ph codeph">UPSERT</code> statement acts
+ as a combination of <code class="ph codeph">INSERT</code> and <code class="ph codeph">UPDATE</code>, inserting rows
+ where the primary key does not already exist, and updating the non-primary key columns
+ where the primary key does already exist in the table.
+ </p>
+
+ <p class="p">
+ The <code class="ph codeph">INSERT</code> statement for Kudu tables honors the unique and <code class="ph codeph">NOT
+ NULL</code> requirements for the primary key columns.
+ </p>
+
+ <p class="p">
+ Because Impala and Kudu do not support transactions, the effects of any
+ <code class="ph codeph">INSERT</code>, <code class="ph codeph">UPDATE</code>, or <code class="ph codeph">DELETE</code> statement
+ are immediately visible. For example, you cannot do a sequence of
+ <code class="ph codeph">UPDATE</code> statements and only make the changes visible after all the
+ statements are finished. Also, if a DML statement fails partway through, any rows that
+ were already inserted, deleted, or changed remain in the table; there is no rollback
+ mechanism to undo the changes.
+ </p>
+
+ <p class="p">
+ In particular, an <code class="ph codeph">INSERT ... SELECT</code> statement that refers to the table
+ being inserted into might insert more rows than expected, because the
+ <code class="ph codeph">SELECT</code> part of the statement sees some of the new rows being inserted
+ and processes them again.
+ </p>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ The <code class="ph codeph">LOAD DATA</code> statement, which involves manipulation of HDFS data files,
+ does not apply to Kudu tables.
+ </p>
+ </div>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title22" id="impala_kudu__kudu_consistency">
+
+ <h2 class="title topictitle2" id="ariaid-title22">Consistency Considerations for Kudu Tables</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Kudu tables have consistency characteristics such as uniqueness, controlled by the
+ primary key columns, and non-nullable columns. The emphasis for consistency is on
+ preventing duplicate or incomplete data from being stored in a table.
+ </p>
+
+ <p class="p">
+ Currently, Kudu does not enforce strong consistency for order of operations, total
+ success or total failure of a multi-row statement, or data that is read while a write
+ operation is in progress. Changes are applied atomically to each row, but not applied
+ as a single unit to all rows affected by a multi-row DML statement. That is, Kudu does
+ not currently have atomic multi-row statements or isolation between statements.
+ </p>
+
+ <p class="p">
+ If some rows are rejected during a DML operation because of a mismatch with duplicate
+ primary key values, <code class="ph codeph">NOT NULL</code> constraints, and so on, the statement
+ succeeds with a warning. Impala still inserts, deletes, or updates the other rows that
+ are not affected by the constraint violation.
+ </p>
+
+ <p class="p">
+ Consequently, the number of rows affected by a DML operation on a Kudu table might be
+ different than you expect.
+ </p>
+
+ <p class="p">
+ Because there is no strong consistency guarantee for information being inserted into,
+ deleted from, or updated across multiple tables simultaneously, consider denormalizing
+ the data where practical. That is, if you run separate <code class="ph codeph">INSERT</code>
+ statements to insert related rows into two different tables, one <code class="ph codeph">INSERT</code>
+ might fail while the other succeeds, leaving the data in an inconsistent state. Even if
+ both inserts succeed, a join query might happen during the interval between the
+ completion of the first and second statements, and the query would encounter incomplete
+ inconsistent data. Denormalizing the data into a single wide table can reduce the
+ possibility of inconsistency due to multi-table operations.
+ </p>
+
+ <p class="p">
+ Information about the number of rows affected by a DML operation is reported in
+ <span class="keyword cmdname">impala-shell</span> output, and in the <code class="ph codeph">PROFILE</code> output, but
+ is not currently reported to HiveServer2 clients such as JDBC or ODBC applications.
+ </p>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title23" id="impala_kudu__kudu_security">
+
+ <h2 class="title topictitle2" id="ariaid-title23">Security Considerations for Kudu Tables</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Security for Kudu tables involves:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ Sentry authorization.
+ </p>
+ <p class="p">
+ Access to Kudu tables must be granted to and revoked from roles as usual.
+ Only users with <code class="ph codeph">ALL</code> privileges on <code class="ph codeph">SERVER</code> can create external Kudu tables.
+ Currently, access to a Kudu table is <span class="q">"all or nothing"</span>:
+ enforced at the table level rather than the column level, and applying to all
+ SQL operations rather than individual statements such as <code class="ph codeph">INSERT</code>.
+ Because non-SQL APIs can access Kudu data without going through Sentry
+ authorization, currently the Sentry support is considered preliminary
+ and subject to change.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Lineage tracking.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Auditing.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Redaction of sensitive information from log files.
+ </p>
+ </li>
+ </ul>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title24" id="impala_kudu__kudu_performance">
+
+ <h2 class="title topictitle2" id="ariaid-title24">Impala Query Performance for Kudu Tables</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ For queries involving Kudu tables, Impala can delegate much of the work of filtering the
+ result set to Kudu, avoiding some of the I/O involved in full table scans of tables
+ containing HDFS data files. This type of optimization is especially effective for
+ partitioned Kudu tables, where the Impala query <code class="ph codeph">WHERE</code> clause refers to
+ one or more primary key columns that are also used as partition key columns. For
+ example, if a partitioned Kudu table uses a <code class="ph codeph">HASH</code> clause for
+ <code class="ph codeph">col1</code> and a <code class="ph codeph">RANGE</code> clause for <code class="ph codeph">col2</code>, a
+ query using a clause such as <code class="ph codeph">WHERE col1 IN (1,2,3) AND col2 > 100</code>
+ can determine exactly which tablet servers contain relevant data, and therefore
+ parallelize the query very efficiently.
+ </p>
+
+ <p class="p">
+ See <a class="xref" href="impala_explain.html">EXPLAIN Statement</a> for examples of evaluating the effectiveness of
+ the predicate pushdown for a specific query against a Kudu table.
+ </p>
+
+
+
+
+ </div>
+
+
+
+
+
+ </article>
+
+</article></main></body></html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_langref.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_langref.html b/docs/build/html/topics/impala_langref.html
new file mode 100644
index 0000000..b7f8546
--- /dev/null
+++ b/docs/build/html/topics/impala_langref.html
@@ -0,0 +1,66 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_comments.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_datatypes.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_literals.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_operators.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_schema_objects.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_functions.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_unsupported.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_porting.html"><met
a name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="langref"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Impala SQL Language Reference</title></head><body id="langref"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">Impala SQL Language Reference</h1>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ Impala uses SQL as its query language. To protect user investment in skills development and query
+ design, Impala provides a high degree of compatibility with the Hive Query Language (HiveQL):
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ Because Impala uses the same metadata store as Hive to record information about table structure and
+ properties, Impala can access tables defined through the native Impala <code class="ph codeph">CREATE TABLE</code>
+ command, or tables created using the Hive data definition language (DDL).
+ </li>
+
+ <li class="li">
+ Impala supports data manipulation (DML) statements similar to the DML component of HiveQL.
+ </li>
+
+ <li class="li">
+ Impala provides many <a class="xref" href="impala_functions.html#builtins">built-in functions</a> with the same
+ names and parameter types as their HiveQL equivalents.
+ </li>
+ </ul>
+
+ <p class="p">
+ Impala supports most of the same <a class="xref" href="impala_langref_sql.html#langref_sql">statements and
+ clauses</a> as HiveQL, including, but not limited to <code class="ph codeph">JOIN</code>, <code class="ph codeph">AGGREGATE</code>,
+ <code class="ph codeph">DISTINCT</code>, <code class="ph codeph">UNION ALL</code>, <code class="ph codeph">ORDER BY</code>, <code class="ph codeph">LIMIT</code> and
+ (uncorrelated) subquery in the <code class="ph codeph">FROM</code> clause. Impala also supports <code class="ph codeph">INSERT
+ INTO</code> and <code class="ph codeph">INSERT OVERWRITE</code>.
+ </p>
+
+ <p class="p">
+ Impala supports data types with the same names and semantics as the equivalent Hive data types:
+ <code class="ph codeph">STRING</code>, <code class="ph codeph">TINYINT</code>, <code class="ph codeph">SMALLINT</code>, <code class="ph codeph">INT</code>,
+ <code class="ph codeph">BIGINT</code>, <code class="ph codeph">FLOAT</code>, <code class="ph codeph">DOUBLE</code>, <code class="ph codeph">BOOLEAN</code>,
+ <code class="ph codeph">STRING</code>, <code class="ph codeph">TIMESTAMP</code>.
+ </p>
+
+ <p class="p">
+ For full details about Impala SQL syntax and semantics, see
+ <a class="xref" href="impala_langref_sql.html#langref_sql">Impala SQL Statements</a>.
+ </p>
+
+ <p class="p">
+ Most HiveQL <code class="ph codeph">SELECT</code> and <code class="ph codeph">INSERT</code> statements run unmodified with Impala. For
+ information about Hive syntax not available in Impala, see
+ <a class="xref" href="impala_langref_unsupported.html#langref_hiveql_delta">SQL Differences Between Impala and Hive</a>.
+ </p>
+
+ <p class="p">
+ For a list of the built-in functions available in Impala queries, see
+ <a class="xref" href="impala_functions.html#builtins">Impala Built-In Functions</a>.
+ </p>
+
+ <p class="p toc"></p>
+ </div>
+<nav role="navigation" class="related-links"><ul class="ullinks"><li class="link ulchildlink"><strong><a href="../topics/impala_comments.html">Comments</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_datatypes.html">Data Types</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_literals.html">Literals</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_operators.html">SQL Operators</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_schema_objects.html">Impala Schema Objects and Object Names</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_langref_sql.html">Impala SQL Statements</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_functions.html">Impala Built-In Functions</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_langref_unsupported.html">SQL Diff
erences Between Impala and Hive</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_porting.html">Porting SQL from Other Database Systems to Impala</a></strong><br></li></ul></nav></article></main></body></html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_langref_sql.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_langref_sql.html b/docs/build/html/topics/impala_langref_sql.html
new file mode 100644
index 0000000..9d6d33d
--- /dev/null
+++ b/docs/build/html/topics/impala_langref_sql.html
@@ -0,0 +1,28 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_ddl.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_dml.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_alter_table.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_alter_view.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_compute_stats.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_create_database.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_create_function.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_create_role.html"><meta name
="DC.Relation" scheme="URI" content="../topics/impala_create_table.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_create_view.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_delete.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_describe.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_drop_database.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_drop_function.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_drop_role.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_drop_stats.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_drop_table.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_drop_view.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_explain.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_grant.html"><meta name="DC.Relation" scheme="URI" cont
ent="../topics/impala_insert.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_invalidate_metadata.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_load_data.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_refresh.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_revoke.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_select.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_set.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_show.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_truncate_table.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_update.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_upsert.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_use.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version"
content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="langref_sql"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Impala SQL Statements</title></head><body id="langref_sql"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">Impala SQL Statements</h1>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ The Impala SQL dialect supports a range of standard elements, plus some extensions for Big Data use cases
+ related to data loading and data warehousing.
+ </p>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ In the <span class="keyword cmdname">impala-shell</span> interpreter, a semicolon at the end of each statement is required.
+ Since the semicolon is not actually part of the SQL syntax, we do not include it in the syntax definition
+ of each statement, but we do show it in examples intended to be run in <span class="keyword cmdname">impala-shell</span>.
+ </p>
+ </div>
+
+ <p class="p toc all">
+ The following sections show the major SQL statements that you work with in Impala:
+ </p>
+ </div>
+<nav role="navigation" class="related-links"><ul class="ullinks"><li class="link ulchildlink"><strong><a href="../topics/impala_ddl.html">DDL Statements</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_dml.html">DML Statements</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_alter_table.html">ALTER TABLE Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_alter_view.html">ALTER VIEW Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_compute_stats.html">COMPUTE STATS Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_create_database.html">CREATE DATABASE Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_create_function.html">CREATE FUNCTION Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_create_role.h
tml">CREATE ROLE Statement (Impala 2.0 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_create_table.html">CREATE TABLE Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_create_view.html">CREATE VIEW Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_delete.html">DELETE Statement (Impala 2.8 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_describe.html">DESCRIBE Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_drop_database.html">DROP DATABASE Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_drop_function.html">DROP FUNCTION Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_drop_role.html">DROP ROLE Statement (Impala 2.0 or higher only)</a></strong><br></li><li cla
ss="link ulchildlink"><strong><a href="../topics/impala_drop_stats.html">DROP STATS Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_drop_table.html">DROP TABLE Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_drop_view.html">DROP VIEW Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_explain.html">EXPLAIN Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_grant.html">GRANT Statement (Impala 2.0 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_insert.html">INSERT Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_invalidate_metadata.html">INVALIDATE METADATA Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_load_data.html">LOAD DATA Statement</a></strong><br></li><li class=
"link ulchildlink"><strong><a href="../topics/impala_refresh.html">REFRESH Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_revoke.html">REVOKE Statement (Impala 2.0 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_select.html">SELECT Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_set.html">SET Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_show.html">SHOW Statement</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_truncate_table.html">TRUNCATE TABLE Statement (Impala 2.3 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_update.html">UPDATE Statement (Impala 2.8 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_upsert.html">UPSERT Statement (Impala 2.8 or higher on
ly)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_use.html">USE Statement</a></strong><br></li></ul><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref.html">Impala SQL Language Reference</a></div></div></nav></article></main></body></html>
\ No newline at end of file